2024-09-14T03:26:19.812 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-14T03:26:19.839 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-14T03:26:19.930 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-14_03:08:15-upgrade-main-distro-default-smithi/7904534 branch: main description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} email: ceph-qa@ceph.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps - echo wait for minority of mons to upgrade - while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error; do sleep 2 ; done - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for majority of mons to upgrade - 'while ! ceph mon versions | grep $sha1 | egrep '': [23]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for all mons to upgrade - 'while ! ceph mon versions | grep $sha1 | grep '': 3'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for half of osds to upgrade - 'while ! ceph osd versions | grep $sha1 | egrep '': [45678]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch upgrade pause - ceph orch ps - ceph orch ps - ceph versions first-half-tasks: - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' first_in_suite: false job_id: '7904534' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run basic import/export cli tests for rbd ' name: teuthology-2024-09-14_03:08:15-upgrade-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSDMAP_FLAGS - PG_AVAILABILITY sha1: a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 1c941a66c1b420259b8fc5ac19e792dbbf60aded owner: scheduled_teuthology@teuthology priority: 850 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - ceph orch upgrade resume - sleep 60 - echo wait for upgrade to complete - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' seed: 8038 sha1: a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: quincy clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 31/32 suite: upgrade suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c941a66c1b420259b8fc5ac19e792dbbf60aded/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c941a66c1b420259b8fc5ac19e792dbbf60aded targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBELTeGqqB8oVBe2XFcA69Tno6cQ/EvM7VtH/eTaQnIt078iCmx5F0hwzXFSgiCVcdvcxJ9EfXpKBXxNMmSOwUFE= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWciQPFIg2zj+ZmrmWwdbjztPvjihaOhxorcu96r4Ewuayg4+kiWNHFq5kYSkKkzc0wiq234Lpl3U1VenA5+rs= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: branch: quincy exclude_packages: - ceph-volume - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-14_03:08:15 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-14T03:26:19.930 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c941a66c1b420259b8fc5ac19e792dbbf60aded/qa; will attempt to use it 2024-09-14T03:26:19.931 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c941a66c1b420259b8fc5ac19e792dbbf60aded/qa/tasks 2024-09-14T03:26:19.931 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-14T03:26:19.933 INFO:teuthology.task.internal:Checking packages... 2024-09-14T03:26:19.955 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a2782315255a3e7d5f6cdc9eb35785187ff6ddf8' 2024-09-14T03:26:19.955 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T03:26:19.955 INFO:teuthology.packaging:ref: None 2024-09-14T03:26:19.955 INFO:teuthology.packaging:tag: None 2024-09-14T03:26:19.955 INFO:teuthology.packaging:branch: main 2024-09-14T03:26:19.955 INFO:teuthology.packaging:sha1: a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:26:19.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-14T03:26:20.183 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4926.g1c941a66 2024-09-14T03:26:20.185 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-14T03:26:20.191 INFO:teuthology.task.internal:no buildpackages task found 2024-09-14T03:26:20.191 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-14T03:26:20.229 INFO:teuthology.task.internal:Saving configuration 2024-09-14T03:26:20.252 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-14T03:26:20.263 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-14T03:26:20.284 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-14_03:08:15-upgrade-main-distro-default-smithi/7904534', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-14 03:21:16.024327', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBELTeGqqB8oVBe2XFcA69Tno6cQ/EvM7VtH/eTaQnIt078iCmx5F0hwzXFSgiCVcdvcxJ9EfXpKBXxNMmSOwUFE='} 2024-09-14T03:26:20.307 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-14_03:08:15-upgrade-main-distro-default-smithi/7904534', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-14 03:21:16.025605', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWciQPFIg2zj+ZmrmWwdbjztPvjihaOhxorcu96r4Ewuayg4+kiWNHFq5kYSkKkzc0wiq234Lpl3U1VenA5+rs='} 2024-09-14T03:26:20.307 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-14T03:26:20.314 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-14T03:26:20.314 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-14T03:26:20.314 INFO:teuthology.run_tasks:Running task console_log... 2024-09-14T03:26:20.409 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f707ca1bac0>, signals=[15]) 2024-09-14T03:26:20.409 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-14T03:26:20.415 INFO:teuthology.task.internal:Opening connections... 2024-09-14T03:26:20.415 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-09-14T03:26:20.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:26:20.495 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2024-09-14T03:26:20.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:26:20.572 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-14T03:26:20.578 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-09-14T03:26:20.597 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-09-14T03:26:20.597 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-09-14T03:26:20.655 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-09-14T03:26:20.655 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-09-14T03:26:20.655 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-09-14T03:26:20.655 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-09-14T03:26:20.655 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-09-14T03:26:20.655 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-09-14T03:26:20.655 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-14T03:26:20.655 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-09-14T03:26:20.656 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-09-14T03:26:20.656 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-14T03:26:20.656 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-09-14T03:26:20.656 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-14T03:26:20.656 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-14T03:26:20.656 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-14T03:26:20.657 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-09-14T03:26:20.680 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2024-09-14T03:26:20.699 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2024-09-14T03:26:20.699 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2024-09-14T03:26:20.757 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2024-09-14T03:26:20.758 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="9" 2024-09-14T03:26:20.758 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2024-09-14T03:26:20.758 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2024-09-14T03:26:20.758 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="9" 2024-09-14T03:26:20.758 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el9" 2024-09-14T03:26:20.758 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-14T03:26:20.758 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2024-09-14T03:26:20.758 INFO:teuthology.orchestra.run.smithi160.stdout:LOGO="fedora-logo-icon" 2024-09-14T03:26:20.758 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-14T03:26:20.758 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2024-09-14T03:26:20.759 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-14T03:26:20.759 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-14T03:26:20.759 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-14T03:26:20.759 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2024-09-14T03:26:20.780 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-14T03:26:20.788 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-14T03:26:20.837 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-14T03:26:20.838 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-09-14T03:26:20.840 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2024-09-14T03:26:20.858 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-14T03:26:20.871 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-14T03:26:20.871 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-09-14T03:26:20.898 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2024-09-14T03:26:20.988 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-14T03:26:21.037 INFO:teuthology.run_tasks:Running task kernel... 2024-09-14T03:26:21.053 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-14T03:26:21.053 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-14T03:26:21.054 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-14T03:26:21.054 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-14T03:26:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T03:26:21.072 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-14T03:26:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T03:26:21.073 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-09-14T03:26:21.130 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-503.el9.x86_64 2024-09-14T03:26:21.130 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-503.el9.x86_64 2024-09-14T03:26:21.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-09-14T03:26:21.131 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-503.el9.x86_64 2024-09-14T03:26:21.131 INFO:teuthology.task.kernel:Running kernel on smithi160: 5.14.0-503.el9.x86_64 2024-09-14T03:26:21.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum install -y kernel 2024-09-14T03:26:23.472 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-14T03:26:23.562 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-14T03:26:25.498 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-14T03:26:25.522 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-14T03:26:29.772 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-14T03:26:31.315 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.5 MB 00:02 2024-09-14T03:26:31.365 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 19 kB 00:00 2024-09-14T03:26:32.128 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-14T03:26:33.757 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 19 kB 00:01 2024-09-14T03:26:34.673 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-14T03:26:37.675 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-14T03:26:39.656 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-14T03:26:39.657 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-14T03:26:39.751 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:26:39.757 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:26:39.757 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-14T03:26:39.757 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:26:39.757 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:26:39.757 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-14T03:26:39.757 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-14T03:26:39.757 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-14T03:26:39.757 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-14T03:26:39.757 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-14T03:26:39.757 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:26:39.758 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:26:39.758 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:26:39.758 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-09-14T03:26:39.758 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:26:39.758 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 79 M 2024-09-14T03:26:39.758 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 126 M 2024-09-14T03:26:39.759 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:26:40.183 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-14T03:26:40.468 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 39 kB/s | 23 kB 00:00 2024-09-14T03:26:41.393 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-14T03:26:41.951 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 17 MB/s | 35 MB 00:02 2024-09-14T03:26:42.224 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-14T03:26:42.224 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-14T03:26:42.313 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:26:42.317 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:26:42.317 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-14T03:26:42.317 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:26:42.317 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:26:42.317 INFO:teuthology.orchestra.run.smithi160.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-14T03:26:42.317 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-14T03:26:42.317 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-14T03:26:42.318 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-14T03:26:42.318 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-14T03:26:42.318 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:26:42.318 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:26:42.318 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:26:42.318 INFO:teuthology.orchestra.run.smithi160.stdout:Install 4 Packages 2024-09-14T03:26:42.318 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:26:42.319 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 79 M 2024-09-14T03:26:42.319 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 126 M 2024-09-14T03:26:42.319 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:26:42.719 INFO:teuthology.orchestra.run.smithi160.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 188 kB/s | 23 kB 00:00 2024-09-14T03:26:42.801 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-14T03:26:42.802 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:26:42.802 INFO:teuthology.orchestra.run.smithi114.stdout:Total 26 MB/s | 79 MB 00:03 2024-09-14T03:26:43.294 INFO:teuthology.orchestra.run.smithi160.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-09-14T03:26:43.487 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:26:43.794 INFO:teuthology.orchestra.run.smithi160.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-14T03:26:43.855 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:26:43.856 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:26:44.569 INFO:teuthology.orchestra.run.smithi160.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 15 MB/s | 29 MB 00:01 2024-09-14T03:26:44.569 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:26:44.569 INFO:teuthology.orchestra.run.smithi160.stdout:Total 35 MB/s | 79 MB 00:02 2024-09-14T03:26:44.830 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:26:44.830 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:26:45.254 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:26:45.598 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:26:45.598 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:26:46.597 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:26:46.598 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:26:47.275 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:26:47.994 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-14T03:26:48.079 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T03:26:49.056 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:26:49.978 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-14T03:26:50.061 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T03:26:50.877 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T03:26:51.003 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T03:26:52.880 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T03:26:53.014 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T03:26:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T03:26:55.945 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:26:57.832 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T03:26:57.872 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:00.773 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:02.701 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:38.741 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:38.780 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:39.134 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:39.134 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-14T03:27:39.135 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T03:27:39.135 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T03:27:40.077 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:40.077 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:27:40.077 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:27:40.078 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-14T03:27:40.078 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-14T03:27:40.078 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-14T03:27:40.078 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-14T03:27:40.078 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:27:40.078 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:27:40.439 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T03:27:40.528 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T03:27:40.528 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T03:27:40.529 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-14T03:27:40.529 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-14T03:27:40.529 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-14T03:27:40.529 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-14T03:27:40.529 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-09-14T03:27:40.529 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T03:27:40.587 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T03:27:40.587 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T03:27:40.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-09-14T03:27:40.620 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-09-14T03:27:40.620 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-09-14T03:27:40.621 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-09-14T03:27:40.621 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 82.el9 2024-09-14T03:27:40.621 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-09-14T03:27:40.621 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-14T03:27:40.621 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-09-14T03:27:40.621 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8274670 2024-09-14T03:27:40.622 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-09-14T03:27:40.622 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-14T03:27:40.622 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-14T03:27:40.622 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-14T03:27:40.622 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-14T03:27:40.622 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-09-14T03:27:40.623 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-09-14T03:27:40.623 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-14T03:27:40.623 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-09-14T03:27:40.623 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-09-14T03:27:40.623 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:27:40.623 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-14T03:27:40.624 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-14T03:27:40.624 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-14T03:27:40.624 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-09-14T03:27:40.624 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:27:40.624 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-09-14T03:27:40.626 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-14T03:27:40.626 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-505.el9.x86_64 2024-09-14T03:27:40.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-14T03:27:41.203 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:41.251 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:41.251 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-09-14T03:27:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-14T03:27:41.610 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T03:27:41.611 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T03:27:42.498 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T03:27:42.498 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:27:42.499 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:27:42.499 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-14T03:27:42.499 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-14T03:27:42.499 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-14T03:27:42.499 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-14T03:27:42.499 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:27:42.499 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:27:42.541 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-14T03:27:42.571 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-09-14T03:27:42.574 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-09-14T03:27:42.589 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.6Dku1H9AwC 2024-09-14T03:27:42.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6Dku1H9AwC 2024-09-14T03:27:42.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.6Dku1H9AwC 2024-09-14T03:27:42.798 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.6Dku1H9AwC is 6KB 2024-09-14T03:27:42.847 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.6Dku1H9AwC 2024-09-14T03:27:42.853 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T03:27:42.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-09-14T03:27:42.933 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T03:27:42.934 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T03:27:42.934 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-14T03:27:42.934 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-14T03:27:42.934 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-14T03:27:42.934 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-14T03:27:42.934 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2024-09-14T03:27:42.934 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T03:27:42.945 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-14T03:27:42.945 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-14T03:27:42.945 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-14T03:27:42.945 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-14T03:27:42.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-14T03:27:42.993 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T03:27:42.993 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T03:27:42.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo rpm -qi grub2-tools 2024-09-14T03:27:43.025 INFO:teuthology.orchestra.run.smithi160.stdout:Name : grub2-tools 2024-09-14T03:27:43.025 INFO:teuthology.orchestra.run.smithi160.stdout:Epoch : 1 2024-09-14T03:27:43.025 INFO:teuthology.orchestra.run.smithi160.stdout:Version : 2.06 2024-09-14T03:27:43.025 INFO:teuthology.orchestra.run.smithi160.stdout:Release : 82.el9 2024-09-14T03:27:43.025 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: x86_64 2024-09-14T03:27:43.025 INFO:teuthology.orchestra.run.smithi160.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-14T03:27:43.025 INFO:teuthology.orchestra.run.smithi160.stdout:Group : Unspecified 2024-09-14T03:27:43.025 INFO:teuthology.orchestra.run.smithi160.stdout:Size : 8274670 2024-09-14T03:27:43.025 INFO:teuthology.orchestra.run.smithi160.stdout:License : GPLv3+ 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:Packager : builder@centos.org 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:Vendor : CentOS 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:Summary : Support tools for GRUB. 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:Description : 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-14T03:27:43.026 INFO:teuthology.orchestra.run.smithi160.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-14T03:27:43.027 INFO:teuthology.orchestra.run.smithi160.stdout:hardware devices. 2024-09-14T03:27:43.027 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:27:43.027 INFO:teuthology.orchestra.run.smithi160.stdout:This subpackage provides tools for support of all platforms. 2024-09-14T03:27:43.027 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-14T03:27:43.028 INFO:teuthology.task.kernel:Updating grub on smithi160 to boot 5.14.0-505.el9.x86_64 2024-09-14T03:27:43.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-14T03:27:43.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-09-14T03:27:43.690 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2024-09-14T03:27:45.008 INFO:teuthology.orchestra.run.smithi160.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-14T03:27:45.041 INFO:teuthology.orchestra.run.smithi160.stderr:done 2024-09-14T03:27:45.043 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2024-09-14T03:27:45.058 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.B9kgcUmfOE 2024-09-14T03:27:45.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.B9kgcUmfOE 2024-09-14T03:27:45.134 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.B9kgcUmfOE 2024-09-14T03:27:45.272 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.B9kgcUmfOE is 6KB 2024-09-14T03:27:45.320 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.B9kgcUmfOE 2024-09-14T03:27:45.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo /bin/ls /boot/loader/entries || true 2024-09-14T03:27:45.406 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-14T03:27:45.406 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-14T03:27:45.406 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-14T03:27:45.406 INFO:teuthology.orchestra.run.smithi160.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-14T03:27:45.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-14T03:27:45.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2024-09-14T03:28:13.085 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-14T03:28:13.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-14T03:28:13.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:28:15.557 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-14T03:28:15.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-14T03:28:15.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:28:31.553 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-14T03:28:34.109 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-14T03:28:40.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-14T03:28:40.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:28:43.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-14T03:28:43.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:28:43.617 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-14T03:28:46.177 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-14T03:28:55.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-14T03:28:55.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:28:58.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-14T03:28:58.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:29:16.609 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-14T03:29:27.488 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-14T03:29:27.916 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-14T03:29:27.917 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-14T03:29:27.917 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-14T03:29:27.932 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-505.el9.x86_64 2024-09-14T03:29:27.932 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-14T03:29:27.932 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-14T03:29:27.932 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-14T03:29:28.933 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-14T03:29:28.933 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-14T03:29:29.060 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-09-14T03:29:29.080 DEBUG:teuthology.parallel:result is None 2024-09-14T03:29:31.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-14T03:29:31.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:29:32.009 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-09-14T03:29:32.344 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-14T03:29:32.344 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-14T03:29:32.344 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-09-14T03:29:32.359 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-505.el9.x86_64 2024-09-14T03:29:32.359 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-14T03:29:32.359 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-14T03:29:32.359 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-14T03:29:33.360 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-14T03:29:33.360 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-14T03:29:33.466 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2024-09-14T03:29:33.487 DEBUG:teuthology.parallel:result is None 2024-09-14T03:29:33.488 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-14T03:29:33.494 INFO:teuthology.task.internal:Creating test directory... 2024-09-14T03:29:33.495 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-14T03:29:33.498 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-14T03:29:33.514 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-14T03:29:33.523 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-14T03:29:33.569 INFO:teuthology.task.internal:Creating archive directory... 2024-09-14T03:29:33.569 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-14T03:29:33.572 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-14T03:29:33.684 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-14T03:29:33.690 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-14T03:29:33.691 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-14T03:29:33.693 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-14T03:29:33.719 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T03:29:33.729 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T03:29:33.729 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T03:29:33.740 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T03:29:33.742 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-14T03:29:33.749 INFO:teuthology.task.internal:Configuring sudo... 2024-09-14T03:29:33.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-14T03:29:33.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-14T03:29:33.807 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-14T03:29:33.817 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-14T03:29:33.817 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-14T03:29:33.850 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-14T03:29:33.866 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T03:29:33.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T03:29:34.011 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T03:29:34.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T03:29:34.096 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:29:34.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-14T03:29:34.162 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T03:29:34.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T03:29:34.253 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T03:29:34.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T03:29:34.427 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:29:34.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-14T03:29:34.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-09-14T03:29:34.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2024-09-14T03:29:34.525 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T03:29:34.603 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T03:29:34.965 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-14T03:29:34.973 INFO:teuthology.task.internal:Starting timer... 2024-09-14T03:29:34.973 INFO:teuthology.run_tasks:Running task pcp... 2024-09-14T03:29:35.017 INFO:teuthology.run_tasks:Running task selinux... 2024-09-14T03:29:35.066 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-14T03:29:35.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-09-14T03:29:35.118 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-09-14T03:29:35.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2024-09-14T03:29:35.222 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2024-09-14T03:29:35.225 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-14T03:29:35.225 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-09-14T03:29:35.260 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-09-14T03:29:35.260 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2024-09-14T03:29:35.320 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2024-09-14T03:29:35.320 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive'} 2024-09-14T03:29:35.321 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-14T03:29:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T03:29:35.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-14T03:29:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T03:29:35.375 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-14T03:29:35.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-09-14T03:29:35.426 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2024-09-14T03:29:35.465 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-14T03:29:35.475 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-14T03:29:35.487 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-14T03:29:35.488 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-14T03:33:19.245 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2024-09-14T03:33:19.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-14T03:33:19.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:33:19.328 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-14T03:33:19.408 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-14T03:33:19.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-14T03:33:19.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T03:33:19.485 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-09-14T03:33:19.564 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-14T03:33:19.564 INFO:teuthology.run_tasks:Running task clock... 2024-09-14T03:33:19.575 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-14T03:33:19.575 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-14T03:33:19.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T03:33:19.578 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-14T03:33:19.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T03:33:19.605 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-14T03:33:19.619 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-14T03:33:19.634 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-14T03:33:19.642 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-09-14T03:33:19.648 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-14T03:33:19.653 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-09-14T03:33:19.667 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-14T03:33:19.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2024-09-14T03:33:19.681 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-14T03:33:19.683 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2024-09-14T03:33:19.698 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-14T03:33:19.712 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-14T03:33:19.735 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-14T03:33:19.739 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T03:33:19.739 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-14T03:33:19.740 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T03:33:19.740 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T03:33:19.740 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T03:33:19.740 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T03:33:19.768 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-09-14T03:33:19.772 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T03:33:19.772 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-09-14T03:33:19.772 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T03:33:19.772 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T03:33:19.772 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T03:33:19.772 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T03:33:19.773 INFO:teuthology.run_tasks:Running task pexec... 2024-09-14T03:33:19.783 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-14T03:33:19.783 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi114.front.sepia.ceph.com 2024-09-14T03:33:19.783 DEBUG:teuthology.orchestra.run.smithi114:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-14T03:33:19.784 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi160.front.sepia.ceph.com 2024-09-14T03:33:19.784 DEBUG:teuthology.orchestra.run.smithi160:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-14T03:33:20.134 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:33:20.135 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:33:20.135 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout:Removing: 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout:Removing unused dependencies: 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout:Remove 2 Packages 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout:Freed space: 5.5 M 2024-09-14T03:33:20.136 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:33:20.140 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:33:20.140 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:33:20.162 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:33:20.180 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:33:20.180 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-14T03:33:20.180 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:33:20.180 INFO:teuthology.orchestra.run.smithi160.stdout:Removing: 2024-09-14T03:33:20.181 INFO:teuthology.orchestra.run.smithi160.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-14T03:33:20.181 INFO:teuthology.orchestra.run.smithi160.stdout:Removing unused dependencies: 2024-09-14T03:33:20.181 INFO:teuthology.orchestra.run.smithi160.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-14T03:33:20.181 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:20.181 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:33:20.181 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:33:20.181 INFO:teuthology.orchestra.run.smithi160.stdout:Remove 2 Packages 2024-09-14T03:33:20.181 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:20.181 INFO:teuthology.orchestra.run.smithi160.stdout:Freed space: 5.5 M 2024-09-14T03:33:20.181 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:33:20.185 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:33:20.185 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:33:20.220 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:33:20.221 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:33:20.245 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:33:20.245 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:33:20.353 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:33:20.376 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:33:20.428 INFO:teuthology.orchestra.run.smithi114.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-14T03:33:20.469 INFO:teuthology.orchestra.run.smithi160.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-14T03:33:20.485 INFO:teuthology.orchestra.run.smithi114.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-14T03:33:20.534 INFO:teuthology.orchestra.run.smithi160.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-14T03:33:20.921 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-14T03:33:20.921 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-14T03:33:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-14T03:33:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-14T03:33:21.165 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-14T03:33:21.165 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:21.165 INFO:teuthology.orchestra.run.smithi160.stdout:Removed: 2024-09-14T03:33:21.165 INFO:teuthology.orchestra.run.smithi160.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-14T03:33:21.165 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:21.165 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:33:21.295 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-14T03:33:21.295 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:21.295 INFO:teuthology.orchestra.run.smithi114.stdout:Removed: 2024-09-14T03:33:21.295 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-14T03:33:21.296 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:21.296 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:33:21.984 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:48 ago on Sat 14 Sep 2024 03:32:33 AM UTC. 2024-09-14T03:33:22.123 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:33:22.124 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:33:22.124 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-14T03:33:22.124 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:33:22.125 INFO:teuthology.orchestra.run.smithi160.stdout:Install 7 Packages 2024-09-14T03:33:22.126 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:22.140 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 5.2 M 2024-09-14T03:33:22.140 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 19 M 2024-09-14T03:33:22.140 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:33:22.170 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:48 ago on Sat 14 Sep 2024 03:32:33 AM UTC. 2024-09-14T03:33:22.303 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:33:22.303 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:22.304 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:33:22.305 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:33:22.305 INFO:teuthology.orchestra.run.smithi114.stdout:Install 7 Packages 2024-09-14T03:33:22.305 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:22.315 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 5.2 M 2024-09-14T03:33:22.315 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 19 M 2024-09-14T03:33:22.315 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:33:22.525 INFO:teuthology.orchestra.run.smithi114.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 514 kB/s | 44 kB 00:00 2024-09-14T03:33:22.550 INFO:teuthology.orchestra.run.smithi114.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 947 kB/s | 106 kB 00:00 2024-09-14T03:33:22.584 INFO:teuthology.orchestra.run.smithi114.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 1.2 MB/s | 72 kB 00:00 2024-09-14T03:33:22.609 INFO:teuthology.orchestra.run.smithi114.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-14T03:33:22.676 INFO:teuthology.orchestra.run.smithi114.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.7 MB/s | 906 kB 00:00 2024-09-14T03:33:22.759 INFO:teuthology.orchestra.run.smithi114.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.7 MB/s | 837 kB 00:00 2024-09-14T03:33:22.873 INFO:teuthology.orchestra.run.smithi160.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 73 kB/s | 44 kB 00:00 2024-09-14T03:33:22.932 INFO:teuthology.orchestra.run.smithi160.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 159 kB/s | 106 kB 00:00 2024-09-14T03:33:22.951 INFO:teuthology.orchestra.run.smithi114.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-14T03:33:22.951 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:33:22.951 INFO:teuthology.orchestra.run.smithi114.stdout:Total 8.1 MB/s | 5.2 MB 00:00 2024-09-14T03:33:23.061 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:33:23.076 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:33:23.076 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:33:23.215 INFO:teuthology.orchestra.run.smithi160.stdout:(3/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 954 kB/s | 906 kB 00:00 2024-09-14T03:33:23.379 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:33:23.379 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:33:23.549 INFO:teuthology.orchestra.run.smithi160.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 106 kB/s | 72 kB 00:00 2024-09-14T03:33:23.574 INFO:teuthology.orchestra.run.smithi160.stdout:(5/7): python3-kmod-0.9-32.el9.x86_64.rpm 131 kB/s | 84 kB 00:00 2024-09-14T03:33:23.854 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:33:23.874 INFO:teuthology.orchestra.run.smithi160.stdout:(6/7): runc-1.1.13-4.el9.x86_64.rpm 9.8 MB/s | 3.2 MB 00:00 2024-09-14T03:33:23.937 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-14T03:33:23.972 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-14T03:33:24.000 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-14T03:33:24.205 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-14T03:33:24.208 INFO:teuthology.orchestra.run.smithi160.stdout:(7/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 845 kB/s | 837 kB 00:00 2024-09-14T03:33:24.208 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:33:24.208 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.5 MB/s | 5.2 MB 00:02 2024-09-14T03:33:24.240 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-14T03:33:24.323 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:33:24.338 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:33:24.338 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:33:24.656 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:33:24.656 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:33:24.678 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-14T03:33:24.714 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-14T03:33:24.829 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-14T03:33:24.896 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-14T03:33:25.128 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:33:25.219 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-14T03:33:25.254 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-14T03:33:25.282 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-14T03:33:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-14T03:33:25.521 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-14T03:33:25.882 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-14T03:33:25.882 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-14T03:33:25.883 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-14T03:33:25.883 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-14T03:33:25.883 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-14T03:33:25.883 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-14T03:33:25.883 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-14T03:33:25.960 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-14T03:33:26.115 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-14T03:33:26.209 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-14T03:33:26.226 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-14T03:33:26.226 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:26.227 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:33:26.227 INFO:teuthology.orchestra.run.smithi114.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-14T03:33:26.227 INFO:teuthology.orchestra.run.smithi114.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-14T03:33:26.227 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-14T03:33:26.227 INFO:teuthology.orchestra.run.smithi114.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-14T03:33:26.227 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:26.227 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:33:26.277 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-14T03:33:26.615 DEBUG:teuthology.parallel:result is None 2024-09-14T03:33:27.142 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-14T03:33:27.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-14T03:33:27.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-14T03:33:27.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-14T03:33:27.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-14T03:33:27.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-14T03:33:27.143 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-14T03:33:27.496 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-14T03:33:27.496 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:27.497 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:33:27.497 INFO:teuthology.orchestra.run.smithi160.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-14T03:33:27.497 INFO:teuthology.orchestra.run.smithi160.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-14T03:33:27.497 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-14T03:33:27.497 INFO:teuthology.orchestra.run.smithi160.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-14T03:33:27.497 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:27.497 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:33:27.854 DEBUG:teuthology.parallel:result is None 2024-09-14T03:33:27.855 INFO:teuthology.run_tasks:Running task install... 2024-09-14T03:33:27.866 DEBUG:teuthology.task.install:project ceph 2024-09-14T03:33:27.867 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a2782315255a3e7d5f6cdc9eb35785187ff6ddf8'}} 2024-09-14T03:33:27.867 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'a2782315255a3e7d5f6cdc9eb35785187ff6ddf8'} 2024-09-14T03:33:27.867 INFO:teuthology.task.install:Using flavor: default 2024-09-14T03:33:27.872 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-14T03:33:27.872 INFO:teuthology.task.install:extra packages: [] 2024-09-14T03:33:27.873 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a2782315255a3e7d5f6cdc9eb35785187ff6ddf8', 'tag': None, 'wait_for_package': False} 2024-09-14T03:33:27.873 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T03:33:27.873 INFO:teuthology.packaging:ref: None 2024-09-14T03:33:27.873 INFO:teuthology.packaging:tag: None 2024-09-14T03:33:27.873 INFO:teuthology.packaging:branch: quincy 2024-09-14T03:33:27.874 INFO:teuthology.packaging:sha1: a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:33:27.874 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-14T03:33:27.876 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a2782315255a3e7d5f6cdc9eb35785187ff6ddf8', 'tag': None, 'wait_for_package': False} 2024-09-14T03:33:27.876 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T03:33:27.876 INFO:teuthology.packaging:ref: None 2024-09-14T03:33:27.876 INFO:teuthology.packaging:tag: None 2024-09-14T03:33:27.876 INFO:teuthology.packaging:branch: quincy 2024-09-14T03:33:27.876 INFO:teuthology.packaging:sha1: a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:33:27.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-14T03:33:28.084 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b/centos/9/flavors/default/ 2024-09-14T03:33:28.084 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1629.geb16b97b 2024-09-14T03:33:28.101 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b/centos/9/flavors/default/ 2024-09-14T03:33:28.102 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1629.geb16b97b 2024-09-14T03:33:28.214 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-14T03:33:28.214 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:33:28.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-14T03:33:28.230 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-14T03:33:28.230 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:33:28.230 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-14T03:33:28.240 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-14T03:33:28.240 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T03:33:28.241 INFO:teuthology.packaging:ref: None 2024-09-14T03:33:28.241 INFO:teuthology.packaging:tag: None 2024-09-14T03:33:28.241 INFO:teuthology.packaging:branch: quincy 2024-09-14T03:33:28.241 INFO:teuthology.packaging:sha1: a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:33:28.241 DEBUG:teuthology.orchestra.run.smithi160:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-14T03:33:28.261 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-14T03:33:28.261 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T03:33:28.261 INFO:teuthology.packaging:ref: None 2024-09-14T03:33:28.261 INFO:teuthology.packaging:tag: None 2024-09-14T03:33:28.261 INFO:teuthology.packaging:branch: quincy 2024-09-14T03:33:28.261 INFO:teuthology.packaging:sha1: a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:33:28.261 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-14T03:33:28.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-14T03:33:28.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-14T03:33:28.359 DEBUG:teuthology.orchestra.run.smithi160:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-14T03:33:28.410 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-14T03:33:28.423 INFO:teuthology.orchestra.run.smithi160.stdout:check_obsoletes = 1 2024-09-14T03:33:28.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum clean all 2024-09-14T03:33:28.476 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-09-14T03:33:28.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-09-14T03:33:28.654 INFO:teuthology.orchestra.run.smithi160.stdout:45 files removed 2024-09-14T03:33:28.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-radosgw 2024-09-14T03:33:28.704 INFO:teuthology.orchestra.run.smithi114.stdout:45 files removed 2024-09-14T03:33:28.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw 2024-09-14T03:33:29.247 INFO:teuthology.orchestra.run.smithi160.stdout:ceph packages for x86_64 274 kB/s | 77 kB 00:00 2024-09-14T03:33:29.292 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 281 kB/s | 77 kB 00:00 2024-09-14T03:33:29.539 INFO:teuthology.orchestra.run.smithi160.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-14T03:33:29.576 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-14T03:33:29.815 INFO:teuthology.orchestra.run.smithi160.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-14T03:33:29.852 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-14T03:33:30.393 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-14T03:33:30.557 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-14T03:33:32.408 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-14T03:33:32.430 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-14T03:33:37.226 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.5 MB 00:00 2024-09-14T03:33:37.931 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-14T03:33:38.709 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 101 kB/s | 19 kB 00:00 2024-09-14T03:33:39.340 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 101 kB/s | 19 kB 00:00 2024-09-14T03:33:39.814 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-14T03:33:40.736 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-14T03:33:45.528 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-14T03:33:46.166 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-14T03:33:47.909 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:33:47.911 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:33:47.911 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-14T03:33:47.911 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:33:47.911 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:33:47.911 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 14 M 2024-09-14T03:33:47.911 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrading: 2024-09-14T03:33:47.911 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 3.1 M 2024-09-14T03:33:47.911 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1 x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 2.9 M 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 5.0 M 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 23 M 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 25 k 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2 x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 636 k 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 458 k 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-14T03:33:47.912 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2 x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 4.9 M 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 45 k 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 120 k 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 157 k 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 320 k 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 296 k 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 99 k 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:33:47.913 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:33:47.914 INFO:teuthology.orchestra.run.smithi160.stdout:Install 22 Packages 2024-09-14T03:33:47.914 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrade 2 Packages 2024-09-14T03:33:47.914 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:47.914 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 58 M 2024-09-14T03:33:47.914 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:33:48.488 INFO:teuthology.orchestra.run.smithi160.stdout:(1/24): ceph-base-17.2.7-1629.geb16b97b.el9.x86 9.6 MB/s | 5.0 MB 00:00 2024-09-14T03:33:48.536 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:33:48.538 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:33:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-14T03:33:48.538 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:33:48.538 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:33:48.538 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 14 M 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 3.1 M 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 2.9 M 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 5.0 M 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 23 M 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 25 k 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 636 k 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-14T03:33:48.539 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 458 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 4.9 M 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 45 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 120 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 157 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 320 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 296 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 99 k 2024-09-14T03:33:48.540 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-14T03:33:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:48.541 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:33:48.541 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:33:48.541 INFO:teuthology.orchestra.run.smithi114.stdout:Install 22 Packages 2024-09-14T03:33:48.541 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 2 Packages 2024-09-14T03:33:48.541 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:48.541 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 58 M 2024-09-14T03:33:48.542 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:33:48.546 INFO:teuthology.orchestra.run.smithi160.stdout:(2/24): ceph-selinux-17.2.7-1629.geb16b97b.el9. 433 kB/s | 25 kB 00:00 2024-09-14T03:33:48.663 INFO:teuthology.orchestra.run.smithi160.stdout:(3/24): libcephfs2-17.2.7-1629.geb16b97b.el9.x8 5.3 MB/s | 636 kB 00:00 2024-09-14T03:33:48.763 INFO:teuthology.orchestra.run.smithi160.stdout:(4/24): libradosstriper1-17.2.7-1629.geb16b97b. 4.5 MB/s | 458 kB 00:00 2024-09-14T03:33:49.180 INFO:teuthology.orchestra.run.smithi160.stdout:(5/24): ceph-common-17.2.7-1629.geb16b97b.el9.x 19 MB/s | 23 MB 00:01 2024-09-14T03:33:49.339 INFO:teuthology.orchestra.run.smithi160.stdout:(6/24): ceph-radosgw-17.2.7-1629.geb16b97b.el9. 11 MB/s | 14 MB 00:01 2024-09-14T03:33:49.414 INFO:teuthology.orchestra.run.smithi160.stdout:(7/24): librgw2-17.2.7-1629.geb16b97b.el9.x86_6 7.6 MB/s | 4.9 MB 00:00 2024-09-14T03:33:49.439 INFO:teuthology.orchestra.run.smithi160.stdout:(8/24): python3-ceph-common-17.2.7-1629.geb16b9 1.2 MB/s | 120 kB 00:00 2024-09-14T03:33:49.442 INFO:teuthology.orchestra.run.smithi114.stdout:(1/24): ceph-base-17.2.7-1629.geb16b97b.el9.x86 6.6 MB/s | 5.0 MB 00:00 2024-09-14T03:33:49.464 INFO:teuthology.orchestra.run.smithi160.stdout:(9/24): python3-cephfs-17.2.7-1629.geb16b97b.el 3.0 MB/s | 157 kB 00:00 2024-09-14T03:33:49.498 INFO:teuthology.orchestra.run.smithi160.stdout:(10/24): python3-rados-17.2.7-1629.geb16b97b.el 5.3 MB/s | 320 kB 00:00 2024-09-14T03:33:49.518 INFO:teuthology.orchestra.run.smithi114.stdout:(2/24): ceph-selinux-17.2.7-1629.geb16b97b.el9. 336 kB/s | 25 kB 00:00 2024-09-14T03:33:49.540 INFO:teuthology.orchestra.run.smithi160.stdout:(11/24): python3-rbd-17.2.7-1629.geb16b97b.el9. 3.8 MB/s | 296 kB 00:00 2024-09-14T03:33:49.565 INFO:teuthology.orchestra.run.smithi160.stdout:(12/24): python3-ceph-argparse-17.2.7-1629.geb1 117 kB/s | 45 kB 00:00 2024-09-14T03:33:49.591 INFO:teuthology.orchestra.run.smithi160.stdout:(13/24): python3-rgw-17.2.7-1629.geb16b97b.el9. 1.0 MB/s | 99 kB 00:00 2024-09-14T03:33:49.609 INFO:teuthology.orchestra.run.smithi114.stdout:(3/24): libcephfs2-17.2.7-1629.geb16b97b.el9.x8 6.8 MB/s | 636 kB 00:00 2024-09-14T03:33:49.760 INFO:teuthology.orchestra.run.smithi114.stdout:(4/24): ceph-radosgw-17.2.7-1629.geb16b97b.el9. 14 MB/s | 14 MB 00:01 2024-09-14T03:33:49.785 INFO:teuthology.orchestra.run.smithi114.stdout:(5/24): libradosstriper1-17.2.7-1629.geb16b97b. 2.5 MB/s | 458 kB 00:00 2024-09-14T03:33:49.835 INFO:teuthology.orchestra.run.smithi114.stdout:(6/24): python3-ceph-argparse-17.2.7-1629.geb16 905 kB/s | 45 kB 00:00 2024-09-14T03:33:49.894 INFO:teuthology.orchestra.run.smithi114.stdout:(7/24): python3-ceph-common-17.2.7-1629.geb16b9 2.0 MB/s | 120 kB 00:00 2024-09-14T03:33:49.986 INFO:teuthology.orchestra.run.smithi114.stdout:(8/24): librgw2-17.2.7-1629.geb16b97b.el9.x86_6 22 MB/s | 4.9 MB 00:00 2024-09-14T03:33:50.011 INFO:teuthology.orchestra.run.smithi114.stdout:(9/24): python3-cephfs-17.2.7-1629.geb16b97b.el 1.3 MB/s | 157 kB 00:00 2024-09-14T03:33:50.037 INFO:teuthology.orchestra.run.smithi114.stdout:(10/24): python3-rados-17.2.7-1629.geb16b97b.el 6.2 MB/s | 320 kB 00:00 2024-09-14T03:33:50.070 INFO:teuthology.orchestra.run.smithi114.stdout:(11/24): python3-rbd-17.2.7-1629.geb16b97b.el9. 4.9 MB/s | 296 kB 00:00 2024-09-14T03:33:50.095 INFO:teuthology.orchestra.run.smithi114.stdout:(12/24): python3-rgw-17.2.7-1629.geb16b97b.el9. 1.7 MB/s | 99 kB 00:00 2024-09-14T03:33:50.183 INFO:teuthology.orchestra.run.smithi160.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 73 kB/s | 45 kB 00:00 2024-09-14T03:33:50.291 INFO:teuthology.orchestra.run.smithi160.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 213 kB/s | 160 kB 00:00 2024-09-14T03:33:50.355 INFO:teuthology.orchestra.run.smithi114.stdout:(13/24): ceph-common-17.2.7-1629.geb16b97b.el9. 14 MB/s | 23 MB 00:01 2024-09-14T03:33:50.371 INFO:teuthology.orchestra.run.smithi114.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 164 kB/s | 45 kB 00:00 2024-09-14T03:33:50.421 INFO:teuthology.orchestra.run.smithi114.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 457 kB/s | 160 kB 00:00 2024-09-14T03:33:50.450 INFO:teuthology.orchestra.run.smithi160.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 771 kB/s | 662 kB 00:00 2024-09-14T03:33:50.497 INFO:teuthology.orchestra.run.smithi114.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 662 kB 00:00 2024-09-14T03:33:50.522 INFO:teuthology.orchestra.run.smithi114.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-14T03:33:50.539 INFO:teuthology.orchestra.run.smithi114.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 360 kB/s | 42 kB 00:00 2024-09-14T03:33:50.589 INFO:teuthology.orchestra.run.smithi114.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-09-14T03:33:50.600 INFO:teuthology.orchestra.run.smithi160.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-14T03:33:50.614 INFO:teuthology.orchestra.run.smithi114.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-14T03:33:50.634 INFO:teuthology.orchestra.run.smithi160.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-14T03:33:50.640 INFO:teuthology.orchestra.run.smithi114.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 780 kB/s | 67 kB 00:00 2024-09-14T03:33:50.668 INFO:teuthology.orchestra.run.smithi160.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-14T03:33:50.765 INFO:teuthology.orchestra.run.smithi114.stdout:(22/24): librados2-17.2.7-1629.geb16b97b.el9.x8 21 MB/s | 3.1 MB 00:00 2024-09-14T03:33:50.801 INFO:teuthology.orchestra.run.smithi160.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-14T03:33:50.815 INFO:teuthology.orchestra.run.smithi114.stdout:(23/24): thrift-0.15.0-2.el9.x86_64.rpm 7.1 MB/s | 1.6 MB 00:00 2024-09-14T03:33:50.865 INFO:teuthology.orchestra.run.smithi114.stdout:(24/24): librbd1-17.2.7-1629.geb16b97b.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-09-14T03:33:50.870 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:33:50.870 INFO:teuthology.orchestra.run.smithi114.stdout:Total 25 MB/s | 58 MB 00:02 2024-09-14T03:33:50.928 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:33:50.960 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:33:50.960 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:33:50.976 INFO:teuthology.orchestra.run.smithi160.stdout:(21/24): python3-prettytable-0.7.2-27.el9.noarc 62 kB/s | 42 kB 00:00 2024-09-14T03:33:51.035 INFO:teuthology.orchestra.run.smithi160.stdout:(22/24): librados2-17.2.7-1629.geb16b97b.el9.x8 13 MB/s | 3.1 MB 00:00 2024-09-14T03:33:51.060 INFO:teuthology.orchestra.run.smithi160.stdout:(23/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 333 kB/s | 292 kB 00:00 2024-09-14T03:33:51.235 INFO:teuthology.orchestra.run.smithi160.stdout:(24/24): librbd1-17.2.7-1629.geb16b97b.el9.x86_ 11 MB/s | 2.9 MB 00:00 2024-09-14T03:33:51.240 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:33:51.240 INFO:teuthology.orchestra.run.smithi160.stdout:Total 17 MB/s | 58 MB 00:03 2024-09-14T03:33:51.306 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:33:51.339 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:33:51.339 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:33:51.439 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:33:51.440 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:33:51.817 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:33:51.818 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:33:52.644 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:33:52.970 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-14T03:33:53.031 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:33:53.068 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:17.2.7-1629.geb16b97b.el9.x86_64 2/26 2024-09-14T03:33:53.143 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:17.2.7-1629.geb16b97b.el9.x86_64 2/26 2024-09-14T03:33:53.223 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:17.2.7-1629.geb16b97b.el9.x86_64 3/26 2024-09-14T03:33:53.269 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:17.2.7-1629.geb16b97b.el9.x86_64 4/26 2024-09-14T03:33:53.341 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:17.2.7-1629.geb16b97b.el9.x86_64 4/26 2024-09-14T03:33:53.395 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-14T03:33:53.423 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-14T03:33:53.424 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-14T03:33:53.487 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librados2-2:17.2.7-1629.geb16b97b.el9.x86_64 2/26 2024-09-14T03:33:53.588 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:17.2.7-1629.geb16b97b.el9.x86_64 2/26 2024-09-14T03:33:53.674 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rados-2:17.2.7-1629.geb16b97b.el9.x86_64 3/26 2024-09-14T03:33:53.697 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs2-2:17.2.7-1629.geb16b97b.el9.x86_64 4/26 2024-09-14T03:33:53.770 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs2-2:17.2.7-1629.geb16b97b.el9.x86_64 4/26 2024-09-14T03:33:53.817 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-14T03:33:53.852 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-14T03:33:53.861 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-14T03:33:53.888 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:17.2.7-1629.geb16b97b.el9.x86_64 8/26 2024-09-14T03:33:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:17.2.7-1629.geb16b97b.el9.x86_64 8/26 2024-09-14T03:33:53.980 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-14T03:33:54.024 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:17.2.7-1629.geb16b97b.el9. 10/26 2024-09-14T03:33:54.059 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:17.2.7-1629.geb16b97b.el9.x86_64 11/26 2024-09-14T03:33:54.123 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:17.2.7-1629.geb16b97b.el9.x86_64 12/26 2024-09-14T03:33:54.150 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:17.2.7-1629.geb16b97b.el9.x86_6 13/26 2024-09-14T03:33:54.216 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1629.geb16b97b.el9.x86_6 13/26 2024-09-14T03:33:54.267 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-14T03:33:54.293 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-14T03:33:54.300 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-14T03:33:54.317 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librgw2-2:17.2.7-1629.geb16b97b.el9.x86_64 8/26 2024-09-14T03:33:54.333 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-14T03:33:54.380 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librgw2-2:17.2.7-1629.geb16b97b.el9.x86_64 8/26 2024-09-14T03:33:54.417 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-14T03:33:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-argparse-2:17.2.7-1629.geb16b97b.el9. 10/26 2024-09-14T03:33:54.521 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cephfs-2:17.2.7-1629.geb16b97b.el9.x86_64 11/26 2024-09-14T03:33:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rgw-2:17.2.7-1629.geb16b97b.el9.x86_64 12/26 2024-09-14T03:33:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libradosstriper1-2:17.2.7-1629.geb16b97b.el9.x86_6 13/26 2024-09-14T03:33:54.634 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-14T03:33:54.671 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:17.2.7-1629.geb16b97b.el9.x86_64 18/26 2024-09-14T03:33:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1629.geb16b97b.el9.x86_6 13/26 2024-09-14T03:33:54.726 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-14T03:33:54.760 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:17.2.7-1629.geb16b97b.el9.x86_64 18/26 2024-09-14T03:33:54.761 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-14T03:33:54.794 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-14T03:33:54.834 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:17.2.7-1629.geb16b97b.el9.x86_64 19/26 2024-09-14T03:33:54.865 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:17.2.7-1629.geb16b97b.el9.x8 20/26 2024-09-14T03:33:55.089 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-14T03:33:55.132 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librbd1-2:17.2.7-1629.geb16b97b.el9.x86_64 18/26 2024-09-14T03:33:55.221 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:17.2.7-1629.geb16b97b.el9.x86_64 18/26 2024-09-14T03:33:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rbd-2:17.2.7-1629.geb16b97b.el9.x86_64 19/26 2024-09-14T03:33:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-common-2:17.2.7-1629.geb16b97b.el9.x8 20/26 2024-09-14T03:33:57.295 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 21/26 2024-09-14T03:33:57.339 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 21/26 2024-09-14T03:33:57.885 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 21/26 2024-09-14T03:33:58.017 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 21/26 2024-09-14T03:33:58.055 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 21/26 2024-09-14T03:33:58.082 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:17.2.7-1629.geb16b97b.el9.x86_64 22/26 2024-09-14T03:33:58.188 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:17.2.7-1629.geb16b97b.el9.x86_64 22/26 2024-09-14T03:33:58.188 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-14T03:33:58.188 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-14T03:33:58.188 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:33:58.225 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:17.2.7-1629.geb16b97b.el9.x86_64 23/26 2024-09-14T03:33:58.595 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 21/26 2024-09-14T03:33:58.633 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-base-2:17.2.7-1629.geb16b97b.el9.x86_64 22/26 2024-09-14T03:33:58.730 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-base-2:17.2.7-1629.geb16b97b.el9.x86_64 22/26 2024-09-14T03:33:58.730 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-14T03:33:58.730 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-14T03:33:58.730 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:33:58.768 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-selinux-2:17.2.7-1629.geb16b97b.el9.x86_64 23/26 2024-09-14T03:34:08.075 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1629.geb16b97b.el9.x86_64 23/26 2024-09-14T03:34:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-09-14T03:34:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-09-14T03:34:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-09-14T03:34:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-09-14T03:34:08.075 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-09-14T03:34:08.076 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-09-14T03:34:08.076 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-09-14T03:34:08.076 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:08.562 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1629.geb16b97b.el9.x86_64 23/26 2024-09-14T03:34:08.562 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /sys 2024-09-14T03:34:08.562 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /proc 2024-09-14T03:34:08.562 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /mnt 2024-09-14T03:34:08.563 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /var/tmp 2024-09-14T03:34:08.563 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /home 2024-09-14T03:34:08.563 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /root 2024-09-14T03:34:08.563 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /tmp 2024-09-14T03:34:08.563 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:17.2.7-1629.geb16b97b.el9.x86_64 24/26 2024-09-14T03:34:09.469 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1629.geb16b97b.el9.x86_64 24/26 2024-09-14T03:34:09.470 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:09.470 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-14T03:34:09.470 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T03:34:09.470 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T03:34:09.470 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:09.471 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-14T03:34:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-14T03:34:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-14T03:34:09.885 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-radosgw-2:17.2.7-1629.geb16b97b.el9.x86_64 24/26 2024-09-14T03:34:09.955 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1629.geb16b97b.el9.x86_64 24/26 2024-09-14T03:34:09.955 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:09.956 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-14T03:34:09.956 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T03:34:09.956 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T03:34:09.956 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:09.957 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-14T03:34:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-14T03:34:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-14T03:34:10.817 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-14T03:34:10.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:17.2.7-1629.geb16b97b.el9.x86_64 1/26 2024-09-14T03:34:10.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 2/26 2024-09-14T03:34:10.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:17.2.7-1629.geb16b97b.el9.x86_64 3/26 2024-09-14T03:34:10.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:17.2.7-1629.geb16b97b.el9.x86_64 4/26 2024-09-14T03:34:10.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:17.2.7-1629.geb16b97b.el9.x86_64 5/26 2024-09-14T03:34:10.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:17.2.7-1629.geb16b97b.el9.x86_6 6/26 2024-09-14T03:34:10.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:17.2.7-1629.geb16b97b.el9.x86_64 7/26 2024-09-14T03:34:10.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1629.geb16b97b.el9. 8/26 2024-09-14T03:34:10.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:17.2.7-1629.geb16b97b.el9.x8 9/26 2024-09-14T03:34:10.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:17.2.7-1629.geb16b97b.el9.x86_64 10/26 2024-09-14T03:34:10.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:17.2.7-1629.geb16b97b.el9.x86_64 11/26 2024-09-14T03:34:10.820 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:17.2.7-1629.geb16b97b.el9.x86_64 12/26 2024-09-14T03:34:10.820 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:17.2.7-1629.geb16b97b.el9.x86_64 13/26 2024-09-14T03:34:10.820 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-14T03:34:10.820 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-14T03:34:10.820 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-14T03:34:10.820 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-14T03:34:10.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-14T03:34:10.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-14T03:34:10.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-14T03:34:10.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-14T03:34:10.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-14T03:34:10.821 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:17.2.7-1629.geb16b97b.el9.x86_64 23/26 2024-09-14T03:34:10.822 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-14T03:34:10.822 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:17.2.7-1629.geb16b97b.el9.x86_64 25/26 2024-09-14T03:34:11.340 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-14T03:34:11.340 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-base-2:17.2.7-1629.geb16b97b.el9.x86_64 1/26 2024-09-14T03:34:11.340 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 2/26 2024-09-14T03:34:11.340 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-radosgw-2:17.2.7-1629.geb16b97b.el9.x86_64 3/26 2024-09-14T03:34:11.340 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-selinux-2:17.2.7-1629.geb16b97b.el9.x86_64 4/26 2024-09-14T03:34:11.340 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs2-2:17.2.7-1629.geb16b97b.el9.x86_64 5/26 2024-09-14T03:34:11.340 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libradosstriper1-2:17.2.7-1629.geb16b97b.el9.x86_6 6/26 2024-09-14T03:34:11.340 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librgw2-2:17.2.7-1629.geb16b97b.el9.x86_64 7/26 2024-09-14T03:34:11.340 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1629.geb16b97b.el9. 8/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-common-2:17.2.7-1629.geb16b97b.el9.x8 9/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cephfs-2:17.2.7-1629.geb16b97b.el9.x86_64 10/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rados-2:17.2.7-1629.geb16b97b.el9.x86_64 11/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rbd-2:17.2.7-1629.geb16b97b.el9.x86_64 12/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rgw-2:17.2.7-1629.geb16b97b.el9.x86_64 13/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-14T03:34:11.341 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-14T03:34:11.342 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-14T03:34:11.342 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-14T03:34:11.342 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:17.2.7-1629.geb16b97b.el9.x86_64 23/26 2024-09-14T03:34:11.342 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-14T03:34:11.342 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:17.2.7-1629.geb16b97b.el9.x86_64 25/26 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.557 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.558 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-14T03:34:11.559 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.559 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.559 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:11.559 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-14T03:34:11.559 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:11.559 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:34:11.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-test 2024-09-14T03:34:12.157 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-14T03:34:12.157 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:12.157 INFO:teuthology.orchestra.run.smithi160.stdout:Upgraded: 2024-09-14T03:34:12.157 INFO:teuthology.orchestra.run.smithi160.stdout: librados2-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.157 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.157 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:34:12.157 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.157 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.157 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.157 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.158 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.159 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.159 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-14T03:34:12.159 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.159 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.159 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:12.159 INFO:teuthology.orchestra.run.smithi160.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-14T03:34:12.159 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:12.159 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:34:12.492 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:26 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:34:12.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-test 2024-09-14T03:34:12.629 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:34:12.630 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:12.630 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-14T03:34:12.630 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:12.630 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:34:12.631 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 48 M 2024-09-14T03:34:12.631 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-14T03:34:12.631 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 163 k 2024-09-14T03:34:12.631 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-14T03:34:12.631 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-14T03:34:12.631 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:12.632 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:34:12.632 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:12.632 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-09-14T03:34:12.632 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:12.633 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 48 M 2024-09-14T03:34:12.634 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 193 M 2024-09-14T03:34:12.634 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:34:12.806 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-14T03:34:12.839 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-14T03:34:12.898 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): libcephsqlite-17.2.7-1629.geb16b97b.el9. 749 kB/s | 163 kB 00:00 2024-09-14T03:34:13.092 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:28 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:34:13.234 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 48 M 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 163 k 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:34:13.235 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:13.236 INFO:teuthology.orchestra.run.smithi160.stdout:Install 4 Packages 2024-09-14T03:34:13.236 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:13.236 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 48 M 2024-09-14T03:34:13.236 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 193 M 2024-09-14T03:34:13.236 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:34:13.494 INFO:teuthology.orchestra.run.smithi160.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-14T03:34:13.544 INFO:teuthology.orchestra.run.smithi160.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-14T03:34:13.604 INFO:teuthology.orchestra.run.smithi160.stdout:(3/4): libcephsqlite-17.2.7-1629.geb16b97b.el9. 476 kB/s | 163 kB 00:00 2024-09-14T03:34:14.240 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): ceph-test-17.2.7-1629.geb16b97b.el9.x86_ 31 MB/s | 48 MB 00:01 2024-09-14T03:34:14.242 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:14.242 INFO:teuthology.orchestra.run.smithi114.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-14T03:34:14.270 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:34:14.289 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:34:14.289 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:34:14.654 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:34:14.655 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:34:15.170 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:34:15.237 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-14T03:34:15.281 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-14T03:34:15.308 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:17.2.7-1629.geb16b97b.el9.x86_64 3/4 2024-09-14T03:34:15.445 INFO:teuthology.orchestra.run.smithi160.stdout:(4/4): ceph-test-17.2.7-1629.geb16b97b.el9.x86_ 22 MB/s | 48 MB 00:02 2024-09-14T03:34:15.449 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:15.449 INFO:teuthology.orchestra.run.smithi160.stdout:Total 22 MB/s | 48 MB 00:02 2024-09-14T03:34:15.475 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:34:15.494 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:34:15.494 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:34:15.862 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:34:15.862 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:34:16.392 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:34:16.474 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-14T03:34:16.524 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-14T03:34:16.554 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephsqlite-2:17.2.7-1629.geb16b97b.el9.x86_64 3/4 2024-09-14T03:34:19.671 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1629.geb16b97b.el9.x86_64 3/4 2024-09-14T03:34:19.751 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:17.2.7-1629.geb16b97b.el9.x86_64 4/4 2024-09-14T03:34:20.962 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1629.geb16b97b.el9.x86_64 3/4 2024-09-14T03:34:21.035 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-test-2:17.2.7-1629.geb16b97b.el9.x86_64 4/4 2024-09-14T03:34:21.901 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-test-2:17.2.7-1629.geb16b97b.el9.x86_64 4/4 2024-09-14T03:34:21.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:17.2.7-1629.geb16b97b.el9.x86_64 1/4 2024-09-14T03:34:21.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:17.2.7-1629.geb16b97b.el9.x86_64 2/4 2024-09-14T03:34:21.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-14T03:34:22.229 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-14T03:34:22.229 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:22.229 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:34:22.229 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:22.229 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:22.229 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-14T03:34:22.229 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-14T03:34:22.229 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:22.229 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:34:22.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph 2024-09-14T03:34:23.128 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:37 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:34:23.264 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:34:23.266 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:23.266 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-14T03:34:23.266 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 6.5 k 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 2.1 M 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 1.4 M 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 240 k 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 3.9 M 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 15 M 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-14T03:34:23.267 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-14T03:34:23.268 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-14T03:34:23.269 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:23.270 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:34:23.270 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:23.270 INFO:teuthology.orchestra.run.smithi114.stdout:Install 34 Packages 2024-09-14T03:34:23.270 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:23.270 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 27 M 2024-09-14T03:34:23.270 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 96 M 2024-09-14T03:34:23.270 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:34:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-test-2:17.2.7-1629.geb16b97b.el9.x86_64 4/4 2024-09-14T03:34:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-test-2:17.2.7-1629.geb16b97b.el9.x86_64 1/4 2024-09-14T03:34:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephsqlite-2:17.2.7-1629.geb16b97b.el9.x86_64 2/4 2024-09-14T03:34:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-14T03:34:23.626 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-14T03:34:23.626 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:23.626 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:34:23.626 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:23.626 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:23.626 INFO:teuthology.orchestra.run.smithi160.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-14T03:34:23.626 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-14T03:34:23.626 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:23.626 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:34:23.844 INFO:teuthology.orchestra.run.smithi114.stdout:(1/34): ceph-17.2.7-1629.geb16b97b.el9.x86_64.r 51 kB/s | 6.5 kB 00:00 2024-09-14T03:34:23.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph 2024-09-14T03:34:24.027 INFO:teuthology.orchestra.run.smithi114.stdout:(2/34): ceph-mgr-17.2.7-1629.geb16b97b.el9.x86_ 4.5 MB/s | 1.4 MB 00:00 2024-09-14T03:34:24.069 INFO:teuthology.orchestra.run.smithi114.stdout:(3/34): ceph-mds-17.2.7-1629.geb16b97b.el9.x86_ 5.9 MB/s | 2.1 MB 00:00 2024-09-14T03:34:24.170 INFO:teuthology.orchestra.run.smithi114.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1629.geb16 2.3 MB/s | 240 kB 00:00 2024-09-14T03:34:24.372 INFO:teuthology.orchestra.run.smithi114.stdout:(5/34): ceph-mon-17.2.7-1629.geb16b97b.el9.x86_ 7.6 MB/s | 3.9 MB 00:00 2024-09-14T03:34:24.387 INFO:teuthology.orchestra.run.smithi114.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-14T03:34:24.458 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:39 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:34:24.562 INFO:teuthology.orchestra.run.smithi114.stdout:(7/34): ceph-osd-17.2.7-1629.geb16b97b.el9.x86_ 29 MB/s | 15 MB 00:00 2024-09-14T03:34:24.587 INFO:teuthology.orchestra.run.smithi114.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 532 kB/s | 106 kB 00:00 2024-09-14T03:34:24.592 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:34:24.594 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:24.594 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-14T03:34:24.594 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:24.594 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:34:24.594 INFO:teuthology.orchestra.run.smithi160.stdout: ceph x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 6.5 k 2024-09-14T03:34:24.594 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-14T03:34:24.594 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 2.1 M 2024-09-14T03:34:24.594 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 1.4 M 2024-09-14T03:34:24.594 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 240 k 2024-09-14T03:34:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 3.9 M 2024-09-14T03:34:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 15 M 2024-09-14T03:34:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-14T03:34:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-14T03:34:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-14T03:34:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-14T03:34:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-14T03:34:24.595 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-14T03:34:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-14T03:34:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections 2024-09-14T03:34:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-14T03:34:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-14T03:34:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-14T03:34:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-14T03:34:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-14T03:34:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-14T03:34:24.596 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-14T03:34:24.597 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-14T03:34:24.598 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-14T03:34:24.598 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:24.598 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:34:24.598 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:24.598 INFO:teuthology.orchestra.run.smithi160.stdout:Install 34 Packages 2024-09-14T03:34:24.598 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:24.598 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 27 M 2024-09-14T03:34:24.599 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 96 M 2024-09-14T03:34:24.599 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:34:24.638 INFO:teuthology.orchestra.run.smithi114.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-14T03:34:24.696 INFO:teuthology.orchestra.run.smithi114.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-14T03:34:24.730 INFO:teuthology.orchestra.run.smithi114.stdout:(11/34): python3-pycparser-2.20-6.el9.noarch.rp 808 kB/s | 135 kB 00:00 2024-09-14T03:34:24.763 INFO:teuthology.orchestra.run.smithi114.stdout:(12/34): python3-cryptography-36.0.1-4.el9.x86_ 3.1 MB/s | 1.2 MB 00:00 2024-09-14T03:34:25.013 INFO:teuthology.orchestra.run.smithi114.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 122 kB/s | 35 kB 00:00 2024-09-14T03:34:25.140 INFO:teuthology.orchestra.run.smithi114.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 205 kB/s | 77 kB 00:00 2024-09-14T03:34:25.206 INFO:teuthology.orchestra.run.smithi114.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 265 kB/s | 51 kB 00:00 2024-09-14T03:34:25.240 INFO:teuthology.orchestra.run.smithi114.stdout:(16/34): python3-mako-1.1.4-6.el9.noarch.rpm 317 kB/s | 172 kB 00:00 2024-09-14T03:34:25.265 INFO:teuthology.orchestra.run.smithi114.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 331 kB/s | 42 kB 00:00 2024-09-14T03:34:25.286 INFO:teuthology.orchestra.run.smithi160.stdout:(1/34): ceph-17.2.7-1629.geb16b97b.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-14T03:34:25.299 INFO:teuthology.orchestra.run.smithi114.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 471 kB/s | 43 kB 00:00 2024-09-14T03:34:25.332 INFO:teuthology.orchestra.run.smithi114.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 439 kB/s | 11 kB 00:00 2024-09-14T03:34:25.358 INFO:teuthology.orchestra.run.smithi114.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-14T03:34:25.383 INFO:teuthology.orchestra.run.smithi114.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 364 kB/s | 18 kB 00:00 2024-09-14T03:34:25.408 INFO:teuthology.orchestra.run.smithi114.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-09-14T03:34:25.434 INFO:teuthology.orchestra.run.smithi114.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-14T03:34:25.459 INFO:teuthology.orchestra.run.smithi114.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-14T03:34:25.478 INFO:teuthology.orchestra.run.smithi160.stdout:(2/34): ceph-mgr-17.2.7-1629.geb16b97b.el9.x86_ 4.5 MB/s | 1.4 MB 00:00 2024-09-14T03:34:25.484 INFO:teuthology.orchestra.run.smithi114.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-09-14T03:34:25.509 INFO:teuthology.orchestra.run.smithi114.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-14T03:34:25.519 INFO:teuthology.orchestra.run.smithi160.stdout:(3/34): ceph-mds-17.2.7-1629.geb16b97b.el9.x86_ 5.8 MB/s | 2.1 MB 00:00 2024-09-14T03:34:25.535 INFO:teuthology.orchestra.run.smithi114.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 656 kB/s | 16 kB 00:00 2024-09-14T03:34:25.560 INFO:teuthology.orchestra.run.smithi114.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-14T03:34:25.577 INFO:teuthology.orchestra.run.smithi114.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-14T03:34:25.602 INFO:teuthology.orchestra.run.smithi114.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 853 kB/s | 36 kB 00:00 2024-09-14T03:34:25.627 INFO:teuthology.orchestra.run.smithi114.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-14T03:34:25.637 INFO:teuthology.orchestra.run.smithi160.stdout:(4/34): ceph-mon-17.2.7-1629.geb16b97b.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-09-14T03:34:25.653 INFO:teuthology.orchestra.run.smithi114.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-14T03:34:25.662 INFO:teuthology.orchestra.run.smithi160.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1629.geb16 1.7 MB/s | 240 kB 00:00 2024-09-14T03:34:25.678 INFO:teuthology.orchestra.run.smithi114.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-14T03:34:25.703 INFO:teuthology.orchestra.run.smithi114.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-14T03:34:25.708 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:25.708 INFO:teuthology.orchestra.run.smithi114.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-14T03:34:25.779 INFO:teuthology.orchestra.run.smithi160.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-14T03:34:25.836 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:34:25.837 INFO:teuthology.orchestra.run.smithi160.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-14T03:34:25.874 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:34:25.874 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:34:25.879 INFO:teuthology.orchestra.run.smithi160.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-14T03:34:26.021 INFO:teuthology.orchestra.run.smithi160.stdout:(9/34): ceph-osd-17.2.7-1629.geb16b97b.el9.x86_ 28 MB/s | 15 MB 00:00 2024-09-14T03:34:26.047 INFO:teuthology.orchestra.run.smithi160.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 748 kB/s | 125 kB 00:00 2024-09-14T03:34:26.080 INFO:teuthology.orchestra.run.smithi160.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.2 MB 00:00 2024-09-14T03:34:26.222 INFO:teuthology.orchestra.run.smithi160.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-14T03:34:26.242 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:34:26.243 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:34:26.372 INFO:teuthology.orchestra.run.smithi160.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 119 kB/s | 35 kB 00:00 2024-09-14T03:34:26.572 INFO:teuthology.orchestra.run.smithi160.stdout:(14/34): python3-pytz-2021.1-5.el9.noarch.rpm 255 kB/s | 51 kB 00:00 2024-09-14T03:34:26.623 INFO:teuthology.orchestra.run.smithi160.stdout:(15/34): python3-mako-1.1.4-6.el9.noarch.rpm 299 kB/s | 172 kB 00:00 2024-09-14T03:34:26.673 INFO:teuthology.orchestra.run.smithi160.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 867 kB/s | 43 kB 00:00 2024-09-14T03:34:26.707 INFO:teuthology.orchestra.run.smithi160.stdout:(17/34): python3-packaging-20.9-5.el9.noarch.rp 159 kB/s | 77 kB 00:00 2024-09-14T03:34:26.732 INFO:teuthology.orchestra.run.smithi160.stdout:(18/34): python3-toml-0.10.2-6.el9.noarch.rpm 261 kB/s | 42 kB 00:00 2024-09-14T03:34:26.774 INFO:teuthology.orchestra.run.smithi160.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-14T03:34:26.799 INFO:teuthology.orchestra.run.smithi160.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 768 kB/s | 18 kB 00:00 2024-09-14T03:34:26.825 INFO:teuthology.orchestra.run.smithi160.stdout:(21/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 126 kB/s | 11 kB 00:00 2024-09-14T03:34:26.850 INFO:teuthology.orchestra.run.smithi160.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-14T03:34:26.875 INFO:teuthology.orchestra.run.smithi160.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-14T03:34:26.900 INFO:teuthology.orchestra.run.smithi160.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-14T03:34:26.926 INFO:teuthology.orchestra.run.smithi160.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-09-14T03:34:26.951 INFO:teuthology.orchestra.run.smithi160.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-14T03:34:26.976 INFO:teuthology.orchestra.run.smithi160.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-14T03:34:27.002 INFO:teuthology.orchestra.run.smithi160.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-14T03:34:27.027 INFO:teuthology.orchestra.run.smithi160.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-14T03:34:27.052 INFO:teuthology.orchestra.run.smithi160.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-14T03:34:27.078 INFO:teuthology.orchestra.run.smithi160.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-14T03:34:27.103 INFO:teuthology.orchestra.run.smithi160.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 807 kB/s | 20 kB 00:00 2024-09-14T03:34:27.128 INFO:teuthology.orchestra.run.smithi160.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-14T03:34:27.153 INFO:teuthology.orchestra.run.smithi160.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-14T03:34:27.157 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:27.158 INFO:teuthology.orchestra.run.smithi160.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-14T03:34:27.291 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:34:27.329 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:34:27.329 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:34:27.473 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:34:27.540 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-14T03:34:27.580 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-14T03:34:27.613 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-14T03:34:27.706 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-14T03:34:27.707 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:34:27.707 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:34:27.792 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-14T03:34:27.832 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-14T03:34:27.867 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-14T03:34:27.897 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-14T03:34:27.931 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-14T03:34:27.965 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-14T03:34:28.005 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-14T03:34:28.049 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-14T03:34:28.091 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-14T03:34:28.140 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-14T03:34:28.181 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-14T03:34:28.253 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-14T03:34:28.401 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-14T03:34:28.479 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-14T03:34:28.533 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-14T03:34:28.579 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-14T03:34:28.628 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-14T03:34:28.684 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-14T03:34:28.879 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-14T03:34:28.933 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-14T03:34:28.938 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:34:29.003 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-14T03:34:29.006 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-14T03:34:29.046 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-14T03:34:29.088 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-14T03:34:29.091 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-14T03:34:29.133 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-14T03:34:29.188 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-14T03:34:29.249 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-14T03:34:29.274 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-14T03:34:29.316 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-14T03:34:29.351 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-14T03:34:29.389 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-14T03:34:29.405 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1629.geb16b97b.el9. 29/34 2024-09-14T03:34:29.432 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-14T03:34:29.434 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:17.2.7-1629.geb16b97b.el9.x86_64 30/34 2024-09-14T03:34:29.460 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1629.geb16b97b.el9.x86_64 30/34 2024-09-14T03:34:29.461 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:29.461 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-14T03:34:29.461 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T03:34:29.462 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T03:34:29.462 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:29.465 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-14T03:34:29.506 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-14T03:34:29.574 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-14T03:34:29.609 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-14T03:34:29.657 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-14T03:34:29.697 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-14T03:34:29.770 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-14T03:34:29.919 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-14T03:34:29.999 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-14T03:34:30.060 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-14T03:34:30.099 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-14T03:34:30.156 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-14T03:34:30.211 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-14T03:34:30.399 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-14T03:34:30.453 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-14T03:34:30.531 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-14T03:34:30.619 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-14T03:34:30.668 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-14T03:34:30.783 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-14T03:34:30.828 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:17.2.7-1629.geb16b97b.el9.x86_64 31/34 2024-09-14T03:34:30.857 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:17.2.7-1629.geb16b97b.el9.x86_64 31/34 2024-09-14T03:34:30.858 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:30.858 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-14T03:34:30.858 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T03:34:30.858 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T03:34:30.858 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:30.942 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1629.geb16b97b.el9. 29/34 2024-09-14T03:34:30.970 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-2:17.2.7-1629.geb16b97b.el9.x86_64 30/34 2024-09-14T03:34:30.996 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1629.geb16b97b.el9.x86_64 30/34 2024-09-14T03:34:30.996 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:30.996 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-14T03:34:30.997 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T03:34:30.997 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T03:34:30.997 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:31.237 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:17.2.7-1629.geb16b97b.el9.x86_64 32/34 2024-09-14T03:34:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:17.2.7-1629.geb16b97b.el9.x86_64 32/34 2024-09-14T03:34:31.262 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:31.262 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-14T03:34:31.263 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T03:34:31.263 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T03:34:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:31.463 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:17.2.7-1629.geb16b97b.el9.x86_64 33/34 2024-09-14T03:34:31.487 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:17.2.7-1629.geb16b97b.el9.x86_64 33/34 2024-09-14T03:34:31.487 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:31.488 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-14T03:34:31.488 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T03:34:31.488 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T03:34:31.488 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:31.552 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:17.2.7-1629.geb16b97b.el9.x86_64 34/34 2024-09-14T03:34:32.347 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-osd-2:17.2.7-1629.geb16b97b.el9.x86_64 31/34 2024-09-14T03:34:32.376 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-osd-2:17.2.7-1629.geb16b97b.el9.x86_64 31/34 2024-09-14T03:34:32.376 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:32.376 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-14T03:34:32.376 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T03:34:32.377 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T03:34:32.377 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:32.856 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mon-2:17.2.7-1629.geb16b97b.el9.x86_64 32/34 2024-09-14T03:34:32.881 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mon-2:17.2.7-1629.geb16b97b.el9.x86_64 32/34 2024-09-14T03:34:32.881 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:32.881 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-14T03:34:32.881 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T03:34:32.881 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T03:34:32.881 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:33.090 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mds-2:17.2.7-1629.geb16b97b.el9.x86_64 33/34 2024-09-14T03:34:33.114 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mds-2:17.2.7-1629.geb16b97b.el9.x86_64 33/34 2024-09-14T03:34:33.114 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:33.114 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-14T03:34:33.114 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T03:34:33.115 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T03:34:33.115 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-2:17.2.7-1629.geb16b97b.el9.x86_64 34/34 2024-09-14T03:34:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:17.2.7-1629.geb16b97b.el9.x86_64 1/34 2024-09-14T03:34:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:17.2.7-1629.geb16b97b.el9.x86_64 2/34 2024-09-14T03:34:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:17.2.7-1629.geb16b97b.el9.x86_64 3/34 2024-09-14T03:34:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:17.2.7-1629.geb16b97b.el9.x86_64 4/34 2024-09-14T03:34:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:17.2.7-1629.geb16b97b.el9.x86_64 5/34 2024-09-14T03:34:33.135 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1629.geb16b97b.el9. 6/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-14T03:34:33.136 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-14T03:34:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-14T03:34:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-14T03:34:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-14T03:34:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-14T03:34:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-14T03:34:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-14T03:34:33.137 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-14T03:34:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-14T03:34:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-14T03:34:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-14T03:34:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-14T03:34:33.138 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-14T03:34:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-14T03:34:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-14T03:34:33.139 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-14T03:34:33.179 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-2:17.2.7-1629.geb16b97b.el9.x86_64 34/34 2024-09-14T03:34:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-14T03:34:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:34.039 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:34:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-14T03:34:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-14T03:34:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-14T03:34:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-14T03:34:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-14T03:34:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-14T03:34:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-14T03:34:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-14T03:34:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-14T03:34:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-14T03:34:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-14T03:34:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-14T03:34:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-14T03:34:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-14T03:34:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-14T03:34:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-14T03:34:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-09-14T03:34:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-14T03:34:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-14T03:34:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-14T03:34:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-14T03:34:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-14T03:34:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-14T03:34:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-14T03:34:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-14T03:34:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-14T03:34:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-14T03:34:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-14T03:34:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:34.047 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:34:34.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-base 2024-09-14T03:34:34.936 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-2:17.2.7-1629.geb16b97b.el9.x86_64 34/34 2024-09-14T03:34:34.936 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-2:17.2.7-1629.geb16b97b.el9.x86_64 1/34 2024-09-14T03:34:34.936 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mds-2:17.2.7-1629.geb16b97b.el9.x86_64 2/34 2024-09-14T03:34:34.936 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-2:17.2.7-1629.geb16b97b.el9.x86_64 3/34 2024-09-14T03:34:34.936 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mon-2:17.2.7-1629.geb16b97b.el9.x86_64 4/34 2024-09-14T03:34:34.936 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-osd-2:17.2.7-1629.geb16b97b.el9.x86_64 5/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1629.geb16b97b.el9. 6/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-14T03:34:34.937 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-14T03:34:34.938 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-14T03:34:34.939 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-14T03:34:34.939 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-14T03:34:34.939 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-14T03:34:34.973 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:48 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:34:35.064 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-base-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:34:35.105 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:34:35.106 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-14T03:34:35.107 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:34:35.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install cephadm 2024-09-14T03:34:35.735 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:49 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:34:35.776 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-14T03:34:35.777 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-14T03:34:35.778 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply-3.11-14.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:35.779 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:34:35.867 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:34:35.868 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:35.868 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-14T03:34:35.868 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:35.868 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:34:35.868 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 83 k 2024-09-14T03:34:35.868 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:35.868 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:34:35.869 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:35.869 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-14T03:34:35.869 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:35.869 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 83 k 2024-09-14T03:34:35.869 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 375 k 2024-09-14T03:34:35.869 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:34:36.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-base 2024-09-14T03:34:36.237 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm-17.2.7-1629.geb16b97b.el9.noarch.rpm 224 kB/s | 83 kB 00:00 2024-09-14T03:34:36.238 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:36.238 INFO:teuthology.orchestra.run.smithi114.stdout:Total 223 kB/s | 83 kB 00:00 2024-09-14T03:34:36.238 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:34:36.243 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:34:36.244 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:34:36.245 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:34:36.246 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:34:36.323 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:34:36.704 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 1/1 2024-09-14T03:34:36.708 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:51 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:34:36.791 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 1/1 2024-09-14T03:34:36.800 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-base-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:34:36.843 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:34:36.844 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-14T03:34:36.844 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:34:36.901 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install cephadm 2024-09-14T03:34:37.170 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 1/1 2024-09-14T03:34:37.437 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 1/1 2024-09-14T03:34:37.438 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:37.438 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:34:37.438 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:37.438 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:37.438 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:34:37.463 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:52 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:34:37.596 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:34:37.597 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:37.597 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-14T03:34:37.597 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:37.597 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:34:37.597 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 83 k 2024-09-14T03:34:37.597 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:37.597 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:34:37.598 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:37.598 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-14T03:34:37.598 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:37.598 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 83 k 2024-09-14T03:34:37.598 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 375 k 2024-09-14T03:34:37.598 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:34:37.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-immutable-object-cache 2024-09-14T03:34:37.789 INFO:teuthology.orchestra.run.smithi160.stdout:cephadm-17.2.7-1629.geb16b97b.el9.noarch.rpm 435 kB/s | 83 kB 00:00 2024-09-14T03:34:37.789 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:37.790 INFO:teuthology.orchestra.run.smithi160.stdout:Total 431 kB/s | 83 kB 00:00 2024-09-14T03:34:37.790 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:34:37.795 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:34:37.795 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:34:37.796 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:34:37.797 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:34:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:34:38.205 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:52 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:34:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 1/1 2024-09-14T03:34:38.339 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:34:38.339 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-14T03:34:38.339 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:38.339 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:34:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 137 k 2024-09-14T03:34:38.339 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:38.339 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:34:38.339 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:38.339 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-14T03:34:38.340 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:38.340 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 137 k 2024-09-14T03:34:38.340 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 379 k 2024-09-14T03:34:38.340 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:34:38.433 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 1/1 2024-09-14T03:34:38.543 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-immutable-object-cache-17.2.7-1629.geb16b9 677 kB/s | 137 kB 00:00 2024-09-14T03:34:38.544 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:38.544 INFO:teuthology.orchestra.run.smithi114.stdout:Total 672 kB/s | 137 kB 00:00 2024-09-14T03:34:38.544 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:34:38.552 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:34:38.552 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:34:38.608 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:34:38.609 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:34:38.776 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:34:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 1/1 2024-09-14T03:34:38.861 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1629.geb16b97b. 1/1 2024-09-14T03:34:38.886 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1629.geb16b97b. 1/1 2024-09-14T03:34:38.886 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:38.886 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-14T03:34:38.886 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 1/1 2024-09-14T03:34:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:39.033 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:34:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:39.033 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:34:39.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-immutable-object-cache 2024-09-14T03:34:39.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1629.geb16b97b. 1/1 2024-09-14T03:34:39.817 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:39.817 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:34:39.817 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:39.817 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:39.817 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:34:39.842 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:54 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:34:39.981 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 137 k 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 137 k 2024-09-14T03:34:39.982 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 379 k 2024-09-14T03:34:39.983 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:34:40.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr 2024-09-14T03:34:40.186 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-immutable-object-cache-17.2.7-1629.geb16b9 678 kB/s | 137 kB 00:00 2024-09-14T03:34:40.186 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:40.187 INFO:teuthology.orchestra.run.smithi160.stdout:Total 672 kB/s | 137 kB 00:00 2024-09-14T03:34:40.187 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:34:40.194 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:34:40.194 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:34:40.251 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:34:40.252 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:34:40.409 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:34:40.470 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1629.geb16b97b. 1/1 2024-09-14T03:34:40.495 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1629.geb16b97b. 1/1 2024-09-14T03:34:40.495 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:34:40.495 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-14T03:34:40.495 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:40.612 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:54 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:34:40.704 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-mgr-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:34:40.746 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:34:40.747 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-14T03:34:40.747 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:34:40.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-dashboard 2024-09-14T03:34:41.375 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:55 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:34:41.427 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1629.geb16b97b. 1/1 2024-09-14T03:34:41.427 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:41.427 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:34:41.427 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:34:41.427 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:41.428 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:34:41.508 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:34:41.509 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:41.509 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-14T03:34:41.509 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:41.509 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:34:41.510 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 3.4 M 2024-09-14T03:34:41.510 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-14T03:34:41.510 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 24 k 2024-09-14T03:34:41.510 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 15 k 2024-09-14T03:34:41.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-14T03:34:41.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-14T03:34:41.510 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-14T03:34:41.510 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-09-14T03:34:41.511 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-14T03:34:41.511 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:41.511 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:34:41.511 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:41.511 INFO:teuthology.orchestra.run.smithi114.stdout:Install 7 Packages 2024-09-14T03:34:41.511 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:41.512 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.7 M 2024-09-14T03:34:41.512 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 80 M 2024-09-14T03:34:41.513 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:34:41.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr 2024-09-14T03:34:41.821 INFO:teuthology.orchestra.run.smithi114.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1629.geb1 162 kB/s | 24 kB 00:00 2024-09-14T03:34:41.847 INFO:teuthology.orchestra.run.smithi114.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1629.geb16 83 kB/s | 15 kB 00:00 2024-09-14T03:34:41.897 INFO:teuthology.orchestra.run.smithi114.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 120 kB/s | 9.0 kB 00:00 2024-09-14T03:34:41.931 INFO:teuthology.orchestra.run.smithi114.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 972 kB/s | 31 kB 00:00 2024-09-14T03:34:41.956 INFO:teuthology.orchestra.run.smithi114.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 376 kB/s | 41 kB 00:00 2024-09-14T03:34:41.989 INFO:teuthology.orchestra.run.smithi114.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-14T03:34:42.073 INFO:teuthology.orchestra.run.smithi114.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1629.geb16b97b 8.3 MB/s | 3.4 MB 00:00 2024-09-14T03:34:42.076 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:Total 6.4 MB/s | 3.7 MB 00:00 2024-09-14T03:34:42.077 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:34:42.085 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:34:42.085 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:34:42.185 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:34:42.185 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:34:42.238 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:57 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:34:42.333 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-mgr-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:34:42.376 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:34:42.377 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-14T03:34:42.377 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:34:42.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-dashboard 2024-09-14T03:34:42.485 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:34:42.558 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-14T03:34:42.592 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-14T03:34:42.638 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-14T03:34:42.670 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-14T03:34:42.699 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1629.geb16b97b.el9.n 5/7 2024-09-14T03:34:43.029 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:58 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:34:43.162 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:34:43.163 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-14T03:34:43.163 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:43.163 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:34:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 3.4 M 2024-09-14T03:34:43.163 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-14T03:34:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 24 k 2024-09-14T03:34:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 15 k 2024-09-14T03:34:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-14T03:34:43.163 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-14T03:34:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-14T03:34:43.164 INFO:teuthology.orchestra.run.smithi160.stdout:Installing weak dependencies: 2024-09-14T03:34:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-14T03:34:43.164 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:43.164 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:34:43.164 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:43.164 INFO:teuthology.orchestra.run.smithi160.stdout:Install 7 Packages 2024-09-14T03:34:43.165 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:43.165 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.7 M 2024-09-14T03:34:43.165 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 80 M 2024-09-14T03:34:43.166 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:34:43.340 INFO:teuthology.orchestra.run.smithi160.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1629.geb16 101 kB/s | 15 kB 00:00 2024-09-14T03:34:43.356 INFO:teuthology.orchestra.run.smithi160.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1629.geb1 151 kB/s | 24 kB 00:00 2024-09-14T03:34:43.390 INFO:teuthology.orchestra.run.smithi160.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 180 kB/s | 9.0 kB 00:00 2024-09-14T03:34:43.415 INFO:teuthology.orchestra.run.smithi160.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-14T03:34:43.432 INFO:teuthology.orchestra.run.smithi160.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 541 kB/s | 41 kB 00:00 2024-09-14T03:34:43.457 INFO:teuthology.orchestra.run.smithi160.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-14T03:34:43.574 INFO:teuthology.orchestra.run.smithi160.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1629.geb16b97b 8.8 MB/s | 3.4 MB 00:00 2024-09-14T03:34:43.578 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:43.578 INFO:teuthology.orchestra.run.smithi160.stdout:Total 8.8 MB/s | 3.7 MB 00:00 2024-09-14T03:34:43.578 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:34:43.586 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:34:43.586 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:34:43.683 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:34:43.683 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:34:43.691 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1629.geb16b97b.el9. 6/7 2024-09-14T03:34:43.738 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1629.geb16b97b.el9.noarc 7/7 2024-09-14T03:34:43.991 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:34:44.059 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-14T03:34:44.093 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-14T03:34:44.148 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-14T03:34:44.180 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-14T03:34:44.209 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1629.geb16b97b.el9.n 5/7 2024-09-14T03:34:44.789 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1629.geb16b97b.el9.noarc 7/7 2024-09-14T03:34:44.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1629.geb16b97b.el9. 1/7 2024-09-14T03:34:44.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1629.geb16b97b.el9.noarc 2/7 2024-09-14T03:34:44.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1629.geb16b97b.el9.n 3/7 2024-09-14T03:34:44.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-14T03:34:44.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-14T03:34:44.789 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-14T03:34:45.084 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-14T03:34:45.084 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:45.084 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:34:45.085 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:45.085 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:45.085 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:45.085 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-14T03:34:45.085 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-14T03:34:45.085 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-14T03:34:45.085 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-14T03:34:45.085 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:45.085 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:34:45.216 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1629.geb16b97b.el9. 6/7 2024-09-14T03:34:45.254 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1629.geb16b97b.el9.noarc 7/7 2024-09-14T03:34:45.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-14T03:34:45.870 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:59 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:34:46.002 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:34:46.003 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-09-14T03:34:46.003 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 7.4 M 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-14T03:34:46.004 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-14T03:34:46.005 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-14T03:34:46.005 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-14T03:34:46.005 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-14T03:34:46.005 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-14T03:34:46.005 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:46.005 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:34:46.005 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-09-14T03:34:46.005 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-09-14T03:34:46.005 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:46.006 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 38 M 2024-09-14T03:34:46.006 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 212 M 2024-09-14T03:34:46.006 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:34:46.249 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-14T03:34:46.283 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-14T03:34:46.289 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1629.geb16b97b.el9.noarc 7/7 2024-09-14T03:34:46.289 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1629.geb16b97b.el9. 1/7 2024-09-14T03:34:46.289 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1629.geb16b97b.el9.noarc 2/7 2024-09-14T03:34:46.290 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1629.geb16b97b.el9.n 3/7 2024-09-14T03:34:46.290 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-14T03:34:46.290 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-14T03:34:46.290 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-14T03:34:46.383 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 244 kB/s | 33 kB 00:00 2024-09-14T03:34:46.434 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 366 kB/s | 18 kB 00:00 2024-09-14T03:34:46.526 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-14T03:34:46.551 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 329 kB/s | 38 kB 00:00 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-14T03:34:46.577 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:46.578 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:34:46.634 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-14T03:34:46.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-14T03:34:46.968 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.4 MB/s | 3.0 MB 00:00 2024-09-14T03:34:47.043 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.3 MB/s | 250 kB 00:00 2024-09-14T03:34:47.377 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:02 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:34:47.435 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.4 MB/s | 5.1 MB 00:00 2024-09-14T03:34:47.512 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:34:47.513 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-09-14T03:34:47.513 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-14T03:34:47.513 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-09-14T03:34:47.513 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:34:47.513 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 7.4 M 2024-09-14T03:34:47.513 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-14T03:34:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-14T03:34:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-14T03:34:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-14T03:34:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-14T03:34:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-14T03:34:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-14T03:34:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-14T03:34:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-14T03:34:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-14T03:34:47.514 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-14T03:34:47.515 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-14T03:34:47.515 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:47.515 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:34:47.515 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-09-14T03:34:47.515 INFO:teuthology.orchestra.run.smithi160.stdout:Install 12 Packages 2024-09-14T03:34:47.515 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:47.516 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 38 M 2024-09-14T03:34:47.516 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 212 M 2024-09-14T03:34:47.516 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:34:47.602 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.6 MB/s | 4.9 MB 00:01 2024-09-14T03:34:47.982 INFO:teuthology.orchestra.run.smithi160.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.9 MB/s | 191 kB 00:00 2024-09-14T03:34:48.024 INFO:teuthology.orchestra.run.smithi160.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 5.5 MB/s | 801 kB 00:00 2024-09-14T03:34:48.083 INFO:teuthology.orchestra.run.smithi160.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 325 kB/s | 33 kB 00:00 2024-09-14T03:34:48.125 INFO:teuthology.orchestra.run.smithi160.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 18 kB 00:00 2024-09-14T03:34:48.167 INFO:teuthology.orchestra.run.smithi160.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 924 kB/s | 38 kB 00:00 2024-09-14T03:34:48.259 INFO:teuthology.orchestra.run.smithi160.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-09-14T03:34:48.301 INFO:teuthology.orchestra.run.smithi160.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 5.8 MB/s | 245 kB 00:00 2024-09-14T03:34:48.401 INFO:teuthology.orchestra.run.smithi160.stdout:(8/12): ceph-mgr-diskprediction-local-17.2.7-16 14 MB/s | 7.4 MB 00:00 2024-09-14T03:34:48.476 INFO:teuthology.orchestra.run.smithi160.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2024-09-14T03:34:48.543 INFO:teuthology.orchestra.run.smithi160.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 5.1 MB 00:00 2024-09-14T03:34:48.627 INFO:teuthology.orchestra.run.smithi160.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.1 MB/s | 250 kB 00:00 2024-09-14T03:34:48.819 INFO:teuthology.orchestra.run.smithi160.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-09-14T03:34:48.819 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:48.820 INFO:teuthology.orchestra.run.smithi160.stdout:Total 29 MB/s | 38 MB 00:01 2024-09-14T03:34:49.180 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:34:49.207 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:34:49.207 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:34:49.610 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.3 MB/s | 16 MB 00:02 2024-09-14T03:34:49.611 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:49.611 INFO:teuthology.orchestra.run.smithi114.stdout:Total 11 MB/s | 38 MB 00:03 2024-09-14T03:34:49.710 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:34:49.711 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:34:49.964 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:34:49.990 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:34:49.990 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:34:50.467 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:34:50.493 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:34:50.494 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:34:50.530 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-14T03:34:50.575 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-14T03:34:50.675 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-14T03:34:50.718 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-14T03:34:50.903 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-14T03:34:50.933 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-14T03:34:51.021 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-14T03:34:51.250 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:34:51.329 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-14T03:34:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-14T03:34:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-14T03:34:51.527 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-14T03:34:51.680 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-14T03:34:51.710 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-14T03:34:51.742 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-14T03:34:51.797 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-14T03:34:51.824 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-14T03:34:52.490 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-14T03:34:52.605 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-14T03:34:53.071 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T03:34:53.849 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T03:34:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-14T03:34:54.132 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1629.geb16b 12/12 2024-09-14T03:34:54.890 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-14T03:34:54.949 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1629.geb16b 12/12 2024-09-14T03:34:56.633 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1629.geb16b 12/12 2024-09-14T03:34:56.633 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1629.geb16b 1/12 2024-09-14T03:34:56.633 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-14T03:34:56.633 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-14T03:34:56.633 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-14T03:34:56.633 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-14T03:34:56.634 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-14T03:34:56.634 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-14T03:34:56.634 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-14T03:34:56.634 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-14T03:34:56.634 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T03:34:56.634 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-14T03:34:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-14T03:34:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:57.058 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:34:57.058 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:57.059 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:34:57.244 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1629.geb16b 12/12 2024-09-14T03:34:57.244 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1629.geb16b 1/12 2024-09-14T03:34:57.244 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-14T03:34:57.244 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-14T03:34:57.244 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-14T03:34:57.244 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-14T03:34:57.244 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-14T03:34:57.244 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-14T03:34:57.245 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-14T03:34:57.245 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-14T03:34:57.245 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T03:34:57.245 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-14T03:34:57.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-rook 2024-09-14T03:34:57.653 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-14T03:34:57.654 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-14T03:34:57.655 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-14T03:34:57.655 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:57.655 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:34:57.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-rook 2024-09-14T03:34:58.072 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:13 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:34:58.206 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:34:58.207 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:58.208 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-14T03:34:58.208 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:58.208 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:34:58.208 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 47 k 2024-09-14T03:34:58.208 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-14T03:34:58.209 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-14T03:34:58.209 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-14T03:34:58.209 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-14T03:34:58.209 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-14T03:34:58.209 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-14T03:34:58.210 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-14T03:34:58.210 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-14T03:34:58.210 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-14T03:34:58.210 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-14T03:34:58.210 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-14T03:34:58.211 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-14T03:34:58.211 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-14T03:34:58.211 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:58.211 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:34:58.211 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:34:58.211 INFO:teuthology.orchestra.run.smithi160.stdout:Install 13 Packages 2024-09-14T03:34:58.212 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:34:58.213 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 2.2 M 2024-09-14T03:34:58.213 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 26 M 2024-09-14T03:34:58.213 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:34:58.482 INFO:teuthology.orchestra.run.smithi160.stdout:(1/13): ceph-mgr-rook-17.2.7-1629.geb16b97b.el9 277 kB/s | 47 kB 00:00 2024-09-14T03:34:58.569 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:12 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:34:58.599 INFO:teuthology.orchestra.run.smithi160.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 66 kB/s | 19 kB 00:00 2024-09-14T03:34:58.624 INFO:teuthology.orchestra.run.smithi160.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 82 kB/s | 26 kB 00:00 2024-09-14T03:34:58.702 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:34:58.703 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-14T03:34:58.703 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:58.703 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:34:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 47 k 2024-09-14T03:34:58.703 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-14T03:34:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:58.704 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:34:58.705 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:34:58.705 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-09-14T03:34:58.705 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:34:58.705 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.2 M 2024-09-14T03:34:58.705 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-09-14T03:34:58.705 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:34:58.983 INFO:teuthology.orchestra.run.smithi160.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 414 kB/s | 159 kB 00:00 2024-09-14T03:34:59.083 INFO:teuthology.orchestra.run.smithi160.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 608 kB/s | 279 kB 00:00 2024-09-14T03:34:59.108 INFO:teuthology.orchestra.run.smithi160.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 428 kB/s | 54 kB 00:00 2024-09-14T03:34:59.134 INFO:teuthology.orchestra.run.smithi160.stdout:(7/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 341 kB/s | 222 kB 00:00 2024-09-14T03:34:59.184 INFO:teuthology.orchestra.run.smithi160.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-14T03:34:59.209 INFO:teuthology.orchestra.run.smithi160.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-14T03:34:59.243 INFO:teuthology.orchestra.run.smithi160.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-14T03:34:59.268 INFO:teuthology.orchestra.run.smithi160.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.8 MB/s | 220 kB 00:00 2024-09-14T03:34:59.293 INFO:teuthology.orchestra.run.smithi160.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-14T03:34:59.327 INFO:teuthology.orchestra.run.smithi160.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.3 MB/s | 1.0 MB 00:00 2024-09-14T03:34:59.330 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:59.330 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.0 MB/s | 2.2 MB 00:01 2024-09-14T03:34:59.391 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 231 kB/s | 19 kB 00:00 2024-09-14T03:34:59.397 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:34:59.413 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:34:59.413 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:34:59.434 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 203 kB/s | 26 kB 00:00 2024-09-14T03:34:59.492 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-14T03:34:59.501 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:34:59.501 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:34:59.517 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-09-14T03:34:59.543 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): ceph-mgr-rook-17.2.7-1629.geb16b97b.el9 203 kB/s | 47 kB 00:00 2024-09-14T03:34:59.568 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-14T03:34:59.593 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-14T03:34:59.643 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-14T03:34:59.669 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-14T03:34:59.694 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-14T03:34:59.719 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-14T03:34:59.744 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-14T03:34:59.778 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-14T03:34:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:34:59.781 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-09-14T03:34:59.843 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:34:59.858 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:34:59.858 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:34:59.946 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:34:59.946 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:35:00.000 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:35:00.169 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-14T03:35:00.229 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-14T03:35:00.292 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-14T03:35:00.329 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-14T03:35:00.371 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-14T03:35:00.454 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:35:00.475 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-14T03:35:00.585 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-14T03:35:00.615 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-14T03:35:00.647 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-14T03:35:00.679 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-14T03:35:00.734 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-14T03:35:00.771 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-14T03:35:00.813 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-14T03:35:00.915 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-14T03:35:01.025 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-14T03:35:01.073 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-14T03:35:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-14T03:35:01.436 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-14T03:35:01.469 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-14T03:35:01.515 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-14T03:35:01.539 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-rook-2:17.2.7-1629.geb16b97b.el9.noarch 13/13 2024-09-14T03:35:01.788 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-14T03:35:01.837 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-14T03:35:01.871 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-14T03:35:01.917 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-14T03:35:01.940 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:17.2.7-1629.geb16b97b.el9.noarch 13/13 2024-09-14T03:35:02.383 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1629.geb16b97b.el9.noarch 13/13 2024-09-14T03:35:02.383 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1629.geb16b97b.el9.noarch 1/13 2024-09-14T03:35:02.383 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-14T03:35:02.383 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-14T03:35:02.383 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-14T03:35:02.384 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-14T03:35:02.384 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-14T03:35:02.384 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-14T03:35:02.384 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-14T03:35:02.384 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-14T03:35:02.384 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-14T03:35:02.384 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-14T03:35:02.384 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-14T03:35:02.732 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-14T03:35:02.732 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:02.732 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:35:02.732 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-14T03:35:02.733 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:02.734 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1629.geb16b97b.el9.noarch 13/13 2024-09-14T03:35:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1629.geb16b97b.el9.noarch 1/13 2024-09-14T03:35:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-14T03:35:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-14T03:35:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-14T03:35:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-14T03:35:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-14T03:35:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-14T03:35:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-14T03:35:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-14T03:35:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-14T03:35:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-14T03:35:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-14T03:35:02.994 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-cephadm 2024-09-14T03:35:03.103 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-14T03:35:03.104 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-14T03:35:03.105 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-14T03:35:03.105 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-14T03:35:03.105 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-14T03:35:03.105 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:03.105 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:03.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-cephadm 2024-09-14T03:35:03.574 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:18 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:03.710 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:03.711 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-14T03:35:03.711 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:03.711 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:35:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 124 k 2024-09-14T03:35:03.711 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-14T03:35:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-14T03:35:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-14T03:35:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-14T03:35:03.711 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-14T03:35:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-14T03:35:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:03.712 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:35:03.712 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:03.712 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2024-09-14T03:35:03.712 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:03.712 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 7.0 M 2024-09-14T03:35:03.713 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 33 M 2024-09-14T03:35:03.713 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:35:03.916 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:17 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:04.045 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-14T03:35:04.049 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:04.050 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-14T03:35:04.050 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1629.geb16b97b.el9 ceph-noarch 124 k 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-09-14T03:35:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:04.052 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 7.0 M 2024-09-14T03:35:04.052 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 33 M 2024-09-14T03:35:04.052 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:35:04.071 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1629.geb16b97b.e 493 kB/s | 124 kB 00:00 2024-09-14T03:35:04.121 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-14T03:35:04.146 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-14T03:35:04.171 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.3 MB/s | 548 kB 00:00 2024-09-14T03:35:04.463 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1629.geb16b97b.e 667 kB/s | 124 kB 00:00 2024-09-14T03:35:04.539 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-14T03:35:04.564 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-14T03:35:04.589 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-14T03:35:05.056 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 319 kB/s | 248 kB 00:00 2024-09-14T03:35:05.506 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2024-09-14T03:35:05.508 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:05.508 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.8 MB/s | 7.0 MB 00:01 2024-09-14T03:35:05.580 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:35:05.592 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:35:05.592 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:35:05.692 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:35:05.692 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:35:06.008 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:35:06.171 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-14T03:35:06.180 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.5 MB/s | 6.0 MB 00:02 2024-09-14T03:35:06.183 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:06.183 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.8 MB/s | 7.0 MB 00:02 2024-09-14T03:35:06.232 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-14T03:35:06.255 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:35:06.267 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:35:06.267 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:35:06.366 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:35:06.366 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:35:06.668 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:35:06.751 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-14T03:35:06.844 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-14T03:35:06.846 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-14T03:35:06.897 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-14T03:35:06.899 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-14T03:35:06.928 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 6/6 2024-09-14T03:35:07.417 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-14T03:35:07.500 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-14T03:35:07.556 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-14T03:35:07.585 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 6/6 2024-09-14T03:35:07.701 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 6/6 2024-09-14T03:35:07.701 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 1/6 2024-09-14T03:35:07.701 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-14T03:35:07.701 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-14T03:35:07.702 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-14T03:35:07.702 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-14T03:35:07.964 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-14T03:35:07.964 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:07.964 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:35:07.965 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:35:07.965 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-14T03:35:07.965 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-14T03:35:07.965 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-14T03:35:07.965 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-14T03:35:07.965 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-14T03:35:07.965 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:07.965 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:08.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-fuse 2024-09-14T03:35:08.347 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 6/6 2024-09-14T03:35:08.347 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 1/6 2024-09-14T03:35:08.347 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-14T03:35:08.347 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-14T03:35:08.347 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-14T03:35:08.347 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-14T03:35:08.622 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-14T03:35:08.622 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:08.622 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:35:08.622 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm-2:17.2.7-1629.geb16b97b.el9.noarch 2024-09-14T03:35:08.622 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-14T03:35:08.622 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-14T03:35:08.622 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-14T03:35:08.623 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-14T03:35:08.623 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-14T03:35:08.623 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:08.623 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:08.772 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:22 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:08.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-fuse 2024-09-14T03:35:08.906 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:08.906 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:08.906 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-14T03:35:08.907 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:08.907 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:35:08.907 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 813 k 2024-09-14T03:35:08.907 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-14T03:35:08.907 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-14T03:35:08.907 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:08.907 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:35:08.907 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:08.907 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-09-14T03:35:08.907 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:08.908 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 893 k 2024-09-14T03:35:08.908 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 2.6 M 2024-09-14T03:35:08.908 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:35:09.341 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 521 kB/s | 80 kB 00:00 2024-09-14T03:35:09.435 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:24 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:09.569 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:09.570 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:09.570 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-09-14T03:35:09.570 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:09.570 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:35:09.570 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 813 k 2024-09-14T03:35:09.570 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-09-14T03:35:09.570 INFO:teuthology.orchestra.run.smithi160.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-14T03:35:09.571 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:09.571 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:35:09.571 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:09.571 INFO:teuthology.orchestra.run.smithi160.stdout:Install 2 Packages 2024-09-14T03:35:09.571 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:09.571 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 893 k 2024-09-14T03:35:09.572 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 2.6 M 2024-09-14T03:35:09.572 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:35:09.733 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-fuse-17.2.7-1629.geb16b97b.el9.x86_ 1.5 MB/s | 813 kB 00:00 2024-09-14T03:35:09.733 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:09.733 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.1 MB/s | 893 kB 00:00 2024-09-14T03:35:09.745 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:35:09.756 INFO:teuthology.orchestra.run.smithi160.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 489 kB/s | 80 kB 00:00 2024-09-14T03:35:09.767 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:35:09.767 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:35:09.849 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:35:09.849 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:35:09.957 INFO:teuthology.orchestra.run.smithi160.stdout:(2/2): ceph-fuse-17.2.7-1629.geb16b97b.el9.x86_ 2.2 MB/s | 813 kB 00:00 2024-09-14T03:35:09.958 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:09.958 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.3 MB/s | 893 kB 00:00 2024-09-14T03:35:09.969 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:35:09.990 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:35:09.990 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:35:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:35:10.069 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:35:10.070 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:35:10.154 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-14T03:35:10.232 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 2/2 2024-09-14T03:35:10.270 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:35:10.411 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-14T03:35:10.480 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 2/2 2024-09-14T03:35:10.901 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 2/2 2024-09-14T03:35:10.901 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 1/2 2024-09-14T03:35:11.149 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 2/2 2024-09-14T03:35:11.149 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 1/2 2024-09-14T03:35:11.176 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-14T03:35:11.176 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:11.176 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:35:11.176 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-14T03:35:11.176 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:11.177 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:11.417 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-14T03:35:11.417 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:11.417 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:35:11.417 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-14T03:35:11.417 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:11.417 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:11.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados-devel 2024-09-14T03:35:11.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados-devel 2024-09-14T03:35:12.010 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:26 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:12.145 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:12.146 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:12.146 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-14T03:35:12.146 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:12.146 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:35:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 126 k 2024-09-14T03:35:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:12.147 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:35:12.147 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:12.147 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-14T03:35:12.147 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:12.147 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 126 k 2024-09-14T03:35:12.147 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 448 k 2024-09-14T03:35:12.147 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:35:12.257 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:27 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:12.354 INFO:teuthology.orchestra.run.smithi114.stdout:librados-devel-17.2.7-1629.geb16b97b.el9.x86_64 610 kB/s | 126 kB 00:00 2024-09-14T03:35:12.355 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:12.355 INFO:teuthology.orchestra.run.smithi114.stdout:Total 606 kB/s | 126 kB 00:00 2024-09-14T03:35:12.355 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:35:12.360 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:35:12.360 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:35:12.394 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:12.395 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:12.395 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-14T03:35:12.395 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:12.395 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:35:12.395 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 126 k 2024-09-14T03:35:12.395 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:12.395 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:35:12.395 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:12.396 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-14T03:35:12.396 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:12.396 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 126 k 2024-09-14T03:35:12.396 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 448 k 2024-09-14T03:35:12.396 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:35:12.423 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:35:12.423 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:35:12.572 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:35:12.587 INFO:teuthology.orchestra.run.smithi160.stdout:librados-devel-17.2.7-1629.geb16b97b.el9.x86_64 662 kB/s | 126 kB 00:00 2024-09-14T03:35:12.587 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:12.588 INFO:teuthology.orchestra.run.smithi160.stdout:Total 656 kB/s | 126 kB 00:00 2024-09-14T03:35:12.588 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:35:12.593 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:35:12.593 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:35:12.670 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:12.670 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:35:12.670 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:35:12.819 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:35:12.917 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librados-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:13.094 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:13.352 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:13.355 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:13.355 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:13.355 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:35:13.355 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:35:13.355 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:13.355 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:13.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs2 2024-09-14T03:35:13.624 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:13.624 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:13.624 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:35:13.624 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:35:13.624 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:13.624 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:13.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs2 2024-09-14T03:35:14.144 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:28 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:14.236 INFO:teuthology.orchestra.run.smithi114.stdout:Package libcephfs2-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:14.280 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:14.280 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-14T03:35:14.281 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:14.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs-devel 2024-09-14T03:35:14.470 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:29 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:14.563 INFO:teuthology.orchestra.run.smithi160.stdout:Package libcephfs2-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:14.606 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:14.607 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-14T03:35:14.607 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:14.670 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs-devel 2024-09-14T03:35:14.933 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:28 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:15.066 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:15.066 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:15.066 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-14T03:35:15.066 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:15.067 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:35:15.067 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 25 k 2024-09-14T03:35:15.067 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:15.067 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:35:15.067 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:15.067 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-14T03:35:15.067 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:15.067 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 25 k 2024-09-14T03:35:15.067 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 110 k 2024-09-14T03:35:15.067 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:35:15.224 INFO:teuthology.orchestra.run.smithi114.stdout:libcephfs-devel-17.2.7-1629.geb16b97b.el9.x86_6 159 kB/s | 25 kB 00:00 2024-09-14T03:35:15.225 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:15.225 INFO:teuthology.orchestra.run.smithi114.stdout:Total 157 kB/s | 25 kB 00:00 2024-09-14T03:35:15.225 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:35:15.229 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:35:15.229 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:35:15.230 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:35:15.230 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:35:15.241 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:30 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:15.319 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:35:15.373 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:15.374 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:15.375 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-14T03:35:15.375 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:15.375 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:35:15.375 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 25 k 2024-09-14T03:35:15.375 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:15.375 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:35:15.375 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:15.375 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-14T03:35:15.375 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:15.375 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 25 k 2024-09-14T03:35:15.376 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 110 k 2024-09-14T03:35:15.376 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:35:15.428 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:15.533 INFO:teuthology.orchestra.run.smithi160.stdout:libcephfs-devel-17.2.7-1629.geb16b97b.el9.x86_6 159 kB/s | 25 kB 00:00 2024-09-14T03:35:15.533 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:15.533 INFO:teuthology.orchestra.run.smithi160.stdout:Total 157 kB/s | 25 kB 00:00 2024-09-14T03:35:15.533 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:35:15.537 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:35:15.537 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:35:15.538 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:35:15.539 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:35:15.614 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:35:15.721 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:15.794 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:16.020 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:16.020 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:35:16.020 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:35:16.020 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:16.020 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:16.094 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:16.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados2 2024-09-14T03:35:16.328 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:16.328 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:16.328 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:35:16.328 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:35:16.329 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:16.329 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:16.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados2 2024-09-14T03:35:16.758 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:30 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:16.848 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:16.890 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:16.891 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-14T03:35:16.891 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:16.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librbd1 2024-09-14T03:35:17.087 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:32 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:17.179 INFO:teuthology.orchestra.run.smithi160.stdout:Package librados2-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:17.221 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:17.222 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-14T03:35:17.222 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:17.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librbd1 2024-09-14T03:35:17.515 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:31 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:17.605 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:17.647 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:17.648 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-14T03:35:17.648 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:17.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rados 2024-09-14T03:35:17.851 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:32 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:17.941 INFO:teuthology.orchestra.run.smithi160.stdout:Package librbd1-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:17.984 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:17.985 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-14T03:35:17.985 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:18.061 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rados 2024-09-14T03:35:18.283 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:32 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:18.373 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rados-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:18.415 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:18.416 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-14T03:35:18.416 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:18.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rgw 2024-09-14T03:35:18.627 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:33 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:18.717 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rados-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:18.760 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:18.761 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-14T03:35:18.761 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:18.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rgw 2024-09-14T03:35:19.044 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:33 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:19.136 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rgw-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:19.179 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-14T03:35:19.180 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:19.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-cephfs 2024-09-14T03:35:19.389 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:34 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:19.485 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rgw-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:19.528 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:19.529 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-14T03:35:19.529 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:19.595 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-cephfs 2024-09-14T03:35:19.807 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:33 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-cephfs-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:19.939 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:19.940 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-14T03:35:19.940 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:20.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rbd 2024-09-14T03:35:20.159 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:35 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:20.250 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-cephfs-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:20.292 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:20.293 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-14T03:35:20.293 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:20.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rbd 2024-09-14T03:35:20.567 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:20.658 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rbd-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:20.701 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:20.702 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-09-14T03:35:20.702 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:20.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-fuse 2024-09-14T03:35:20.924 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:35 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:21.015 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rbd-2:17.2.7-1629.geb16b97b.el9.x86_64 is already installed. 2024-09-14T03:35:21.057 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:21.058 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-09-14T03:35:21.058 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:21.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-fuse 2024-09-14T03:35:21.328 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:35 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:21.464 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:21.464 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:21.464 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 82 k 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 82 k 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 222 k 2024-09-14T03:35:21.465 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:35:21.698 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:36 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:21.832 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:21.832 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:21.833 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-14T03:35:21.833 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:21.833 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:35:21.833 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 82 k 2024-09-14T03:35:21.833 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:21.833 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:35:21.833 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:21.833 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-14T03:35:21.833 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:21.834 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 82 k 2024-09-14T03:35:21.834 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 222 k 2024-09-14T03:35:21.834 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:35:21.844 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-fuse-17.2.7-1629.geb16b97b.el9.x86_64.rpm 215 kB/s | 82 kB 00:00 2024-09-14T03:35:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:Total 214 kB/s | 82 kB 00:00 2024-09-14T03:35:21.845 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:35:21.852 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:35:21.852 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:35:21.915 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:35:21.915 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:35:22.070 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:35:22.174 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:22.228 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-fuse-17.2.7-1629.geb16b97b.el9.x86_64.rpm 208 kB/s | 82 kB 00:00 2024-09-14T03:35:22.228 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:22.229 INFO:teuthology.orchestra.run.smithi160.stdout:Total 207 kB/s | 82 kB 00:00 2024-09-14T03:35:22.229 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:35:22.235 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:35:22.236 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:35:22.299 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:35:22.300 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:35:22.452 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:35:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:22.575 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:22.852 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:22.852 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:22.852 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:35:22.853 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:35:22.853 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:22.853 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:22.936 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:23.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-mirror 2024-09-14T03:35:23.203 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:23.203 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:23.203 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:35:23.204 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:35:23.204 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:23.204 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:23.435 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-mirror 2024-09-14T03:35:23.642 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:37 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:23.776 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:23.776 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:23.777 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-09-14T03:35:23.777 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:23.777 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:35:23.777 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 2.9 M 2024-09-14T03:35:23.777 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:23.777 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:35:23.777 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:23.777 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-14T03:35:23.777 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:23.778 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.9 M 2024-09-14T03:35:23.778 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 12 M 2024-09-14T03:35:23.778 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:35:24.011 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:39 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:24.149 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 2.9 M 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:24.150 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 2.9 M 2024-09-14T03:35:24.151 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 12 M 2024-09-14T03:35:24.151 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:35:24.480 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-mirror-17.2.7-1629.geb16b97b.el9.x86_64.rpm 4.2 MB/s | 2.9 MB 00:00 2024-09-14T03:35:24.481 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:24.481 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.2 MB/s | 2.9 MB 00:00 2024-09-14T03:35:24.482 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:35:24.491 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:35:24.491 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:35:24.573 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:35:24.574 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:35:24.822 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-mirror-17.2.7-1629.geb16b97b.el9.x86_64.rpm 4.4 MB/s | 2.9 MB 00:00 2024-09-14T03:35:24.823 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:24.823 INFO:teuthology.orchestra.run.smithi160.stdout:Total 4.4 MB/s | 2.9 MB 00:00 2024-09-14T03:35:24.824 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:35:24.833 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:35:24.833 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:35:24.913 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:35:24.914 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:35:24.995 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:35:25.165 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:25.190 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:25.190 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:35:25.190 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-14T03:35:25.190 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T03:35:25.190 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T03:35:25.190 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:25.335 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:35:25.428 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-mirror-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:25.453 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T03:35:25.453 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-14T03:35:25.453 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T03:35:25.453 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T03:35:25.453 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:26.197 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:26.198 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:26.198 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:35:26.198 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:35:26.198 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:26.198 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:26.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-nbd 2024-09-14T03:35:26.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-mirror-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:26.582 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:26.582 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:35:26.582 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:35:26.582 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:26.582 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:26.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-nbd 2024-09-14T03:35:26.985 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:40 ago on Sat 14 Sep 2024 03:33:46 AM UTC. 2024-09-14T03:35:27.120 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-14T03:35:27.120 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:27.121 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-14T03:35:27.121 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:27.121 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-14T03:35:27.121 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 165 k 2024-09-14T03:35:27.121 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:27.121 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-14T03:35:27.121 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-14T03:35:27.121 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-09-14T03:35:27.121 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:27.122 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 165 k 2024-09-14T03:35:27.122 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 477 k 2024-09-14T03:35:27.122 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-14T03:35:27.325 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-nbd-17.2.7-1629.geb16b97b.el9.x86_64.rpm 811 kB/s | 165 kB 00:00 2024-09-14T03:35:27.326 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:27.326 INFO:teuthology.orchestra.run.smithi114.stdout:Total 805 kB/s | 165 kB 00:00 2024-09-14T03:35:27.326 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-14T03:35:27.334 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-14T03:35:27.334 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-14T03:35:27.396 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 03:33:45 AM UTC. 2024-09-14T03:35:27.399 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-14T03:35:27.399 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-14T03:35:27.532 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-09-14T03:35:27.533 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:27.533 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-09-14T03:35:27.533 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:27.533 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-09-14T03:35:27.533 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd x86_64 2:17.2.7-1629.geb16b97b.el9 ceph 165 k 2024-09-14T03:35:27.533 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:27.533 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-09-14T03:35:27.533 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-09-14T03:35:27.533 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-09-14T03:35:27.534 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:27.534 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 165 k 2024-09-14T03:35:27.534 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 477 k 2024-09-14T03:35:27.534 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-09-14T03:35:27.550 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-14T03:35:27.656 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:27.743 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-nbd-17.2.7-1629.geb16b97b.el9.x86_64.rpm 792 kB/s | 165 kB 00:00 2024-09-14T03:35:27.743 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:35:27.743 INFO:teuthology.orchestra.run.smithi160.stdout:Total 786 kB/s | 165 kB 00:00 2024-09-14T03:35:27.743 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-09-14T03:35:27.751 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-09-14T03:35:27.751 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-09-14T03:35:27.816 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-09-14T03:35:27.816 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-09-14T03:35:27.973 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-09-14T03:35:28.074 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-nbd-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:28.089 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:28.331 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:28.332 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:28.332 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-14T03:35:28.332 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:35:28.332 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:35:28.332 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-14T03:35:28.523 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:28.538 DEBUG:teuthology.parallel:result is None 2024-09-14T03:35:28.747 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-nbd-2:17.2.7-1629.geb16b97b.el9.x86_64 1/1 2024-09-14T03:35:28.747 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:28.747 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-09-14T03:35:28.747 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd-2:17.2.7-1629.geb16b97b.el9.x86_64 2024-09-14T03:35:28.747 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:35:28.747 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-09-14T03:35:29.006 DEBUG:teuthology.parallel:result is None 2024-09-14T03:35:29.006 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T03:35:29.006 INFO:teuthology.packaging:ref: None 2024-09-14T03:35:29.006 INFO:teuthology.packaging:tag: None 2024-09-14T03:35:29.006 INFO:teuthology.packaging:branch: quincy 2024-09-14T03:35:29.006 INFO:teuthology.packaging:sha1: a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:35:29.007 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-14T03:35:29.215 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-14T03:35:29.240 INFO:teuthology.orchestra.run.smithi114.stdout:17.2.7-1629.geb16b97b.el9 2024-09-14T03:35:29.241 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1629.geb16b97b.el9 2024-09-14T03:35:29.241 INFO:teuthology.task.install:The correct ceph version 17.2.7-1629.geb16b97b is installed. 2024-09-14T03:35:29.243 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T03:35:29.243 INFO:teuthology.packaging:ref: None 2024-09-14T03:35:29.243 INFO:teuthology.packaging:tag: None 2024-09-14T03:35:29.243 INFO:teuthology.packaging:branch: quincy 2024-09-14T03:35:29.243 INFO:teuthology.packaging:sha1: a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:35:29.243 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-14T03:35:29.443 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-14T03:35:29.464 INFO:teuthology.orchestra.run.smithi160.stdout:17.2.7-1629.geb16b97b.el9 2024-09-14T03:35:29.465 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1629.geb16b97b.el9 2024-09-14T03:35:29.465 INFO:teuthology.task.install:The correct ceph version 17.2.7-1629.geb16b97b is installed. 2024-09-14T03:35:29.466 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-14T03:35:29.467 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:35:29.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-14T03:35:29.497 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:35:29.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-14T03:35:29.532 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-14T03:35:29.532 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:35:29.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-09-14T03:35:29.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-14T03:35:29.633 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:35:29.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2024-09-14T03:35:29.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-14T03:35:29.724 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-14T03:35:29.724 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:35:29.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-14T03:35:29.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-14T03:35:29.817 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:35:29.817 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-14T03:35:29.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-14T03:35:29.907 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-14T03:35:29.907 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:35:29.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-09-14T03:35:29.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-14T03:35:30.000 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:35:30.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2024-09-14T03:35:30.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-14T03:35:30.090 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-14T03:35:30.191 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': 'a2782315255a3e7d5f6cdc9eb35785187ff6ddf8'} 2024-09-14T03:35:30.191 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-09-14T03:35:30.191 INFO:tasks.cephadm:Cluster fsid is 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:35:30.191 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-14T03:35:30.191 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.114', 'mon.c': '[v2:172.21.15.114:3301,v1:172.21.15.114:6790]', 'mon.b': '172.21.15.160'} 2024-09-14T03:35:30.192 INFO:tasks.cephadm:First mon is mon.a on smithi114 2024-09-14T03:35:30.192 INFO:tasks.cephadm:First mgr is y 2024-09-14T03:35:30.192 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-14T03:35:30.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-09-14T03:35:30.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo hostname $(hostname -s) 2024-09-14T03:35:30.243 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-09-14T03:35:30.243 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-14T03:35:30.388 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 14 03:35 /home/ubuntu/cephtest/cephadm 2024-09-14T03:35:30.388 DEBUG:teuthology.orchestra.run.smithi160:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-14T03:35:30.521 INFO:teuthology.orchestra.run.smithi160.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 14 03:35 /home/ubuntu/cephtest/cephadm 2024-09-14T03:35:30.522 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-14T03:35:30.539 DEBUG:teuthology.orchestra.run.smithi160:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-14T03:35:30.565 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-09-14T03:35:30.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-14T03:35:30.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-14T03:35:32.181 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-14T03:35:32.273 INFO:teuthology.orchestra.run.smithi160.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-14T03:36:19.469 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T03:36:19.470 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_version": "ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)", 2024-09-14T03:36:19.470 INFO:teuthology.orchestra.run.smithi160.stdout: "image_id": "ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446", 2024-09-14T03:36:19.470 INFO:teuthology.orchestra.run.smithi160.stdout: "repo_digests": [ 2024-09-14T03:36:19.470 INFO:teuthology.orchestra.run.smithi160.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6" 2024-09-14T03:36:19.470 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T03:36:19.470 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T03:36:21.258 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T03:36:21.258 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)", 2024-09-14T03:36:21.258 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446", 2024-09-14T03:36:21.258 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-09-14T03:36:21.258 INFO:teuthology.orchestra.run.smithi114.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6" 2024-09-14T03:36:21.258 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T03:36:21.258 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T03:36:21.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-09-14T03:36:21.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph 2024-09-14T03:36:21.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-09-14T03:36:21.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /etc/ceph 2024-09-14T03:36:21.430 INFO:tasks.cephadm:Writing seed config... 2024-09-14T03:36:21.432 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-14T03:36:21.432 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-14T03:36:21.432 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-14T03:36:21.432 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-14T03:36:21.432 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-14T03:36:21.432 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-14T03:36:21.432 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-14T03:36:21.432 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-14T03:36:21.432 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-14T03:36:21.433 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-14T03:36:21.433 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-14T03:36:21.433 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:36:21.433 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-14T03:36:21.450 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 63f4e762-724a-11ef-bceb-c7b262605968 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-14T03:36:21.450 DEBUG:teuthology.orchestra.run.smithi114:mon.a> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.a.service 2024-09-14T03:36:21.493 DEBUG:teuthology.orchestra.run.smithi114:mgr.y> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.y.service 2024-09-14T03:36:21.536 INFO:tasks.cephadm:Bootstrapping... 2024-09-14T03:36:21.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 63f4e762-724a-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.114 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-14T03:36:21.732 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-14T03:36:21.733 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '63f4e762-724a-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.114', '--skip-admin-label'] 2024-09-14T03:36:21.756 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.2.2 2024-09-14T03:36:21.756 INFO:teuthology.orchestra.run.smithi114.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-14T03:36:21.757 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying podman|docker is present... 2024-09-14T03:36:21.778 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.2.2 2024-09-14T03:36:21.778 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying lvm2 is present... 2024-09-14T03:36:21.778 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying time synchronization is in place... 2024-09-14T03:36:21.786 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-14T03:36:21.786 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-14T03:36:21.793 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-14T03:36:21.793 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-14T03:36:21.800 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-09-14T03:36:21.806 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-09-14T03:36:21.806 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-09-14T03:36:21.806 INFO:teuthology.orchestra.run.smithi114.stdout:Repeating the final host check... 2024-09-14T03:36:21.828 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.2.2 2024-09-14T03:36:21.828 INFO:teuthology.orchestra.run.smithi114.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-14T03:36:21.828 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl is present 2024-09-14T03:36:21.828 INFO:teuthology.orchestra.run.smithi114.stdout:lvcreate is present 2024-09-14T03:36:21.835 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-14T03:36:21.835 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-14T03:36:21.841 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-14T03:36:21.841 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-14T03:36:21.848 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-09-14T03:36:21.855 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-09-14T03:36:21.855 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-09-14T03:36:21.855 INFO:teuthology.orchestra.run.smithi114.stdout:Host looks OK 2024-09-14T03:36:21.855 INFO:teuthology.orchestra.run.smithi114.stdout:Cluster fsid: 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:36:21.855 INFO:teuthology.orchestra.run.smithi114.stdout:Acquiring lock 139728471454624 on /run/cephadm/63f4e762-724a-11ef-bceb-c7b262605968.lock 2024-09-14T03:36:21.855 INFO:teuthology.orchestra.run.smithi114.stdout:Lock 139728471454624 acquired on /run/cephadm/63f4e762-724a-11ef-bceb-c7b262605968.lock 2024-09-14T03:36:21.855 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 3300 ... 2024-09-14T03:36:21.856 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 6789 ... 2024-09-14T03:36:21.856 INFO:teuthology.orchestra.run.smithi114.stdout:Base mon IP(s) is [172.21.15.114:3300, 172.21.15.114:6789], mon addrv is [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-14T03:36:21.859 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.114 metric 100 2024-09-14T03:36:21.859 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.114 metric 100 2024-09-14T03:36:21.862 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-14T03:36:21.862 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-14T03:36:21.862 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-14T03:36:21.865 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-14T03:36:21.865 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-14T03:36:21.865 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-14T03:36:21.865 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-14T03:36:21.865 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link noprefixroute 2024-09-14T03:36:21.865 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-14T03:36:21.866 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-09-14T03:36:21.866 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-09-14T03:36:21.866 INFO:teuthology.orchestra.run.smithi114.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-14T03:36:21.866 INFO:teuthology.orchestra.run.smithi114.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-14T03:36:21.866 INFO:teuthology.orchestra.run.smithi114.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-14T03:36:22.326 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 2024-09-14T03:36:22.326 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-14T03:36:22.326 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Getting image source signatures 2024-09-14T03:36:22.326 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:9b559dbc30e0607f8b5bc9e421c2d3b7f5b89b8ae7a0518eaf6d2b64276fd6bc 2024-09-14T03:36:22.326 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-14T03:36:22.326 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying config sha256:ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 2024-09-14T03:36:22.326 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-14T03:36:23.262 INFO:teuthology.orchestra.run.smithi114.stdout:ceph: stdout ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable) 2024-09-14T03:36:23.262 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph version: ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable) 2024-09-14T03:36:23.262 INFO:teuthology.orchestra.run.smithi114.stdout:Extracting ceph user uid/gid from container image... 2024-09-14T03:36:24.293 INFO:teuthology.orchestra.run.smithi114.stdout:stat: stdout 167 167 2024-09-14T03:36:24.293 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial keys... 2024-09-14T03:36:25.215 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQC4BOVmZ/tDIxAATlkkkqqWzG98d5BspP3v2A== 2024-09-14T03:36:26.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQC5BOVmRYyZHxAAzt9s2pm3GtBqpSBpqYf0UA== 2024-09-14T03:36:27.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQC6BOVm2mLvIBAAjuO6eZTxFxrFWrwsUBORQA== 2024-09-14T03:36:27.151 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial monmap... 2024-09-14T03:36:28.140 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-14T03:36:28.140 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-14T03:36:28.141 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:36:28.141 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-14T03:36:28.141 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool for a [v2:172.21.15.114:3300,v1:172.21.15.114:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-14T03:36:28.141 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = octopus 2024-09-14T03:36:28.141 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: set fsid to 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:36:28.141 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-14T03:36:28.141 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:36:28.141 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mon... 2024-09-14T03:36:29.112 INFO:teuthology.orchestra.run.smithi114.stdout:create mon.a on 2024-09-14T03:36:29.333 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-14T03:36:29.476 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-14T03:36:29.633 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-63f4e762-724a-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-63f4e762-724a-11ef-bceb-c7b262605968.target. 2024-09-14T03:36:29.633 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-63f4e762-724a-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-63f4e762-724a-11ef-bceb-c7b262605968.target. 2024-09-14T03:36:29.805 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.a 2024-09-14T03:36:29.805 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-14T03:36:29.965 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-63f4e762-724a-11ef-bceb-c7b262605968.target.wants/ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-63f4e762-724a-11ef-bceb-c7b262605968@.service. 2024-09-14T03:36:30.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:30 smithi114 ceph-mon[29119]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-14T03:36:30.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:30 smithi114 bash[29081]: 0cf6f0863a748fd73629de8fc9fadf99b36b7bbd424c6d02dfabcb4954f8ccf2 2024-09-14T03:36:30.487 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-14T03:36:30.487 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-14T03:36:30.494 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-14T03:36:30.494 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-14T03:36:30.494 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-14T03:36:30.494 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-09-14T03:36:30.495 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon to start... 2024-09-14T03:36:30.495 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon... 2024-09-14T03:36:30.817 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:30 smithi114 systemd[1]: Started Ceph mon.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:36:31.475 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout cluster: 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout id: 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout services: 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.515956s) 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout data: 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-14T03:36:31.476 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-14T03:36:31.477 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-14T03:36:31.477 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pgs: 2024-09-14T03:36:31.477 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:31.477 INFO:teuthology.orchestra.run.smithi114.stdout:mon is available 2024-09-14T03:36:31.477 INFO:teuthology.orchestra.run.smithi114.stdout:Assimilating anything we can from ceph.conf... 2024-09-14T03:36:31.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:31 smithi114 ceph-mon[29119]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-14T03:36:31.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:31 smithi114 ceph-mon[29119]: monmap e1: 1 mons at {a=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-14T03:36:31.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:31 smithi114 ceph-mon[29119]: fsmap 2024-09-14T03:36:31.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:31 smithi114 ceph-mon[29119]: osdmap e1: 0 total, 0 up, 0 in 2024-09-14T03:36:31.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:31 smithi114 ceph-mon[29119]: mgrmap e1: no daemons active 2024-09-14T03:36:31.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:31 smithi114 ceph-mon[29119]: from='client.? 172.21.15.114:0/2601612640' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-14T03:36:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-09-14T03:36:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:36:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-14T03:36:32.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-14T03:36:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-14T03:36:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-14T03:36:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-14T03:36:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-14T03:36:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:32.451 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-14T03:36:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-14T03:36:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-09-14T03:36:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-14T03:36:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-14T03:36:32.452 INFO:teuthology.orchestra.run.smithi114.stdout:Generating new minimal ceph.conf... 2024-09-14T03:36:32.788 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:32 smithi114 ceph-mon[29119]: from='client.? 172.21.15.114:0/2170260941' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-14T03:36:32.788 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:32 smithi114 ceph-mon[29119]: from='client.? 172.21.15.114:0/2170260941' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-14T03:36:33.419 INFO:teuthology.orchestra.run.smithi114.stdout:Restarting the monitor... 2024-09-14T03:36:33.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:33 smithi114 systemd[1]: Stopping Ceph mon.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:36:33.705 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:33 smithi114 ceph-mon[29119]: from='client.? 172.21.15.114:0/1760832393' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:36:34.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:33 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[29096]: 2024-09-14T03:36:33.703+0000 7fc455687640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T03:36:34.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:33 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[29096]: 2024-09-14T03:36:33.703+0000 7fc455687640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-14T03:36:34.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:33 smithi114 podman[29438]: 2024-09-14 03:36:33.87741227 +0000 UTC m=+0.299792074 container died 0cf6f0863a748fd73629de8fc9fadf99b36b7bbd424c6d02dfabcb4954f8ccf2 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-14T03:36:34.479 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:34 smithi114 podman[29438]: 2024-09-14 03:36:34.260477433 +0000 UTC m=+0.682857237 container cleanup 0cf6f0863a748fd73629de8fc9fadf99b36b7bbd424c6d02dfabcb4954f8ccf2 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, RELEASE=quincy-eb16b97, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731) 2024-09-14T03:36:34.479 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:34 smithi114 bash[29438]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a 2024-09-14T03:36:34.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:34 smithi114 podman[29454]: 2024-09-14 03:36:34.478099948 +0000 UTC m=+0.592877949 container remove 0cf6f0863a748fd73629de8fc9fadf99b36b7bbd424c6d02dfabcb4954f8ccf2 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-eb16b97, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-14T03:36:34.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:34 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-14T03:36:34.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:34 smithi114 systemd[1]: Stopped Ceph mon.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:36:34.797 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:34 smithi114 systemd[1]: Starting Ceph mon.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:36:35.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:34 smithi114 podman[29551]: 2024-09-14 03:36:34.7955872 +0000 UTC m=+0.105530092 container create 92df6a472c268de312420180dc902f911c2aa33af1a55a86220a7ed28249113c (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=quincy-eb16b97, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2) 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:34 smithi114 podman[29551]: 2024-09-14 03:36:34.705383254 +0000 UTC m=+0.015326147 image pull ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 quay.ceph.io/ceph-ci/ceph:quincy 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:34 smithi114 podman[29551]: 2024-09-14 03:36:34.981397062 +0000 UTC m=+0.291339954 container init 92df6a472c268de312420180dc902f911c2aa33af1a55a86220a7ed28249113c (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-eb16b97, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:34 smithi114 podman[29551]: 2024-09-14 03:36:34.985722012 +0000 UTC m=+0.295664904 container start 92df6a472c268de312420180dc902f911c2aa33af1a55a86220a7ed28249113c (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: set uid:gid to 167:167 (ceph:ceph) 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable), process ceph-mon, pid 7 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: pidfile_write: ignore empty --pid-file 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: load: jerasure load: lrc 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: RocksDB version: 6.15.5 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Compile date Sep 13 2024 2024-09-14T03:36:35.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: DB SUMMARY 2024-09-14T03:36:35.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: DB Session ID: 9X8SX0HAPE4QCYY7HENI 2024-09-14T03:36:35.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: CURRENT file: CURRENT 2024-09-14T03:36:35.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: IDENTITY file: IDENTITY 2024-09-14T03:36:35.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-14T03:36:35.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-14T03:36:35.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83716 ; 2024-09-14T03:36:35.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.error_if_exists: 0 2024-09-14T03:36:35.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.create_if_missing: 0 2024-09-14T03:36:35.059 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.paranoid_checks: 1 2024-09-14T03:36:35.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-14T03:36:35.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.env: 0x55f3fd3c7440 2024-09-14T03:36:35.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.fs: Posix File System 2024-09-14T03:36:35.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.info_log: 0x55f3ff464ec0 2024-09-14T03:36:35.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_file_opening_threads: 16 2024-09-14T03:36:35.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.statistics: (nil) 2024-09-14T03:36:35.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.use_fsync: 0 2024-09-14T03:36:35.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_log_file_size: 0 2024-09-14T03:36:35.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-14T03:36:35.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.keep_log_file_num: 1000 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.recycle_log_file_num: 0 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.allow_fallocate: 1 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.allow_mmap_reads: 0 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.allow_mmap_writes: 0 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.use_direct_reads: 0 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.create_missing_column_families: 0 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.db_log_dir: 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-14T03:36:35.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.advise_random_on_open: 1 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.db_write_buffer_size: 0 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.write_buffer_manager: 0x55f3ff522780 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-14T03:36:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.rate_limiter: (nil) 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.wal_recovery_mode: 2 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.enable_thread_tracking: 0 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.enable_pipelined_write: 0 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.unordered_write: 0 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.row_cache: None 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.wal_filter: None 2024-09-14T03:36:35.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.allow_ingest_behind: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.preserve_deletes: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.two_write_queues: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.manual_wal_flush: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.atomic_flush: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.log_readahead_size: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.best_efforts_recovery: 0 2024-09-14T03:36:35.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.allow_data_in_errors: 0 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.db_host_id: __hostname__ 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_background_jobs: 2 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_background_compactions: -1 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_subcompactions: 1 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_total_wal_size: 0 2024-09-14T03:36:35.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_open_files: -1 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bytes_per_sync: 0 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_readahead_size: 0 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_background_flushes: -1 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Compression algorithms supported: 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: kZSTD supported: 0 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: kXpressCompression supported: 0 2024-09-14T03:36:35.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: kBZip2Compression supported: 0 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: kLZ4Compression supported: 1 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: kZlibCompression supported: 1 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: kLZ4HCCompression supported: 1 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: kSnappyCompression supported: 1 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.merge_operator: 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_filter: None 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_filter_factory: None 2024-09-14T03:36:35.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.sst_partitioner_factory: None 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f3ff40ba38) 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_type: 0 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_index_type: 0 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_shortening: 1 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: hash_index_allow_collision: 1 2024-09-14T03:36:35.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: checksum: 1 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: no_block_cache: 0 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache: 0x55f3ff4411f0 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_options: 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: capacity : 536870912 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_shard_bits : 4 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: strict_capacity_limit : 0 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_compressed: (nil) 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: persistent_cache: (nil) 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size: 4096 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size_deviation: 10 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_restart_interval: 16 2024-09-14T03:36:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_block_restart_interval: 1 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: metadata_block_size: 4096 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: partition_filters: 0 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: use_delta_encoding: 1 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: whole_key_filtering: 1 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: verify_compression: 0 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: format_version: 4 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_index_compression: 1 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_align: 0 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.write_buffer_size: 33554432 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_write_buffer_number: 2 2024-09-14T03:36:35.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compression: NoCompression 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bottommost_compression: Disabled 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.prefix_extractor: nullptr 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.num_levels: 7 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:36:35.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compression_opts.level: 32767 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compression_opts.strategy: 0 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compression_opts.enabled: false 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.target_file_size_base: 67108864 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-14T03:36:35.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.arena_block_size: 4194304 2024-09-14T03:36:35.074 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.disable_auto_compactions: 0 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-14T03:36:35.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.inplace_update_support: 0 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.bloom_locality: 0 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.max_successive_merges: 0 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.paranoid_file_checks: 0 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.force_consistency_checks: 1 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.report_bg_io_stats: 0 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.ttl: 2592000 2024-09-14T03:36:35.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.enable_blob_files: false 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.min_blob_size: 0 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.blob_file_size: 268435456 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726284995009890, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726284995010913, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80289, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78531, "index_size": 220, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10130, "raw_average_key_size": 47, "raw_value_size": 72744, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726284995, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "d111feed-7c54-45cb-be1d-17d4b0df22db", "db_session_id": "9X8SX0HAPE4QCYY7HENI"}} 2024-09-14T03:36:35.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726284995011368, "job": 1, "event": "recovery_finished"} 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55f3ff474700 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: DB pointer 0x55f3ff444000 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** DB Stats ** 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-14T03:36:35.078 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: L0 2/0 80.19 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.6 0.00 0.00 1 0.001 0 0 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: Sum 2/0 80.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.6 0.00 0.00 1 0.001 0 0 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.6 0.00 0.00 1 0.001 0 0 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:36:35.079 INFO:journalctl@ceph.mon.a.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 111.6 0.00 0.00 1 0.001 0 0 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative compaction: 0.00 GB write, 18.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval compaction: 0.00 GB write, 18.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:36:35.080 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-14T03:36:35.081 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:36:35.081 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:36:35.081 INFO:journalctl@ceph.mon.a.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:36:35.081 INFO:journalctl@ceph.mon.a.smithi114.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:36:35.081 INFO:journalctl@ceph.mon.a.smithi114.stdout: L0 2/0 80.19 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.6 0.00 0.00 1 0.001 0 0 2024-09-14T03:36:35.081 INFO:journalctl@ceph.mon.a.smithi114.stdout: Sum 2/0 80.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.6 0.00 0.00 1 0.001 0 0 2024-09-14T03:36:35.082 INFO:journalctl@ceph.mon.a.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-14T03:36:35.082 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:36:35.082 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:36:35.082 INFO:journalctl@ceph.mon.a.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:36:35.082 INFO:journalctl@ceph.mon.a.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:36:35.082 INFO:journalctl@ceph.mon.a.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 111.6 0.00 0.00 1 0.001 0 0 2024-09-14T03:36:35.082 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:36:35.083 INFO:journalctl@ceph.mon.a.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-14T03:36:35.083 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-14T03:36:35.083 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-14T03:36:35.083 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-14T03:36:35.083 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-14T03:36:35.083 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative compaction: 0.00 GB write, 18.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:36:35.083 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:36:35.083 INFO:journalctl@ceph.mon.a.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: starting mon.a rank 0 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: mon.a@-1(???) e1 preinit fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: mon.a@-1(???).mds e1 new map 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: mon.a@-1(???).mds e1 print_map 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout: e1 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout: legacy client fscid: -1 2024-09-14T03:36:35.084 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:36:35.085 INFO:journalctl@ceph.mon.a.smithi114.stdout: No filesystems configured 2024-09-14T03:36:35.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-14T03:36:35.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-14T03:36:35.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-14T03:36:35.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-14T03:36:35.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-14T03:36:35.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 bash[29551]: 92df6a472c268de312420180dc902f911c2aa33af1a55a86220a7ed28249113c 2024-09-14T03:36:35.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 systemd[1]: Started Ceph mon.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:36:35.086 INFO:teuthology.orchestra.run.smithi114.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-14T03:36:35.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-14T03:36:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: monmap e1: 1 mons at {a=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-14T03:36:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: fsmap 2024-09-14T03:36:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: osdmap e1: 0 total, 0 up, 0 in 2024-09-14T03:36:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:35 smithi114 ceph-mon[29589]: mgrmap e1: no daemons active 2024-09-14T03:36:36.134 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-14T03:36:36.134 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-14T03:36:36.134 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mgr... 2024-09-14T03:36:36.134 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-14T03:36:36.135 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-14T03:36:36.314 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.y 2024-09-14T03:36:36.314 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-14T03:36:36.460 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-63f4e762-724a-11ef-bceb-c7b262605968.target.wants/ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-63f4e762-724a-11ef-bceb-c7b262605968@.service. 2024-09-14T03:36:36.764 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:36 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3466392420' entity='client.admin' 2024-09-14T03:36:36.764 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:36 smithi114 podman[29823]: 2024-09-14 03:36:36.602617369 +0000 UTC m=+0.016282825 image pull ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 quay.ceph.io/ceph-ci/ceph:quincy 2024-09-14T03:36:36.964 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-14T03:36:36.964 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-14T03:36:36.972 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-14T03:36:36.972 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-14T03:36:36.972 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-14T03:36:36.972 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-09-14T03:36:36.982 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-14T03:36:36.982 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-14T03:36:36.990 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-14T03:36:36.990 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-14T03:36:36.990 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-14T03:36:36.990 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-14T03:36:36.990 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr to start... 2024-09-14T03:36:36.991 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr... 2024-09-14T03:36:37.115 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:36 smithi114 podman[29823]: 2024-09-14 03:36:36.857618131 +0000 UTC m=+0.271283586 container init 57c15055b7ead15cffc93972c5164187465a5e69c4db9e29f495606c140c40c0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=quincy-eb16b97, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-09-14T03:36:37.115 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:36 smithi114 podman[29823]: 2024-09-14 03:36:36.86241012 +0000 UTC m=+0.276075577 container start 57c15055b7ead15cffc93972c5164187465a5e69c4db9e29f495606c140c40c0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-eb16b97, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-14T03:36:37.115 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:36 smithi114 bash[29823]: 57c15055b7ead15cffc93972c5164187465a5e69c4db9e29f495606c140c40c0 2024-09-14T03:36:37.115 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:36 smithi114 systemd[1]: Started Ceph mgr.y for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:36:37.115 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:36 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:36.978+0000 7fb524b02ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-14T03:36:37.116 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:37 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:37.039+0000 7fb524b02ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-14T03:36:37.116 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:37 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:37.103+0000 7fb524b02ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-14T03:36:37.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:37 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:37.228+0000 7fb524b02ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:36:37.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:37 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3403261206' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-14T03:36:37.641 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:37 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:37.369+0000 7fb524b02ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-14T03:36:38.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:37 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:37.639+0000 7fb524b02ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-14T03:36:38.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:37 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:37.699+0000 7fb524b02ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-14T03:36:38.121 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:37 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:37.863+0000 7fb524b02ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-14T03:36:38.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:38.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-14T03:36:38.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "63f4e762-724a-11ef-bceb-c7b262605968", 2024-09-14T03:36:38.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-14T03:36:38.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-14T03:36:38.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:38.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-14T03:36:38.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-14T03:36:38.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:38.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-14T03:36:38.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:38.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-14T03:36:38.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-14T03:36:38.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-14T03:36:38.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-14T03:36:38.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-14T03:36:38.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-14T03:36:38.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:38.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-14T03:36:38.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-14T03:36:38.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:38.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-14T03:36:30.445883+0000", 2024-09-14T03:36:38.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-14T03:36:38.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:38.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-14T03:36:38.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-14T03:36:38.155 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (1/15)... 2024-09-14T03:36:39.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:38 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:38.698+0000 7fb524b02ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-14T03:36:39.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:38 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:38.760+0000 7fb524b02ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-14T03:36:39.742 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:39.457+0000 7fb524b02ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-14T03:36:39.742 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:39.566+0000 7fb524b02ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-14T03:36:39.742 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:39.683+0000 7fb524b02ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:36:40.115 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:39.740+0000 7fb524b02ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-14T03:36:40.115 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:39.862+0000 7fb524b02ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-14T03:36:40.115 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:39.927+0000 7fb524b02ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-14T03:36:40.116 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:39.983+0000 7fb524b02ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-14T03:36:40.116 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:40.114+0000 7fb524b02ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-14T03:36:40.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:40.184+0000 7fb524b02ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-14T03:36:40.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:40.248+0000 7fb524b02ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-14T03:36:40.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:40.329+0000 7fb524b02ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-14T03:36:40.651 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:40.389+0000 7fb524b02ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-14T03:36:40.651 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:40.449+0000 7fb524b02ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-14T03:36:40.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:40 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/375752181' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-14T03:36:41.247 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:41.206+0000 7fb524b02ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-14T03:36:41.259 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:41.259 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-14T03:36:41.260 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "63f4e762-724a-11ef-bceb-c7b262605968", 2024-09-14T03:36:41.260 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-14T03:36:41.260 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-14T03:36:41.260 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-14T03:36:41.260 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-14T03:36:41.260 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:41.261 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-14T03:36:41.261 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-14T03:36:41.261 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-14T03:36:41.261 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-14T03:36:41.261 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-14T03:36:41.261 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-09-14T03:36:41.262 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-14T03:36:41.262 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-14T03:36:41.262 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-14T03:36:41.262 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:41.262 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-14T03:36:41.262 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-14T03:36:41.262 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:41.263 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-14T03:36:41.263 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:41.263 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-14T03:36:41.263 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-14T03:36:41.263 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-14T03:36:41.263 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-14T03:36:41.264 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-14T03:36:41.264 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-14T03:36:41.264 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:41.264 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-14T03:36:41.264 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-14T03:36:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-14T03:36:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-14T03:36:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-14T03:36:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-14T03:36:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-14T03:36:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-14T03:36:41.265 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-14T03:36:41.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:41.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-14T03:36:41.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:41.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-14T03:36:41.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-14T03:36:41.266 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:41.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-14T03:36:41.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-14T03:36:41.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-14T03:36:41.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-14T03:36:41.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-14T03:36:41.267 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-14T03:36:41.268 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-14T03:36:41.268 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-14T03:36:41.269 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-14T03:36:41.269 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:41.269 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-14T03:36:41.269 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:41.269 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-14T03:36:30.445883+0000", 2024-09-14T03:36:41.269 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-14T03:36:41.269 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:41.269 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-14T03:36:41.269 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-14T03:36:41.269 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (2/15)... 2024-09-14T03:36:42.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: Activating manager daemon y 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: mgrmap e2: y(active, starting, since 0.00266655s) 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: from='mgr.14100 172.21.15.114:0/3953259354' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: from='mgr.14100 172.21.15.114:0/3953259354' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: from='mgr.14100 172.21.15.114:0/3953259354' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: from='mgr.14100 172.21.15.114:0/3953259354' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: from='mgr.14100 172.21.15.114:0/3953259354' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: Manager daemon y is now available 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: from='mgr.14100 172.21.15.114:0/3953259354' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: from='mgr.14100 172.21.15.114:0/3953259354' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: from='mgr.14100 172.21.15.114:0/3953259354' entity='mgr.y' 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: from='mgr.14100 172.21.15.114:0/3953259354' entity='mgr.y' 2024-09-14T03:36:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:41 smithi114 ceph-mon[29589]: from='mgr.14100 172.21.15.114:0/3953259354' entity='mgr.y' 2024-09-14T03:36:43.465 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:43 smithi114 ceph-mon[29589]: mgrmap e3: y(active, since 1.00722s) 2024-09-14T03:36:44.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:44 smithi114 ceph-mon[29589]: mgrmap e4: y(active, since 2s) 2024-09-14T03:36:44.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:44 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1752875351' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-14T03:36:44.441 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:44.441 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-14T03:36:44.441 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "63f4e762-724a-11ef-bceb-c7b262605968", 2024-09-14T03:36:44.441 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-14T03:36:44.441 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-14T03:36:44.441 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-14T03:36:44.441 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-14T03:36:44.441 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:44.442 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-14T03:36:44.443 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:44.443 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-14T03:36:44.443 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-14T03:36:44.443 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-14T03:36:44.443 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-14T03:36:44.443 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-14T03:36:44.443 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-14T03:36:44.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:44.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-14T03:36:44.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:44.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-14T03:36:44.449 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-14T03:36:44.449 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:44.449 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-14T03:36:44.449 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-14T03:36:44.449 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-14T03:36:44.449 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-14T03:36:44.449 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-14T03:36:44.449 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-14T03:36:44.449 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-14T03:36:44.449 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-14T03:36:44.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-14T03:36:44.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:44.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-14T03:36:44.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-14T03:36:44.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-14T03:36:30.445883+0000", 2024-09-14T03:36:44.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-14T03:36:44.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-14T03:36:44.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-14T03:36:44.450 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-14T03:36:44.450 INFO:teuthology.orchestra.run.smithi114.stdout:mgr is available 2024-09-14T03:36:45.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:45 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/4234476385' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-14T03:36:50.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:50.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-09-14T03:36:50.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:36:50.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-14T03:36:50.999 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-14T03:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling cephadm module... 2024-09-14T03:36:54.542 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:54 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/602576235' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-14T03:36:54.542 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ignoring --setuser ceph since I am not root 2024-09-14T03:36:54.542 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ignoring --setgroup ceph since I am not root 2024-09-14T03:36:54.542 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:54.302+0000 7f49d0b3fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-14T03:36:54.543 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:54.356+0000 7f49d0b3fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-14T03:36:54.543 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:54.419+0000 7f49d0b3fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-14T03:36:54.543 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:54.540+0000 7f49d0b3fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:36:54.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:54.674+0000 7f49d0b3fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-14T03:36:55.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:55 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/602576235' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-14T03:36:55.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:55 smithi114 ceph-mon[29589]: mgrmap e5: y(active, since 12s) 2024-09-14T03:36:55.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:54.929+0000 7f49d0b3fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-14T03:36:55.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:54.988+0000 7f49d0b3fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-14T03:36:55.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:55.149+0000 7f49d0b3fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-14T03:36:56.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:55.982+0000 7f49d0b3fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-14T03:36:56.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:56.046+0000 7f49d0b3fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-14T03:36:57.027 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:56.747+0000 7f49d0b3fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-14T03:36:57.027 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:56.863+0000 7f49d0b3fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-14T03:36:57.027 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:56.982+0000 7f49d0b3fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:36:57.277 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:57 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:57.041+0000 7f49d0b3fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-14T03:36:57.278 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:57 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:57.166+0000 7f49d0b3fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-14T03:36:57.278 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:57 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:57.234+0000 7f49d0b3fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-14T03:36:57.540 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:57 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:57.294+0000 7f49d0b3fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-14T03:36:57.541 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:57 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:57.431+0000 7f49d0b3fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-14T03:36:57.541 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:57 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:57.494+0000 7f49d0b3fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-14T03:36:57.805 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:57 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:57.564+0000 7f49d0b3fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-14T03:36:57.806 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:57 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:57.648+0000 7f49d0b3fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-14T03:36:57.806 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:57 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:57.706+0000 7f49d0b3fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-14T03:36:57.806 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:57 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:57.765+0000 7f49d0b3fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-14T03:36:58.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:58 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3362069512' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-14T03:36:58.609 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:36:58 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:36:58.523+0000 7f49d0b3fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-14T03:36:58.611 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-14T03:36:58.612 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-14T03:36:58.613 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-14T03:36:58.613 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-14T03:36:58.613 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-14T03:36:58.613 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-14T03:36:58.613 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-09-14T03:36:58.613 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 5... 2024-09-14T03:36:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: Active manager daemon y restarted 2024-09-14T03:36:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: Activating manager daemon y 2024-09-14T03:36:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: osdmap e2: 0 total, 0 up, 0 in 2024-09-14T03:36:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: mgrmap e6: y(active, starting, since 0.00340598s) 2024-09-14T03:36:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:36:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:36:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:36:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:36:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:36:59.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: Manager daemon y is now available 2024-09-14T03:36:59.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:36:59.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:36:59.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:36:59.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:36:59.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:36:59.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:36:59 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:37:00.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-mon[29589]: Found migration_current of "None". Setting to last migration. 2024-09-14T03:37:00.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-mon[29589]: mgrmap e7: y(active, since 1.0053s) 2024-09-14T03:37:00.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-14T03:37:00.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-14T03:37:00.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-14T03:37:00.135 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-14T03:37:00.135 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 5 is available 2024-09-14T03:37:00.135 INFO:teuthology.orchestra.run.smithi114.stdout:Setting orchestrator backend to cephadm... 2024-09-14T03:37:00.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:00.306+0000 7f4982f2e640 -1 log_channel(cephadm) log [ERR] : [14/Sep/2024:03:37:00] ENGINE Error in HTTPServer.serve 2024-09-14T03:37:00.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: Traceback (most recent call last): 2024-09-14T03:37:00.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:37:00.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self._connections.run(self.expiration_interval) 2024-09-14T03:37:00.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:37:00.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self._run(expiration_interval) 2024-09-14T03:37:00.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:37:00.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:37:00.461 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: s = self.context.wrap_socket( 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: return self.sslsocket_class._create( 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self.do_handshake() 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self._sslobj.do_handshake() 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:37:00.462 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:01.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:01 smithi114 ceph-mon[29589]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-14T03:37:01.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:01 smithi114 ceph-mon[29589]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-14T03:37:01.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:01 smithi114 ceph-mon[29589]: [14/Sep/2024:03:37:00] ENGINE Bus STARTING 2024-09-14T03:37:01.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:01 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:01.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:01 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:37:01.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:01 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:01.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:01 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:37:02.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:02 smithi114 ceph-mon[29589]: [14/Sep/2024:03:37:00] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:02 smithi114 ceph-mon[29589]: [14/Sep/2024:03:37:00] ENGINE Bus STARTED 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:02 smithi114 ceph-mon[29589]: [14/Sep/2024:03:37:00] ENGINE Error in HTTPServer.serve 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:37:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:37:02.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-14T03:37:02.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:37:02.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-14T03:37:02.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:37:02.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-14T03:37:02.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:37:02.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-14T03:37:02.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:37:02.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:02 smithi114 ceph-mon[29589]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:02.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:02 smithi114 ceph-mon[29589]: mgrmap e8: y(active, since 2s) 2024-09-14T03:37:02.401 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-14T03:37:02.402 INFO:teuthology.orchestra.run.smithi114.stdout:Generating ssh key... 2024-09-14T03:37:03.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-mon[29589]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:03.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: Generating public/private rsa key pair. 2024-09-14T03:37:03.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: Your identification has been saved in /tmp/tmpi0k0f_3c/key 2024-09-14T03:37:03.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: Your public key has been saved in /tmp/tmpi0k0f_3c/key.pub 2024-09-14T03:37:03.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: The key fingerprint is: 2024-09-14T03:37:03.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: SHA256:4p51bZrkdrplfZT+wt50LuIBn/a9vUSwmrfZDUDK//g ceph-63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:37:03.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: The key's randomart image is: 2024-09-14T03:37:03.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: +---[RSA 3072]----+ 2024-09-14T03:37:03.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: | | 2024-09-14T03:37:03.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: | | 2024-09-14T03:37:03.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: | .. | 2024-09-14T03:37:03.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: | . o o .| 2024-09-14T03:37:03.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: | . S + .. o.| 2024-09-14T03:37:03.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: | . . =o+o. | 2024-09-14T03:37:03.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: | . . oo@ooo+| 2024-09-14T03:37:03.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: | . o +.B+=OB=| 2024-09-14T03:37:03.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: | o .*=o*E*X| 2024-09-14T03:37:03.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: +----[SHA256]-----+ 2024-09-14T03:37:04.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:04 smithi114 ceph-mon[29589]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:04.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:04 smithi114 ceph-mon[29589]: Generating ssh key... 2024-09-14T03:37:04.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:04 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:04.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:04 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:04.718 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChEBrZcsZ5k1Zeg2Fs/1FSg1q9HLnEbps//BE8AfinPXYEscj1clzhZHc+s17DQwvkXSC6nJMMdHKV+u7FKokoRpl1wNNiEur6NTLdWZjZBhEszdkLUwy1k0ASrmYv9pPb8u1hWW0+B86nBmyr6DNMGgk6vLl0b06QNoE0wGpq4iP/a9LDkTPypntF+pXEXg1uzbuJNO7lb8x2d0LqzaUlzpjBQrTj0PgxPg0tC4eKR7e/dzTdfpX8muZivgUBNd506J6MtviaHuhGalaSDzUO49KLW/RnyLGaQV9+qF3ZuikiODt+/PuTfYQqHNFGTUEdY1Z2dEZ5JIXw/B9ZqtR55arlC0mFNkJS47f9vHrYCRyFboGkxfBIc8gPuDMG/i5nQlQSZhd9jBby6HN3dTyn/mo+wnDyFSJd+8jOwHXSK2jajnzrDGU6NPCo4i+P9Qq1Kj2jAnc7FUqD4RU3XwVKm3sTawgY4OJjhUt7hdJDBmW014+MIlJg4LTUP0MFuOE= ceph-63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:37:04.719 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-14T03:37:04.719 INFO:teuthology.orchestra.run.smithi114.stdout:Adding key to root@localhost authorized_keys... 2024-09-14T03:37:04.722 INFO:teuthology.orchestra.run.smithi114.stdout:Adding host smithi114... 2024-09-14T03:37:05.411 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:05 smithi114 ceph-mon[29589]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:06.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:06 smithi114 ceph-mon[29589]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:06.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:06 smithi114 ceph-mon[29589]: Deploying cephadm binary to smithi114 2024-09-14T03:37:08.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Added host 'smithi114' with addr '172.21.15.114' 2024-09-14T03:37:08.626 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mon service... 2024-09-14T03:37:09.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:09 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:09.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:09 smithi114 ceph-mon[29589]: Added host smithi114 2024-09-14T03:37:09.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:09 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:37:09.944 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-14T03:37:09.944 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mgr service... 2024-09-14T03:37:10.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:10 smithi114 ceph-mon[29589]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:10.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:10 smithi114 ceph-mon[29589]: Saving service mon spec with placement count:5 2024-09-14T03:37:10.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:10 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:11.923 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-14T03:37:12.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:11 smithi114 ceph-mon[29589]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:12.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:11 smithi114 ceph-mon[29589]: Saving service mgr spec with placement count:2 2024-09-14T03:37:12.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:11 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:12.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:11 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:13.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:12 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:13.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:12 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1807812016' entity='client.admin' 2024-09-14T03:37:14.679 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:14 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:14.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:14 smithi114 ceph-mon[29589]: from='mgr.14118 172.21.15.114:0/1359948863' entity='mgr.y' 2024-09-14T03:37:15.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:15 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3442342628' entity='client.admin' 2024-09-14T03:37:15.998 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling the dashboard module... 2024-09-14T03:37:16.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:16 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1347315060' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-14T03:37:16.941 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:16 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ignoring --setuser ceph since I am not root 2024-09-14T03:37:16.941 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:16 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ignoring --setgroup ceph since I am not root 2024-09-14T03:37:16.941 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:16 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:16.819+0000 7f499679eec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-14T03:37:16.941 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:16 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:16.873+0000 7f499679eec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-14T03:37:16.941 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:16 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:16.938+0000 7f499679eec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-14T03:37:17.200 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:17 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:17.063+0000 7f499679eec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:37:17.200 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:17 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:17.199+0000 7f499679eec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-14T03:37:17.787 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:17 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1347315060' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-14T03:37:17.787 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:17 smithi114 ceph-mon[29589]: mgrmap e9: y(active, since 18s) 2024-09-14T03:37:17.788 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:17 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:17.457+0000 7f499679eec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-14T03:37:17.788 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:17 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:17.521+0000 7f499679eec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-14T03:37:17.788 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:17 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:17.686+0000 7f499679eec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-14T03:37:18.658 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:18 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:18.543+0000 7f499679eec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-14T03:37:18.658 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:18 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:18.608+0000 7f499679eec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-14T03:37:18.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-14T03:37:18.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-14T03:37:18.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-14T03:37:18.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-14T03:37:18.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-14T03:37:18.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-14T03:37:18.662 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-09-14T03:37:18.662 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 9... 2024-09-14T03:37:18.912 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:18 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/466711094' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-14T03:37:19.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:19.332+0000 7f499679eec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-14T03:37:19.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:19.443+0000 7f499679eec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-14T03:37:19.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:19.560+0000 7f499679eec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:37:19.999 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:19.619+0000 7f499679eec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-14T03:37:19.999 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:19.742+0000 7f499679eec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-14T03:37:19.999 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:19.808+0000 7f499679eec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-14T03:37:20.000 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:19.865+0000 7f499679eec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-14T03:37:20.000 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:19.998+0000 7f499679eec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-14T03:37:20.263 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:20.061+0000 7f499679eec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-14T03:37:20.263 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:20.125+0000 7f499679eec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-14T03:37:20.263 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:20.205+0000 7f499679eec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-14T03:37:20.263 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:20.261+0000 7f499679eec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-14T03:37:20.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:20.318+0000 7f499679eec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-14T03:37:21.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:21.052+0000 7f499679eec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-14T03:37:21.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: Active manager daemon y restarted 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: Activating manager daemon y 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: osdmap e3: 0 total, 0 up, 0 in 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: mgrmap e10: y(active, starting, since 0.00308771s) 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: Manager daemon y is now available 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:37:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:37:22.910 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:22.756+0000 7f49472fc640 -1 log_channel(cephadm) log [ERR] : [14/Sep/2024:03:37:22] ENGINE Error in HTTPServer.serve 2024-09-14T03:37:22.910 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: Traceback (most recent call last): 2024-09-14T03:37:22.910 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:37:22.910 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self._connections.run(self.expiration_interval) 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self._run(expiration_interval) 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: s = self.context.wrap_socket( 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: return self.sslsocket_class._create( 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self.do_handshake() 2024-09-14T03:37:22.911 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:37:22.912 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self._sslobj.do_handshake() 2024-09-14T03:37:22.912 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:37:22.912 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:37:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:37:22.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-14T03:37:22.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-14T03:37:22.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-14T03:37:22.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-14T03:37:22.913 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 9 is available 2024-09-14T03:37:22.913 INFO:teuthology.orchestra.run.smithi114.stdout:Generating a dashboard self-signed certificate... 2024-09-14T03:37:23.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:23 smithi114 ceph-mon[29589]: mgrmap e11: y(active, since 1.00678s) 2024-09-14T03:37:23.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:23 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:23.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:23 smithi114 ceph-mon[29589]: [14/Sep/2024:03:37:22] ENGINE Bus STARTING 2024-09-14T03:37:23.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:23 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:24.168 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-14T03:37:24.168 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial admin user... 2024-09-14T03:37:24.591 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:24 smithi114 ceph-mon[29589]: [14/Sep/2024:03:37:22] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:37:24.591 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:24 smithi114 ceph-mon[29589]: [14/Sep/2024:03:37:22] ENGINE Bus STARTED 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:24 smithi114 ceph-mon[29589]: [14/Sep/2024:03:37:22] ENGINE Error in HTTPServer.serve 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:37:24.592 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:24 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:24 smithi114 ceph-mon[29589]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:24 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:24 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:24 smithi114 ceph-mon[29589]: mgrmap e12: y(active, since 2s) 2024-09-14T03:37:26.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:25 smithi114 ceph-mon[29589]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:26.464 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$D1VNlvXo7fC94a/EE6vJy.oQddBippgcluS1WUM0dOKHz1B3BmvUm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726285045, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-14T03:37:26.464 INFO:teuthology.orchestra.run.smithi114.stdout:Fetching dashboard port number... 2024-09-14T03:37:27.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:27.578 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 8443 2024-09-14T03:37:27.588 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-14T03:37:27.588 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-14T03:37:27.596 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-14T03:37:27.596 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-14T03:37:27.596 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-14T03:37:27.596 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-14T03:37:27.598 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph Dashboard is now available at: 2024-09-14T03:37:27.598 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:27.598 INFO:teuthology.orchestra.run.smithi114.stdout: URL: https://smithi114.front.sepia.ceph.com:8443/ 2024-09-14T03:37:27.598 INFO:teuthology.orchestra.run.smithi114.stdout: User: admin 2024-09-14T03:37:27.598 INFO:teuthology.orchestra.run.smithi114.stdout: Password: o30v4nhky0 2024-09-14T03:37:27.598 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:27.599 INFO:teuthology.orchestra.run.smithi114.stdout:Saving cluster configuration to /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config directory 2024-09-14T03:37:27.599 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling autotune for osd_memory_target 2024-09-14T03:37:28.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:27 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1986023489' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-14T03:37:29.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:28 smithi114 ceph-mon[29589]: mgrmap e13: y(active, since 6s) 2024-09-14T03:37:29.975 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-14T03:37:29.976 INFO:teuthology.orchestra.run.smithi114.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-14T03:37:29.976 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:29.976 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-14T03:37:29.976 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:29.976 INFO:teuthology.orchestra.run.smithi114.stdout:Or, if you are only running a single cluster on this host: 2024-09-14T03:37:29.976 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:29.976 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-14T03:37:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:29.977 INFO:teuthology.orchestra.run.smithi114.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-14T03:37:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: ceph telemetry on 2024-09-14T03:37:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:29.977 INFO:teuthology.orchestra.run.smithi114.stdout:For more information see: 2024-09-14T03:37:29.977 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:29.978 INFO:teuthology.orchestra.run.smithi114.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-14T03:37:29.978 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:29.978 INFO:teuthology.orchestra.run.smithi114.stdout:Bootstrap complete. 2024-09-14T03:37:30.010 INFO:tasks.cephadm:Fetching config... 2024-09-14T03:37:30.010 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:37:30.010 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-14T03:37:30.027 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-14T03:37:30.028 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:37:30.028 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-14T03:37:30.083 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-14T03:37:30.083 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:37:30.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-14T03:37:30.147 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-14T03:37:30.147 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:37:30.148 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-14T03:37:30.203 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-14T03:37:30.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChEBrZcsZ5k1Zeg2Fs/1FSg1q9HLnEbps//BE8AfinPXYEscj1clzhZHc+s17DQwvkXSC6nJMMdHKV+u7FKokoRpl1wNNiEur6NTLdWZjZBhEszdkLUwy1k0ASrmYv9pPb8u1hWW0+B86nBmyr6DNMGgk6vLl0b06QNoE0wGpq4iP/a9LDkTPypntF+pXEXg1uzbuJNO7lb8x2d0LqzaUlzpjBQrTj0PgxPg0tC4eKR7e/dzTdfpX8muZivgUBNd506J6MtviaHuhGalaSDzUO49KLW/RnyLGaQV9+qF3ZuikiODt+/PuTfYQqHNFGTUEdY1Z2dEZ5JIXw/B9ZqtR55arlC0mFNkJS47f9vHrYCRyFboGkxfBIc8gPuDMG/i5nQlQSZhd9jBby6HN3dTyn/mo+wnDyFSJd+8jOwHXSK2jajnzrDGU6NPCo4i+P9Qq1Kj2jAnc7FUqD4RU3XwVKm3sTawgY4OJjhUt7hdJDBmW014+MIlJg4LTUP0MFuOE= ceph-63f4e762-724a-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-14T03:37:30.277 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChEBrZcsZ5k1Zeg2Fs/1FSg1q9HLnEbps//BE8AfinPXYEscj1clzhZHc+s17DQwvkXSC6nJMMdHKV+u7FKokoRpl1wNNiEur6NTLdWZjZBhEszdkLUwy1k0ASrmYv9pPb8u1hWW0+B86nBmyr6DNMGgk6vLl0b06QNoE0wGpq4iP/a9LDkTPypntF+pXEXg1uzbuJNO7lb8x2d0LqzaUlzpjBQrTj0PgxPg0tC4eKR7e/dzTdfpX8muZivgUBNd506J6MtviaHuhGalaSDzUO49KLW/RnyLGaQV9+qF3ZuikiODt+/PuTfYQqHNFGTUEdY1Z2dEZ5JIXw/B9ZqtR55arlC0mFNkJS47f9vHrYCRyFboGkxfBIc8gPuDMG/i5nQlQSZhd9jBby6HN3dTyn/mo+wnDyFSJd+8jOwHXSK2jajnzrDGU6NPCo4i+P9Qq1Kj2jAnc7FUqD4RU3XwVKm3sTawgY4OJjhUt7hdJDBmW014+MIlJg4LTUP0MFuOE= ceph-63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:37:30.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChEBrZcsZ5k1Zeg2Fs/1FSg1q9HLnEbps//BE8AfinPXYEscj1clzhZHc+s17DQwvkXSC6nJMMdHKV+u7FKokoRpl1wNNiEur6NTLdWZjZBhEszdkLUwy1k0ASrmYv9pPb8u1hWW0+B86nBmyr6DNMGgk6vLl0b06QNoE0wGpq4iP/a9LDkTPypntF+pXEXg1uzbuJNO7lb8x2d0LqzaUlzpjBQrTj0PgxPg0tC4eKR7e/dzTdfpX8muZivgUBNd506J6MtviaHuhGalaSDzUO49KLW/RnyLGaQV9+qF3ZuikiODt+/PuTfYQqHNFGTUEdY1Z2dEZ5JIXw/B9ZqtR55arlC0mFNkJS47f9vHrYCRyFboGkxfBIc8gPuDMG/i5nQlQSZhd9jBby6HN3dTyn/mo+wnDyFSJd+8jOwHXSK2jajnzrDGU6NPCo4i+P9Qq1Kj2jAnc7FUqD4RU3XwVKm3sTawgY4OJjhUt7hdJDBmW014+MIlJg4LTUP0MFuOE= ceph-63f4e762-724a-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-14T03:37:30.407 INFO:teuthology.orchestra.run.smithi160.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChEBrZcsZ5k1Zeg2Fs/1FSg1q9HLnEbps//BE8AfinPXYEscj1clzhZHc+s17DQwvkXSC6nJMMdHKV+u7FKokoRpl1wNNiEur6NTLdWZjZBhEszdkLUwy1k0ASrmYv9pPb8u1hWW0+B86nBmyr6DNMGgk6vLl0b06QNoE0wGpq4iP/a9LDkTPypntF+pXEXg1uzbuJNO7lb8x2d0LqzaUlzpjBQrTj0PgxPg0tC4eKR7e/dzTdfpX8muZivgUBNd506J6MtviaHuhGalaSDzUO49KLW/RnyLGaQV9+qF3ZuikiODt+/PuTfYQqHNFGTUEdY1Z2dEZ5JIXw/B9ZqtR55arlC0mFNkJS47f9vHrYCRyFboGkxfBIc8gPuDMG/i5nQlQSZhd9jBby6HN3dTyn/mo+wnDyFSJd+8jOwHXSK2jajnzrDGU6NPCo4i+P9Qq1Kj2jAnc7FUqD4RU3XwVKm3sTawgY4OJjhUt7hdJDBmW014+MIlJg4LTUP0MFuOE= ceph-63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:37:30.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-14T03:37:30.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:30 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1659076910' entity='client.admin' 2024-09-14T03:37:30.666 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:37:33.625 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-14T03:37:33.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-14T03:37:33.647 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:33.648 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:33.648 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:37:33.648 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:33.648 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:37:33.648 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:33.648 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:33 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2844590763' entity='client.admin' 2024-09-14T03:37:33.648 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:37:33.648 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:33.943 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:37:36.882 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi160 2024-09-14T03:37:36.882 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:37:36.882 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.conf 2024-09-14T03:37:36.903 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:37:36.903 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:37:36.962 INFO:tasks.cephadm:Adding host smithi160 to orchestrator... 2024-09-14T03:37:36.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch host add smithi160 2024-09-14T03:37:37.205 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:37:37.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:37.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:37.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:37:37.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:37:37.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:37:37.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:37:37.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:37:37.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:37:37.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:37.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:37.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:38.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:38 smithi114 ceph-mon[29589]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:37:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:39 smithi114 ceph-mon[29589]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:41.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:40 smithi114 ceph-mon[29589]: Deploying cephadm binary to smithi160 2024-09-14T03:37:41.485 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi160' with addr '172.21.15.160' 2024-09-14T03:37:41.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-14T03:37:42.211 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:37:42.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:42 smithi114 ceph-mon[29589]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:42.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:42.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:42 smithi114 ceph-mon[29589]: Added host smithi160 2024-09-14T03:37:42.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:37:42.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:42.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:43.838 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:37:43.838 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.160", "hostname": "smithi160", "labels": [], "status": ""}] 2024-09-14T03:37:43.848 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:43 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:43.849 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:43 smithi114 ceph-mon[29589]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:44.386 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-14T03:37:44.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-14T03:37:44.620 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:37:45.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:44 smithi114 ceph-mon[29589]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:37:46.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:45 smithi114 ceph-mon[29589]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:46.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:45 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:46.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:45 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:46.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:45 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:46.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:45 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:46.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:45 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:37:46.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:45 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:37:46.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:45 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:37:46.849 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-09-14T03:37:47.090 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:46 smithi114 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:37:47.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:46 smithi114 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:37:47.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:46 smithi114 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:37:47.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:46 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2832947111' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-14T03:37:47.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:46 smithi114 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:37:47.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:47.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:47.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:47.390 INFO:tasks.cephadm:Adding mon.a on smithi114 2024-09-14T03:37:47.390 INFO:tasks.cephadm:Adding mon.c on smithi114 2024-09-14T03:37:47.390 INFO:tasks.cephadm:Adding mon.b on smithi160 2024-09-14T03:37:47.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi160:172.21.15.160=b' 2024-09-14T03:37:47.628 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:37:48.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:47 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2832947111' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-14T03:37:48.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:47 smithi114 ceph-mon[29589]: osdmap e4: 0 total, 0 up, 0 in 2024-09-14T03:37:48.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:47 smithi114 ceph-mon[29589]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:49.268 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mon update... 2024-09-14T03:37:49.788 DEBUG:teuthology.orchestra.run.smithi114:mon.c> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.c.service 2024-09-14T03:37:49.791 DEBUG:teuthology.orchestra.run.smithi160:mon.b> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.b.service 2024-09-14T03:37:49.793 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-14T03:37:49.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-14T03:37:50.553 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:37:50.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi160:172.21.15.160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi160:172.21.15.160=b;count:3 2024-09-14T03:37:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:37:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:37:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:37:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:37:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:37:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:50 smithi114 ceph-mon[29589]: Deploying daemon mon.b on smithi160 2024-09-14T03:37:52.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:52 smithi114 ceph-mon[29589]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:53.486 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:37:53.486 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","modified":"2024-09-14T03:36:27.521144Z","created":"2024-09-14T03:36:27.521144Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-14T03:37:53.487 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-09-14T03:37:54.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:54 smithi114 ceph-mon[29589]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:54.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:54 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/196651785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-14T03:37:54.790 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 podman[32664]: 2024-09-14 03:37:54.770111142 +0000 UTC m=+0.330534591 container start a30277b5b152463ff7ddd7c9ab8c3067a2a9f7d5bc83262baadf3ce513dd5b2d (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-eb16b97, org.label-schema.schema-version=1.0) 2024-09-14T03:37:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: set uid:gid to 167:167 (ceph:ceph) 2024-09-14T03:37:55.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable), process ceph-mon, pid 7 2024-09-14T03:37:55.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: pidfile_write: ignore empty --pid-file 2024-09-14T03:37:55.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: load: jerasure load: lrc 2024-09-14T03:37:55.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: RocksDB version: 6.15.5 2024-09-14T03:37:55.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Compile date Sep 13 2024 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: DB SUMMARY 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: DB Session ID: 6E0VCG0Y7AFRENXXZGM9 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: CURRENT file: CURRENT 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: IDENTITY file: IDENTITY 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.error_if_exists: 0 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.create_if_missing: 0 2024-09-14T03:37:55.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.paranoid_checks: 1 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.env: 0x562a25b24440 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.fs: Posix File System 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.info_log: 0x562a264acec0 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_file_opening_threads: 16 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.statistics: (nil) 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.use_fsync: 0 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_log_file_size: 0 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.keep_log_file_num: 1000 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.recycle_log_file_num: 0 2024-09-14T03:37:55.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.allow_fallocate: 1 2024-09-14T03:37:55.162 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.allow_mmap_reads: 0 2024-09-14T03:37:55.162 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.allow_mmap_writes: 0 2024-09-14T03:37:55.162 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.use_direct_reads: 0 2024-09-14T03:37:55.162 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-14T03:37:55.162 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.create_missing_column_families: 0 2024-09-14T03:37:55.162 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.db_log_dir: 2024-09-14T03:37:55.162 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-09-14T03:37:55.162 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-14T03:37:55.162 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-14T03:37:55.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-14T03:37:55.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-14T03:37:55.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-14T03:37:55.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-14T03:37:55.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.advise_random_on_open: 1 2024-09-14T03:37:55.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.db_write_buffer_size: 0 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.write_buffer_manager: 0x562a2656a780 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.rate_limiter: (nil) 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.wal_recovery_mode: 2 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.enable_thread_tracking: 0 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.enable_pipelined_write: 0 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.unordered_write: 0 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-14T03:37:55.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.row_cache: None 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.wal_filter: None 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.allow_ingest_behind: 0 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.preserve_deletes: 0 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.two_write_queues: 0 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.manual_wal_flush: 0 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.atomic_flush: 0 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-14T03:37:55.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.log_readahead_size: 0 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.best_efforts_recovery: 0 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.allow_data_in_errors: 0 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.db_host_id: __hostname__ 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_background_jobs: 2 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_background_compactions: -1 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_subcompactions: 1 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-14T03:37:55.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_total_wal_size: 0 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_open_files: -1 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bytes_per_sync: 0 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_readahead_size: 0 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_background_flushes: -1 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Compression algorithms supported: 2024-09-14T03:37:55.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: kZSTD supported: 0 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: kXpressCompression supported: 0 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: kBZip2Compression supported: 0 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: kLZ4Compression supported: 1 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: kZlibCompression supported: 1 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: kLZ4HCCompression supported: 1 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: kSnappyCompression supported: 1 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.merge_operator: 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_filter: None 2024-09-14T03:37:55.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_filter_factory: None 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.sst_partitioner_factory: None 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562a26453a38) 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout: cache_index_and_filter_blocks: 1 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout: pin_top_level_index_and_filter: 1 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_type: 0 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout: data_block_index_type: 0 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_shortening: 1 2024-09-14T03:37:55.170 INFO:journalctl@ceph.mon.b.smithi160.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-14T03:37:55.171 INFO:journalctl@ceph.mon.b.smithi160.stdout: hash_index_allow_collision: 1 2024-09-14T03:37:55.171 INFO:journalctl@ceph.mon.b.smithi160.stdout: checksum: 1 2024-09-14T03:37:55.171 INFO:journalctl@ceph.mon.b.smithi160.stdout: no_block_cache: 0 2024-09-14T03:37:55.171 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache: 0x562a26489350 2024-09-14T03:37:55.171 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_name: BinnedLRUCache 2024-09-14T03:37:55.171 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_options: 2024-09-14T03:37:55.171 INFO:journalctl@ceph.mon.b.smithi160.stdout: capacity : 536870912 2024-09-14T03:37:55.171 INFO:journalctl@ceph.mon.b.smithi160.stdout: num_shard_bits : 4 2024-09-14T03:37:55.171 INFO:journalctl@ceph.mon.b.smithi160.stdout: strict_capacity_limit : 0 2024-09-14T03:37:55.171 INFO:journalctl@ceph.mon.b.smithi160.stdout: high_pri_pool_ratio: 0.000 2024-09-14T03:37:55.172 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_compressed: (nil) 2024-09-14T03:37:55.172 INFO:journalctl@ceph.mon.b.smithi160.stdout: persistent_cache: (nil) 2024-09-14T03:37:55.172 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_size: 4096 2024-09-14T03:37:55.172 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_size_deviation: 10 2024-09-14T03:37:55.172 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_restart_interval: 16 2024-09-14T03:37:55.172 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_block_restart_interval: 1 2024-09-14T03:37:55.172 INFO:journalctl@ceph.mon.b.smithi160.stdout: metadata_block_size: 4096 2024-09-14T03:37:55.172 INFO:journalctl@ceph.mon.b.smithi160.stdout: partition_filters: 0 2024-09-14T03:37:55.172 INFO:journalctl@ceph.mon.b.smithi160.stdout: use_delta_encoding: 1 2024-09-14T03:37:55.172 INFO:journalctl@ceph.mon.b.smithi160.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout: whole_key_filtering: 1 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout: verify_compression: 0 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout: read_amp_bytes_per_bit: 0 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout: format_version: 4 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout: enable_index_compression: 1 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_align: 0 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.write_buffer_size: 33554432 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_write_buffer_number: 2 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compression: NoCompression 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bottommost_compression: Disabled 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.prefix_extractor: nullptr 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-14T03:37:55.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.num_levels: 7 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compression_opts.level: 32767 2024-09-14T03:37:55.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compression_opts.strategy: 0 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compression_opts.enabled: false 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.target_file_size_base: 67108864 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-14T03:37:55.175 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.arena_block_size: 4194304 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-14T03:37:55.176 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.disable_auto_compactions: 0 2024-09-14T03:37:55.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-14T03:37:55.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-14T03:37:55.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-14T03:37:55.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-14T03:37:55.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-14T03:37:55.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-14T03:37:55.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.inplace_update_support: 0 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.bloom_locality: 0 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.max_successive_merges: 0 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-14T03:37:55.178 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.paranoid_file_checks: 0 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.force_consistency_checks: 1 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.report_bg_io_stats: 0 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.ttl: 2592000 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.enable_blob_files: false 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.min_blob_size: 0 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.blob_file_size: 268435456 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-14T03:37:55.179 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285074794716, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285074795034, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1571, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726285074, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "79076f5f-dc04-4aa6-ad78-fefb2da7b4ab", "db_session_id": "6E0VCG0Y7AFRENXXZGM9"}} 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285074795391, "job": 1, "event": "recovery_finished"} 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x562a264bc700 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: DB pointer 0x562a2648c000 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** DB Stats ** 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:37:55.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** Compaction Stats [default] ** 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-09-14T03:37:55.181 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:37:55.182 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** Compaction Stats [default] ** 2024-09-14T03:37:55.182 INFO:journalctl@ceph.mon.b.smithi160.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:37:55.182 INFO:journalctl@ceph.mon.b.smithi160.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:37:55.182 INFO:journalctl@ceph.mon.b.smithi160.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-09-14T03:37:55.182 INFO:journalctl@ceph.mon.b.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:37:55.182 INFO:journalctl@ceph.mon.b.smithi160.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-14T03:37:55.182 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-14T03:37:55.182 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-14T03:37:55.182 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-14T03:37:55.182 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-14T03:37:55.183 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:37:55.183 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:37:55.183 INFO:journalctl@ceph.mon.b.smithi160.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-14T03:37:55.183 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:37:55.183 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-14T03:37:55.183 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:37:55.183 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** Compaction Stats [default] ** 2024-09-14T03:37:55.183 INFO:journalctl@ceph.mon.b.smithi160.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:37:55.183 INFO:journalctl@ceph.mon.b.smithi160.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:37:55.183 INFO:journalctl@ceph.mon.b.smithi160.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-09-14T03:37:55.184 INFO:journalctl@ceph.mon.b.smithi160.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-09-14T03:37:55.184 INFO:journalctl@ceph.mon.b.smithi160.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-14T03:37:55.184 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:37:55.184 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** Compaction Stats [default] ** 2024-09-14T03:37:55.184 INFO:journalctl@ceph.mon.b.smithi160.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:37:55.184 INFO:journalctl@ceph.mon.b.smithi160.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:37:55.184 INFO:journalctl@ceph.mon.b.smithi160.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-09-14T03:37:55.185 INFO:journalctl@ceph.mon.b.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:37:55.185 INFO:journalctl@ceph.mon.b.smithi160.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-14T03:37:55.185 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-14T03:37:55.185 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-14T03:37:55.185 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-14T03:37:55.185 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: using public_addr v2:172.21.15.160:0/0 -> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: starting mon.b rank -1 at public addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] at bind addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(???) e0 preinit fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).mds e1 new map 2024-09-14T03:37:55.186 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout: e1 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout: legacy client fscid: -1 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout: No filesystems configured 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-14T03:37:55.187 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2832947111' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: osdmap e4: 0 total, 0 up, 0 in 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi160:172.21.15.160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi160:172.21.15.160=b;count:3 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:37:55.188 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: Deploying daemon mon.b on smithi160 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/196651785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 ceph-mon[32702]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 bash[32664]: a30277b5b152463ff7ddd7c9ab8c3067a2a9f7d5bc83262baadf3ce513dd5b2d 2024-09-14T03:37:55.189 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:54 smithi160 systemd[1]: Started Ceph mon.b for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:37:55.297 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-14T03:37:55.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-14T03:37:55.547 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:38:00.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: mon.a calling monitor election 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: mon.b calling monitor election 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: monmap e2: 2 mons at {a=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: fsmap 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: osdmap e4: 0 total, 0 up, 0 in 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: mgrmap e13: y(active, since 38s) 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: overall HEALTH_OK 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:00.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:37:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:00.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:38:00.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: mon.a calling monitor election 2024-09-14T03:38:00.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:00.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: mon.b calling monitor election 2024-09-14T03:38:00.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:00.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: monmap e2: 2 mons at {a=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: fsmap 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: osdmap e4: 0 total, 0 up, 0 in 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: mgrmap e13: y(active, since 38s) 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: overall HEALTH_OK 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:00.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:37:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:00.281 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:38:00.281 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":2,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","modified":"2024-09-14T03:37:54.822185Z","created":"2024-09-14T03:36:27.521144Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-14T03:38:00.281 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 2 2024-09-14T03:38:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:00 smithi160 ceph-mon[32702]: Deploying daemon mon.c on smithi114 2024-09-14T03:38:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:00 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/171018201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-14T03:38:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:00 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:01.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:00 smithi114 ceph-mon[29589]: Deploying daemon mon.c on smithi114 2024-09-14T03:38:01.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:00 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/171018201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-14T03:38:01.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:00 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:01.869 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-14T03:38:01.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-14T03:38:02.110 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:38:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:01 smithi160 ceph-mon[32702]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:02.224 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:01 smithi114 ceph-mon[29589]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:02 smithi114 podman[40923]: 2024-09-14 03:38:02.804058356 +0000 UTC m=+0.128505928 container create f8446e48bf23137090eed5dc9a29f49a9a408c3811149877395dba8e4154f601 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=quincy-eb16b97, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-09-14T03:38:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 podman[40923]: 2024-09-14 03:38:03.006407281 +0000 UTC m=+0.330854814 container init f8446e48bf23137090eed5dc9a29f49a9a408c3811149877395dba8e4154f601 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=quincy-eb16b97, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-14T03:38:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 podman[40923]: 2024-09-14 03:38:03.011443642 +0000 UTC m=+0.335891170 container start f8446e48bf23137090eed5dc9a29f49a9a408c3811149877395dba8e4154f601 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=quincy-eb16b97, GIT_CLEAN=True) 2024-09-14T03:38:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: set uid:gid to 167:167 (ceph:ceph) 2024-09-14T03:38:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable), process ceph-mon, pid 7 2024-09-14T03:38:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: pidfile_write: ignore empty --pid-file 2024-09-14T03:38:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: load: jerasure load: lrc 2024-09-14T03:38:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: RocksDB version: 6.15.5 2024-09-14T03:38:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-14T03:38:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Compile date Sep 13 2024 2024-09-14T03:38:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: DB SUMMARY 2024-09-14T03:38:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: DB Session ID: 1ZK0MIOVCS8EYXAQ553U 2024-09-14T03:38:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: CURRENT file: CURRENT 2024-09-14T03:38:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: IDENTITY file: IDENTITY 2024-09-14T03:38:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-14T03:38:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.error_if_exists: 0 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.create_if_missing: 0 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.paranoid_checks: 1 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.env: 0x560d4ac5b440 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.fs: Posix File System 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.info_log: 0x560d4b9daec0 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_file_opening_threads: 16 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.statistics: (nil) 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.use_fsync: 0 2024-09-14T03:38:03.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_log_file_size: 0 2024-09-14T03:38:03.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.keep_log_file_num: 1000 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.recycle_log_file_num: 0 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.allow_fallocate: 1 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.allow_mmap_reads: 0 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.allow_mmap_writes: 0 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.use_direct_reads: 0 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.create_missing_column_families: 0 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.db_log_dir: 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-14T03:38:03.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.advise_random_on_open: 1 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.db_write_buffer_size: 0 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.write_buffer_manager: 0x560d4ba987b0 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-14T03:38:03.376 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.rate_limiter: (nil) 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.wal_recovery_mode: 2 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.enable_thread_tracking: 0 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.enable_pipelined_write: 0 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.unordered_write: 0 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-14T03:38:03.377 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.row_cache: None 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.wal_filter: None 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.allow_ingest_behind: 0 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.preserve_deletes: 0 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.two_write_queues: 0 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.manual_wal_flush: 0 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.atomic_flush: 0 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.log_readahead_size: 0 2024-09-14T03:38:03.378 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.best_efforts_recovery: 0 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.allow_data_in_errors: 0 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.db_host_id: __hostname__ 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_background_jobs: 2 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_background_compactions: -1 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_subcompactions: 1 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_total_wal_size: 0 2024-09-14T03:38:03.379 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_open_files: -1 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bytes_per_sync: 0 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_readahead_size: 0 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_background_flushes: -1 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Compression algorithms supported: 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: kZSTD supported: 0 2024-09-14T03:38:03.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: kXpressCompression supported: 0 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: kBZip2Compression supported: 0 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: kLZ4Compression supported: 1 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: kZlibCompression supported: 1 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: kLZ4HCCompression supported: 1 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: kSnappyCompression supported: 1 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.merge_operator: 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_filter: None 2024-09-14T03:38:03.381 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_filter_factory: None 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.sst_partitioner_factory: None 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560d4b981a38) 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_type: 0 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: data_block_index_type: 0 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_shortening: 1 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: hash_index_allow_collision: 1 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: checksum: 1 2024-09-14T03:38:03.382 INFO:journalctl@ceph.mon.c.smithi114.stdout: no_block_cache: 0 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache: 0x560d4b9b71f0 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_options: 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: capacity : 536870912 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: num_shard_bits : 4 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: strict_capacity_limit : 0 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_compressed: (nil) 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: persistent_cache: (nil) 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_size: 4096 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_size_deviation: 10 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_restart_interval: 16 2024-09-14T03:38:03.383 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_block_restart_interval: 1 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout: metadata_block_size: 4096 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout: partition_filters: 0 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout: use_delta_encoding: 1 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout: whole_key_filtering: 1 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout: verify_compression: 0 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout: format_version: 4 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout: enable_index_compression: 1 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_align: 0 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.write_buffer_size: 33554432 2024-09-14T03:38:03.384 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_write_buffer_number: 2 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compression: NoCompression 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bottommost_compression: Disabled 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.prefix_extractor: nullptr 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.num_levels: 7 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-14T03:38:03.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compression_opts.level: 32767 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compression_opts.strategy: 0 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compression_opts.enabled: false 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-14T03:38:03.386 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.target_file_size_base: 67108864 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-14T03:38:03.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.arena_block_size: 4194304 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.disable_auto_compactions: 0 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-14T03:38:03.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.inplace_update_support: 0 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.bloom_locality: 0 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.max_successive_merges: 0 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-14T03:38:03.389 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.paranoid_file_checks: 0 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.force_consistency_checks: 1 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.report_bg_io_stats: 0 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.ttl: 2592000 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.enable_blob_files: false 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.min_blob_size: 0 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.blob_file_size: 268435456 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-14T03:38:03.390 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-14T03:38:03.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-14T03:38:03.391 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285083035304, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285083035599, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1696, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726285083, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "2a6f2c61-5a28-4a8c-93dc-83e36b501fc4", "db_session_id": "1ZK0MIOVCS8EYXAQ553U"}} 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285083035925, "job": 1, "event": "recovery_finished"} 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x560d4b9ea700 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: DB pointer 0x560d4b9ba000 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** DB Stats ** 2024-09-14T03:38:03.392 INFO:journalctl@ceph.mon.c.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-14T03:38:03.393 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:38:03.394 INFO:journalctl@ceph.mon.c.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:38:03.395 INFO:journalctl@ceph.mon.c.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-14T03:38:03.396 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: using public_addrv [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: starting mon.c rank -1 at public addrs [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] at bind addrs [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(???) e0 preinit fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).mds e1 new map 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout: e1 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout: legacy client fscid: -1 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout: No filesystems configured 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-14T03:38:03.397 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2832947111' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: osdmap e4: 0 total, 0 up, 0 in 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:03.398 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi160:172.21.15.160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi114:[v2:172.21.15.114:3301,v1:172.21.15.114:6790]=c;smithi160:172.21.15.160=b;count:3 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: Deploying daemon mon.b on smithi160 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/196651785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:03.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.a calling monitor election 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.b calling monitor election 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:03.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: monmap e2: 2 mons at {a=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: fsmap 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: osdmap e4: 0 total, 0 up, 0 in 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mgrmap e13: y(active, since 38s) 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: overall HEALTH_OK 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: Deploying daemon mon.c on smithi114 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/171018201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-14T03:38:03.401 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:03.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:03.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 ceph-mon[40961]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-14T03:38:03.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 bash[40923]: f8446e48bf23137090eed5dc9a29f49a9a408c3811149877395dba8e4154f601 2024-09-14T03:38:03.402 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:03 smithi114 systemd[1]: Started Ceph mon.c for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:38:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:38:08.310 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":3,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","modified":"2024-09-14T03:38:03.057799Z","created":"2024-09-14T03:36:27.521144Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3301","nonce":0},{"type":"v1","addr":"172.21.15.114:6790","nonce":0}]},"addr":"172.21.15.114:6790/0","public_addr":"172.21.15.114:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-14T03:38:08.310 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 3 2024-09-14T03:38:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:38:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: mon.a calling monitor election 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: mon.b calling monitor election 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: mon.c calling monitor election 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:08.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:38:08.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: monmap e3: 3 mons at {a=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0],c=[v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-14T03:38:08.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: fsmap 2024-09-14T03:38:08.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: osdmap e4: 0 total, 0 up, 0 in 2024-09-14T03:38:08.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: mgrmap e13: y(active, since 47s) 2024-09-14T03:38:08.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: overall HEALTH_OK 2024-09-14T03:38:08.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:08 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:38:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: mon.a calling monitor election 2024-09-14T03:38:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: mon.b calling monitor election 2024-09-14T03:38:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: mon.c calling monitor election 2024-09-14T03:38:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: monmap e3: 3 mons at {a=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0],c=[v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: fsmap 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: osdmap e4: 0 total, 0 up, 0 in 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: mgrmap e13: y(active, since 47s) 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: overall HEALTH_OK 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: mon.a calling monitor election 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: mon.b calling monitor election 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: mon.c calling monitor election 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:08.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: monmap e3: 3 mons at {a=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],b=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0],c=[v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: fsmap 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: osdmap e4: 0 total, 0 up, 0 in 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: mgrmap e13: y(active, since 47s) 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: overall HEALTH_OK 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:08 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:08.869 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-14T03:38:08.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-14T03:38:09.220 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:38:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:09 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:09 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/3554664209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-14T03:38:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:09 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:09.472 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:09 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:09.472 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:09 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/3554664209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-14T03:38:09.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:09 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:09.473 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:09 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:09.473 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:09 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/3554664209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-14T03:38:09.473 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:09 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:38:10.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:38:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:38:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:11.065 INFO:teuthology.orchestra.run.smithi114.stdout:# minimal ceph.conf for 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:38:11.065 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-09-14T03:38:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:38:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: mon_host = [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] 2024-09-14T03:38:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:11 smithi160 ceph-mon[32702]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:38:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:11 smithi160 ceph-mon[32702]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:38:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:11 smithi160 ceph-mon[32702]: Reconfiguring mon.a (unknown last config time)... 2024-09-14T03:38:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:11 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:11.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:11 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:38:11.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:11 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:11.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:11 smithi160 ceph-mon[32702]: Reconfiguring daemon mon.a on smithi114 2024-09-14T03:38:11.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:11 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/246442553' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:11.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:11 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:11.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:38:11.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[29589]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[29589]: Reconfiguring mon.a (unknown last config time)... 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[29589]: Reconfiguring daemon mon.a on smithi114 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/246442553' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[40961]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[40961]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[40961]: Reconfiguring mon.a (unknown last config time)... 2024-09-14T03:38:11.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:11.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:38:11.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:11.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[40961]: Reconfiguring daemon mon.a on smithi114 2024-09-14T03:38:11.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/246442553' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:11.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:11 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:11.641 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-14T03:38:11.641 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:38:11.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-09-14T03:38:11.668 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:38:11.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:38:11.734 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:38:11.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.conf 2024-09-14T03:38:11.764 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:38:11.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:38:11.832 INFO:tasks.cephadm:Adding mgr.y on smithi114 2024-09-14T03:38:11.832 INFO:tasks.cephadm:Adding mgr.x on smithi160 2024-09-14T03:38:11.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi114=y;smithi160=x' 2024-09-14T03:38:12.109 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:38:12.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:12 smithi160 ceph-mon[32702]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:12.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:12 smithi114 ceph-mon[29589]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:12.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:12 smithi114 ceph-mon[40961]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:13.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:13 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:13.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:13 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:13.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:13 smithi160 ceph-mon[32702]: Reconfiguring mon.c (monmap changed)... 2024-09-14T03:38:13.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:13 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:13.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:13 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:38:13.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:13 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:13.618 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:13 smithi160 ceph-mon[32702]: Reconfiguring daemon mon.c on smithi114 2024-09-14T03:38:13.792 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mgr update... 2024-09-14T03:38:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[29589]: Reconfiguring mon.c (monmap changed)... 2024-09-14T03:38:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:38:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[29589]: Reconfiguring daemon mon.c on smithi114 2024-09-14T03:38:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:13.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:13.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[40961]: Reconfiguring mon.c (monmap changed)... 2024-09-14T03:38:13.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:13.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:38:13.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:13.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:13 smithi114 ceph-mon[40961]: Reconfiguring daemon mon.c on smithi114 2024-09-14T03:38:14.345 DEBUG:teuthology.orchestra.run.smithi160:mgr.x> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.x.service 2024-09-14T03:38:14.347 DEBUG:tasks.cephadm:set 0 configs 2024-09-14T03:38:14.347 INFO:tasks.cephadm:Deploying OSDs... 2024-09-14T03:38:14.347 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:38:14.348 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-09-14T03:38:14.364 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-14T03:38:14.364 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-09-14T03:38:14.419 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-14T03:38:14.420 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T03:38:14.420 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-14T03:38:14.420 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T03:38:14.420 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T03:38:14.420 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-14 03:38:02.518691077 +0000 2024-09-14T03:38:14.420 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-14 03:33:24.784541738 +0000 2024-09-14T03:38:14.420 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-14 03:33:24.784541738 +0000 2024-09-14T03:38:14.420 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-14 03:33:24.784541738 +0000 2024-09-14T03:38:14.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-14T03:38:14.485 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-14T03:38:14.485 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-14T03:38:14.485 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000194218 s, 2.6 MB/s 2024-09-14T03:38:14.486 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-14T03:38:14.543 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-09-14T03:38:14.598 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-14T03:38:14.598 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T03:38:14.598 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-14T03:38:14.598 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T03:38:14.599 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T03:38:14.599 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-14 03:38:02.519691057 +0000 2024-09-14T03:38:14.599 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-14 03:33:24.785541720 +0000 2024-09-14T03:38:14.599 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-14 03:33:24.785541720 +0000 2024-09-14T03:38:14.599 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-14 03:33:24.785541720 +0000 2024-09-14T03:38:14.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-14T03:38:14.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:14 smithi160 ceph-mon[32702]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:14.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:14 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:14.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:14 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:14.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:14 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:14.613 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:14 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:14.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:14 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:38:14.614 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:14 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:14.661 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-14T03:38:14.661 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-14T03:38:14.661 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.00017437 s, 2.9 MB/s 2024-09-14T03:38:14.662 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-14T03:38:14.718 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-09-14T03:38:14.773 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-14T03:38:14.773 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T03:38:14.773 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 968 Links: 1 2024-09-14T03:38:14.774 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T03:38:14.774 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T03:38:14.774 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-14 03:38:02.519691057 +0000 2024-09-14T03:38:14.774 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-14 03:33:24.783541756 +0000 2024-09-14T03:38:14.774 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-14 03:33:24.783541756 +0000 2024-09-14T03:38:14.774 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-14 03:33:24.783541756 +0000 2024-09-14T03:38:14.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-14T03:38:14.835 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[40961]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:14.835 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:14.835 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[29589]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:38:14.836 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:14 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-14T03:38:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-14T03:38:14.837 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000190903 s, 2.7 MB/s 2024-09-14T03:38:14.838 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-14T03:38:14.895 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-09-14T03:38:14.950 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-14T03:38:14.950 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T03:38:14.950 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 961 Links: 1 2024-09-14T03:38:14.950 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T03:38:14.950 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T03:38:14.950 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-14 03:38:02.519691057 +0000 2024-09-14T03:38:14.951 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-14 03:33:24.782541774 +0000 2024-09-14T03:38:14.951 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-14 03:33:24.782541774 +0000 2024-09-14T03:38:14.951 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-14 03:33:24.782541774 +0000 2024-09-14T03:38:14.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-14T03:38:15.013 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-14T03:38:15.014 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-14T03:38:15.014 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000254468 s, 2.0 MB/s 2024-09-14T03:38:15.015 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-14T03:38:15.071 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:38:15.071 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2024-09-14T03:38:15.088 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-14T03:38:15.088 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 2024-09-14T03:38:15.144 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-14T03:38:15.145 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T03:38:15.145 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 950 Links: 1 2024-09-14T03:38:15.145 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T03:38:15.145 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T03:38:15.145 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-14 03:37:53.833257686 +0000 2024-09-14T03:38:15.145 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-14 03:33:26.055050857 +0000 2024-09-14T03:38:15.145 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-14 03:33:26.055050857 +0000 2024-09-14T03:38:15.145 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-14 03:33:26.055050857 +0000 2024-09-14T03:38:15.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-14T03:38:15.212 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-14T03:38:15.212 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-14T03:38:15.212 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000255937 s, 2.0 MB/s 2024-09-14T03:38:15.214 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-14T03:38:15.270 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 2024-09-14T03:38:15.325 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-14T03:38:15.325 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T03:38:15.325 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 946 Links: 1 2024-09-14T03:38:15.325 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T03:38:15.325 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T03:38:15.326 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-14 03:37:53.833257686 +0000 2024-09-14T03:38:15.326 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-14 03:33:26.055050857 +0000 2024-09-14T03:38:15.326 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-14 03:33:26.055050857 +0000 2024-09-14T03:38:15.326 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-14 03:33:26.055050857 +0000 2024-09-14T03:38:15.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-14T03:38:15.388 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-14T03:38:15.389 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-14T03:38:15.389 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000195192 s, 2.6 MB/s 2024-09-14T03:38:15.390 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-14T03:38:15.446 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 2024-09-14T03:38:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-14T03:38:15.502 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T03:38:15.502 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 952 Links: 1 2024-09-14T03:38:15.503 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T03:38:15.503 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T03:38:15.503 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-14 03:37:53.834257672 +0000 2024-09-14T03:38:15.503 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-14 03:33:26.055050857 +0000 2024-09-14T03:38:15.503 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-14 03:33:26.055050857 +0000 2024-09-14T03:38:15.503 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-14 03:33:26.055050857 +0000 2024-09-14T03:38:15.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-14T03:38:15.565 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-14T03:38:15.565 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-14T03:38:15.566 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000222174 s, 2.3 MB/s 2024-09-14T03:38:15.567 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-14T03:38:15.623 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 2024-09-14T03:38:15.679 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-14T03:38:15.679 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T03:38:15.679 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 962 Links: 1 2024-09-14T03:38:15.679 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T03:38:15.679 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T03:38:15.679 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-14 03:37:53.834257672 +0000 2024-09-14T03:38:15.679 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-14 03:33:26.056050843 +0000 2024-09-14T03:38:15.680 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-14 03:33:26.056050843 +0000 2024-09-14T03:38:15.680 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-09-14 03:33:26.056050843 +0000 2024-09-14T03:38:15.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-14T03:38:15.744 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-14T03:38:15.744 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-14T03:38:15.744 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000215802 s, 2.4 MB/s 2024-09-14T03:38:15.745 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-14T03:38:15.803 INFO:tasks.cephadm:Deploying osd.0 on smithi114 with /dev/vg_nvme/lv_4... 2024-09-14T03:38:15.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-14T03:38:15.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:15 smithi114 ceph-mon[29589]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi114=y;smithi160=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:38:15.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:15 smithi114 ceph-mon[29589]: Saving service mgr spec with placement smithi114=y;smithi160=x;count:2 2024-09-14T03:38:15.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:15 smithi114 ceph-mon[29589]: Reconfiguring mon.b (monmap changed)... 2024-09-14T03:38:15.827 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:15 smithi114 ceph-mon[29589]: Reconfiguring daemon mon.b on smithi160 2024-09-14T03:38:15.827 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:15 smithi114 ceph-mon[40961]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi114=y;smithi160=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:38:15.827 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:15 smithi114 ceph-mon[40961]: Saving service mgr spec with placement smithi114=y;smithi160=x;count:2 2024-09-14T03:38:15.827 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:15 smithi114 ceph-mon[40961]: Reconfiguring mon.b (monmap changed)... 2024-09-14T03:38:15.827 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:15 smithi114 ceph-mon[40961]: Reconfiguring daemon mon.b on smithi160 2024-09-14T03:38:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:15 smithi160 ceph-mon[32702]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi114=y;smithi160=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:38:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:15 smithi160 ceph-mon[32702]: Saving service mgr spec with placement smithi114=y;smithi160=x;count:2 2024-09-14T03:38:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:15 smithi160 ceph-mon[32702]: Reconfiguring mon.b (monmap changed)... 2024-09-14T03:38:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:15 smithi160 ceph-mon[32702]: Reconfiguring daemon mon.b on smithi160 2024-09-14T03:38:16.037 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:38:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[29589]: Deploying daemon mgr.x on smithi160 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-14T03:38:16.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:38:16.720 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:16.720 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:16 smithi114 ceph-mon[40961]: Deploying daemon mgr.x on smithi160 2024-09-14T03:38:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:16.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:38:16.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-14T03:38:16.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:38:16.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:16.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:16 smithi160 ceph-mon[32702]: Deploying daemon mgr.x on smithi160 2024-09-14T03:38:17.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:17 smithi160 podman[34729]: 2024-09-14 03:38:17.486734925 +0000 UTC m=+0.015928226 image pull ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6 2024-09-14T03:38:18.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:17 smithi160 podman[34729]: 2024-09-14 03:38:17.803218073 +0000 UTC m=+0.332411374 container init ad77a9b0ace61c11658bc59acc32b7f91486f78fd19aac360fc6ddf9b3b83cbe (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=quincy-eb16b97, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-14T03:38:18.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:17 smithi160 podman[34729]: 2024-09-14 03:38:17.808343309 +0000 UTC m=+0.337536613 container start ad77a9b0ace61c11658bc59acc32b7f91486f78fd19aac360fc6ddf9b3b83cbe (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-14T03:38:18.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:17 smithi160 ceph-mgr[34767]: -- 172.21.15.160:0/3556794346 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564bf6fe4340 con 0x564bf6ff0800 2024-09-14T03:38:18.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:17 smithi160 bash[34729]: ad77a9b0ace61c11658bc59acc32b7f91486f78fd19aac360fc6ddf9b3b83cbe 2024-09-14T03:38:18.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:17 smithi160 systemd[1]: Started Ceph mgr.x for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:38:18.159 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:17.919+0000 7f0172199ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-14T03:38:18.159 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:17.980+0000 7f0172199ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-14T03:38:18.159 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:18.046+0000 7f0172199ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-14T03:38:18.263 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:38:18.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-09-14T03:38:18.558 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:38:18.608 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:18.177+0000 7f0172199ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:38:18.608 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:18.330+0000 7f0172199ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-14T03:38:18.608 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:18.606+0000 7f0172199ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-14T03:38:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-mon[32702]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:18.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:18.673+0000 7f0172199ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-14T03:38:18.909 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:18.848+0000 7f0172199ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-14T03:38:18.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[40961]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:18.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:18.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[29589]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:18.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.912 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.912 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:18.912 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:20.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:19.711+0000 7f0172199ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-14T03:38:20.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:19.775+0000 7f0172199ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-14T03:38:20.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:20.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:20.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:20.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:20.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:20.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:20.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: Reconfiguring mgr.y (unknown last config time)... 2024-09-14T03:38:20.483 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:38:20.484 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:38:20.484 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:20.484 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-mon[32702]: Reconfiguring daemon mgr.y on smithi114 2024-09-14T03:38:20.545 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:20.545 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:20.545 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:20.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:20.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:20.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:20.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: Reconfiguring mgr.y (unknown last config time)... 2024-09-14T03:38:20.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:38:20.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:38:20.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:20.546 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[29589]: Reconfiguring daemon mgr.y on smithi114 2024-09-14T03:38:20.546 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:20.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:20.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:20.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:20.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:20.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:20.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: Reconfiguring mgr.y (unknown last config time)... 2024-09-14T03:38:20.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:38:20.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:38:20.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:20.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:20 smithi114 ceph-mon[40961]: Reconfiguring daemon mgr.y on smithi114 2024-09-14T03:38:20.769 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:20.480+0000 7f0172199ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-14T03:38:20.769 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:20.590+0000 7f0172199ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-14T03:38:20.769 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:20.708+0000 7f0172199ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:38:21.149 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:20.766+0000 7f0172199ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-14T03:38:21.149 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:20.890+0000 7f0172199ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-14T03:38:21.149 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:20.956+0000 7f0172199ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-14T03:38:21.149 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:21.013+0000 7f0172199ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-14T03:38:21.149 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:21.146+0000 7f0172199ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-14T03:38:21.298 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:21 smithi114 ceph-mon[29589]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:38:21.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:38:21.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:38:21.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:21.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:21.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:21 smithi114 ceph-mon[40961]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:38:21.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:21 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:38:21.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:21 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:38:21.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:21 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:21.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:21 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:21.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:21.209+0000 7f0172199ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-14T03:38:21.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:21.275+0000 7f0172199ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-14T03:38:21.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:21.355+0000 7f0172199ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-14T03:38:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-mon[32702]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:38:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:38:21.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:38:21.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:21.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:21.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:21.414+0000 7f0172199ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-14T03:38:21.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:21.473+0000 7f0172199ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-14T03:38:22.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:22.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:22.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:22.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:22.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:22.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:22.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:22.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: Standby manager daemon x started 2024-09-14T03:38:22.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:38:22.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:38:22.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:38:22.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:38:22.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:22.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:22.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:22.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: Standby manager daemon x started 2024-09-14T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:38:22.553 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:38:22.554 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:38:22.554 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:22 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:38:22.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:38:22.214+0000 7f0172199ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-14T03:38:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: Standby manager daemon x started 2024-09-14T03:38:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:38:22.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:38:22.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:38:22.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:22 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/3224586800' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:38:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:23 smithi160 ceph-mon[32702]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-14T03:38:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:23 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:38:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:23 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1267707888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a021a3a-45d2-4825-a701-d7c3793ef6b2"}]: dispatch 2024-09-14T03:38:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:23 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a021a3a-45d2-4825-a701-d7c3793ef6b2"}]: dispatch 2024-09-14T03:38:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:23 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a021a3a-45d2-4825-a701-d7c3793ef6b2"}]': finished 2024-09-14T03:38:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:23 smithi160 ceph-mon[32702]: osdmap e5: 1 total, 0 up, 1 in 2024-09-14T03:38:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:23 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:23.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[40961]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-14T03:38:23.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:38:23.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1267707888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a021a3a-45d2-4825-a701-d7c3793ef6b2"}]: dispatch 2024-09-14T03:38:23.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a021a3a-45d2-4825-a701-d7c3793ef6b2"}]: dispatch 2024-09-14T03:38:23.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a021a3a-45d2-4825-a701-d7c3793ef6b2"}]': finished 2024-09-14T03:38:23.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[40961]: osdmap e5: 1 total, 0 up, 1 in 2024-09-14T03:38:23.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:23.699 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[29589]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-14T03:38:23.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:38:23.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1267707888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a021a3a-45d2-4825-a701-d7c3793ef6b2"}]: dispatch 2024-09-14T03:38:23.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a021a3a-45d2-4825-a701-d7c3793ef6b2"}]: dispatch 2024-09-14T03:38:23.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a021a3a-45d2-4825-a701-d7c3793ef6b2"}]': finished 2024-09-14T03:38:23.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[29589]: osdmap e5: 1 total, 0 up, 1 in 2024-09-14T03:38:23.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:23 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:24 smithi160 ceph-mon[32702]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:24 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/216042987' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:38:24.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:24 smithi114 ceph-mon[29589]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:24.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:24 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/216042987' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:38:24.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:24 smithi114 ceph-mon[40961]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:24.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:24 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/216042987' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:38:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:26 smithi160 ceph-mon[32702]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:26.662 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:26 smithi114 ceph-mon[29589]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:26.662 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:26 smithi114 ceph-mon[40961]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:28.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:28 smithi114 ceph-mon[29589]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:28.405 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:28 smithi114 ceph-mon[40961]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:28 smithi160 ceph-mon[32702]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:29 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-14T03:38:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:29 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:29.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:29 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-14T03:38:29.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:29 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:29.735 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:29 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-14T03:38:29.735 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:29 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:30 smithi160 ceph-mon[32702]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:30 smithi160 ceph-mon[32702]: Deploying daemon osd.0 on smithi114 2024-09-14T03:38:30.716 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:30 smithi114 ceph-mon[29589]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:30.717 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:30 smithi114 ceph-mon[29589]: Deploying daemon osd.0 on smithi114 2024-09-14T03:38:30.717 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:30 smithi114 ceph-mon[40961]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:30.717 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:30 smithi114 ceph-mon[40961]: Deploying daemon osd.0 on smithi114 2024-09-14T03:38:32.595 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:32 smithi114 ceph-mon[29589]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:32.595 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:32 smithi114 ceph-mon[40961]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:32 smithi160 ceph-mon[32702]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:34.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:34 smithi114 ceph-mon[40961]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:34.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:34.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:34.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:34.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:34 smithi114 ceph-mon[29589]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:34.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:34.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:34.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:34 smithi160 ceph-mon[32702]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:36 smithi160 ceph-mon[32702]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:36 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:36 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:36.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:36 smithi114 ceph-mon[40961]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:36.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:36 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:36.659 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:36 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:36.659 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:36 smithi114 ceph-mon[29589]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:36.659 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:36 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:36.659 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:36 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:37.591 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:37.591 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:37.591 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:37.591 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:37 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:37.592 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:37 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:37.592 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:37 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 0 on host 'smithi114' 2024-09-14T03:38:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:37 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:37 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:37 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:38.268 DEBUG:teuthology.orchestra.run.smithi114:osd.0> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.0.service 2024-09-14T03:38:38.271 INFO:tasks.cephadm:Deploying osd.1 on smithi114 with /dev/vg_nvme/lv_3... 2024-09-14T03:38:38.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-14T03:38:38.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:38 smithi114 ceph-mon[40961]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:38.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:38 smithi114 ceph-mon[40961]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-14T03:38:38.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:38.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:38.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:38.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:38 smithi114 ceph-mon[29589]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:38.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:38 smithi114 ceph-mon[29589]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-14T03:38:38.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:38.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:38.497 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:38 smithi160 ceph-mon[32702]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:38 smithi160 ceph-mon[32702]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-14T03:38:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:39.086 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:38:39.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:39 smithi114 ceph-mon[29589]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-14T03:38:39.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:39 smithi114 ceph-mon[29589]: osdmap e6: 1 total, 0 up, 1 in 2024-09-14T03:38:39.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:39 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:39.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:39 smithi114 ceph-mon[29589]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:38:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:39 smithi114 ceph-mon[40961]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-14T03:38:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:39 smithi114 ceph-mon[40961]: osdmap e6: 1 total, 0 up, 1 in 2024-09-14T03:38:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:39 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:39.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:39 smithi114 ceph-mon[40961]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:38:39.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:38:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[45233]: 2024-09-14T03:38:39.412+0000 7f7c07f6f640 -1 osd.0 0 waiting for initial osdmap 2024-09-14T03:38:39.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:38:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[45233]: 2024-09-14T03:38:39.415+0000 7f7c035a7640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:38:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:39 smithi160 ceph-mon[32702]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-14T03:38:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:39 smithi160 ceph-mon[32702]: osdmap e6: 1 total, 0 up, 1 in 2024-09-14T03:38:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:39 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:39 smithi160 ceph-mon[32702]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:38:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[40961]: purged_snaps scrub starts 2024-09-14T03:38:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[40961]: purged_snaps scrub ok 2024-09-14T03:38:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[40961]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[40961]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:38:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[40961]: osdmap e7: 1 total, 0 up, 1 in 2024-09-14T03:38:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:40.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[29589]: purged_snaps scrub starts 2024-09-14T03:38:40.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[29589]: purged_snaps scrub ok 2024-09-14T03:38:40.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[29589]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:40.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[29589]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:38:40.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[29589]: osdmap e7: 1 total, 0 up, 1 in 2024-09-14T03:38:40.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:40.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:40 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:40.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:40 smithi160 ceph-mon[32702]: purged_snaps scrub starts 2024-09-14T03:38:40.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:40 smithi160 ceph-mon[32702]: purged_snaps scrub ok 2024-09-14T03:38:40.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:40 smithi160 ceph-mon[32702]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:40 smithi160 ceph-mon[32702]: from='osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:38:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:40 smithi160 ceph-mon[32702]: osdmap e7: 1 total, 0 up, 1 in 2024-09-14T03:38:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:40 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:40 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:41 smithi160 ceph-mon[32702]: osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523] boot 2024-09-14T03:38:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:41 smithi160 ceph-mon[32702]: osdmap e8: 1 total, 1 up, 1 in 2024-09-14T03:38:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:41 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:41.697 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:41 smithi114 ceph-mon[29589]: osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523] boot 2024-09-14T03:38:41.697 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:41 smithi114 ceph-mon[29589]: osdmap e8: 1 total, 1 up, 1 in 2024-09-14T03:38:41.697 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:41 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:41.698 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:41 smithi114 ceph-mon[40961]: osd.0 [v2:172.21.15.114:6802/2918802523,v1:172.21.15.114:6803/2918802523] boot 2024-09-14T03:38:41.698 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:41 smithi114 ceph-mon[40961]: osdmap e8: 1 total, 1 up, 1 in 2024-09-14T03:38:41.698 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:41 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:38:43.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:43.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: Detected new or changed devices on smithi114 2024-09-14T03:38:43.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi114 to 16064M 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[29589]: osdmap e9: 1 total, 1 up, 1 in 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: Detected new or changed devices on smithi114 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.047 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:38:43.048 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi114 to 16064M 2024-09-14T03:38:43.048 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.048 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:43.048 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:43.048 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.048 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:42 smithi114 ceph-mon[40961]: osdmap e9: 1 total, 1 up, 1 in 2024-09-14T03:38:43.052 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:38:43.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-09-14T03:38:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-14T03:38:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: Detected new or changed devices on smithi114 2024-09-14T03:38:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:38:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi114 to 16064M 2024-09-14T03:38:43.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:43.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:38:43.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:43.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:42 smithi160 ceph-mon[32702]: osdmap e9: 1 total, 1 up, 1 in 2024-09-14T03:38:43.316 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:38:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:44 smithi114 ceph-mon[29589]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:44 smithi114 ceph-mon[40961]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:44 smithi160 ceph-mon[32702]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:46.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:45 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:38:46.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:45 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:38:46.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:45 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:46.077 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:45 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:38:46.077 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:45 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:38:46.077 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:45 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:45 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:38:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:45 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:38:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:45 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:46.971 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:46 smithi114 ceph-mon[29589]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:38:46.971 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:46 smithi114 ceph-mon[29589]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:46.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:46 smithi114 ceph-mon[40961]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:38:46.971 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:46 smithi114 ceph-mon[40961]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:46 smithi160 ceph-mon[32702]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:38:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:46 smithi160 ceph-mon[32702]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:48.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:47 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/607675143' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "076291c4-5927-456d-bf74-2ab79be4c756"}]: dispatch 2024-09-14T03:38:48.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:47 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "076291c4-5927-456d-bf74-2ab79be4c756"}]: dispatch 2024-09-14T03:38:48.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:47 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "076291c4-5927-456d-bf74-2ab79be4c756"}]': finished 2024-09-14T03:38:48.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:47 smithi114 ceph-mon[40961]: osdmap e10: 2 total, 1 up, 2 in 2024-09-14T03:38:48.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:47 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:38:48.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:47 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/607675143' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "076291c4-5927-456d-bf74-2ab79be4c756"}]: dispatch 2024-09-14T03:38:48.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:47 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "076291c4-5927-456d-bf74-2ab79be4c756"}]: dispatch 2024-09-14T03:38:48.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:47 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "076291c4-5927-456d-bf74-2ab79be4c756"}]': finished 2024-09-14T03:38:48.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:47 smithi114 ceph-mon[29589]: osdmap e10: 2 total, 1 up, 2 in 2024-09-14T03:38:48.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:47 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:38:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:47 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/607675143' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "076291c4-5927-456d-bf74-2ab79be4c756"}]: dispatch 2024-09-14T03:38:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:47 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "076291c4-5927-456d-bf74-2ab79be4c756"}]: dispatch 2024-09-14T03:38:48.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:47 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "076291c4-5927-456d-bf74-2ab79be4c756"}]': finished 2024-09-14T03:38:48.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:47 smithi160 ceph-mon[32702]: osdmap e10: 2 total, 1 up, 2 in 2024-09-14T03:38:48.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:47 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:38:49.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:48 smithi114 ceph-mon[29589]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:49.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:48 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3313574181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:38:49.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:48 smithi114 ceph-mon[40961]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:49.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:48 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3313574181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:38:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:48 smithi160 ceph-mon[32702]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:48 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3313574181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:38:50.998 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:50 smithi114 ceph-mon[40961]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:50.998 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:50 smithi114 ceph-mon[29589]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:51.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:50 smithi160 ceph-mon[32702]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:52.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:52 smithi114 ceph-mon[40961]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:52.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:52 smithi114 ceph-mon[29589]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:52 smithi160 ceph-mon[32702]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:53.779 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:53 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-14T03:38:53.779 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:53 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:53.779 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:53 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-14T03:38:53.779 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:53 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:53 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-14T03:38:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:53 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:38:55.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:54 smithi114 ceph-mon[29589]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:55.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:54 smithi114 ceph-mon[29589]: Deploying daemon osd.1 on smithi114 2024-09-14T03:38:55.014 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:54 smithi114 ceph-mon[40961]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:55.014 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:54 smithi114 ceph-mon[40961]: Deploying daemon osd.1 on smithi114 2024-09-14T03:38:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:54 smithi160 ceph-mon[32702]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:54 smithi160 ceph-mon[32702]: Deploying daemon osd.1 on smithi114 2024-09-14T03:38:55.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:55 smithi114 ceph-mon[29589]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:55.945 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:55 smithi114 ceph-mon[40961]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:55 smithi160 ceph-mon[32702]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:58.178 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:58 smithi114 ceph-mon[29589]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:58.178 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:58 smithi114 ceph-mon[40961]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:58 smithi160 ceph-mon[32702]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:38:59.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:59 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:59 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:38:59 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:38:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:38:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:38:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:38:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:00.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:00 smithi114 ceph-mon[29589]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:39:00.447 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:00 smithi114 ceph-mon[40961]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:39:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:00 smithi160 ceph-mon[32702]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:39:01.710 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 1 on host 'smithi114' 2024-09-14T03:39:02.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:02.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:02.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[29589]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-14T03:39:02.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:02.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:02.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:02.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[40961]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-14T03:39:02.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:02.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:01 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:01 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:01 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:01 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:01 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:01 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:01 smithi160 ceph-mon[32702]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-14T03:39:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:01 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:01 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:01 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:02.420 DEBUG:teuthology.orchestra.run.smithi114:osd.1> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.1.service 2024-09-14T03:39:02.422 INFO:tasks.cephadm:Deploying osd.2 on smithi114 with /dev/vg_nvme/lv_2... 2024-09-14T03:39:02.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-14T03:39:03.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-mon[29589]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:39:03.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-mon[29589]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-14T03:39:03.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-mon[29589]: osdmap e11: 2 total, 1 up, 2 in 2024-09-14T03:39:03.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:03.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-mon[29589]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:03.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-mon[40961]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:39:03.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-mon[40961]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-14T03:39:03.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-mon[40961]: osdmap e11: 2 total, 1 up, 2 in 2024-09-14T03:39:03.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:03.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-mon[40961]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:03.122 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[51524]: 2024-09-14T03:39:02.743+0000 7f89072a7640 -1 osd.1 0 waiting for initial osdmap 2024-09-14T03:39:03.123 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:39:02 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[51524]: 2024-09-14T03:39:02.748+0000 7f89028df640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:39:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:02 smithi160 ceph-mon[32702]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:39:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:02 smithi160 ceph-mon[32702]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-14T03:39:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:02 smithi160 ceph-mon[32702]: osdmap e11: 2 total, 1 up, 2 in 2024-09-14T03:39:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:02 smithi160 ceph-mon[32702]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:03.494 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:39:04.000 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:03 smithi114 ceph-mon[29589]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:39:04.000 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:03 smithi114 ceph-mon[29589]: osdmap e12: 2 total, 1 up, 2 in 2024-09-14T03:39:04.000 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:03 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:04.000 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:03 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:04.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:03 smithi114 ceph-mon[40961]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:39:04.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:03 smithi114 ceph-mon[40961]: osdmap e12: 2 total, 1 up, 2 in 2024-09-14T03:39:04.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:04.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:03 smithi160 ceph-mon[32702]: from='osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:39:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:03 smithi160 ceph-mon[32702]: osdmap e12: 2 total, 1 up, 2 in 2024-09-14T03:39:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:03 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:03 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:04.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[40961]: purged_snaps scrub starts 2024-09-14T03:39:04.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[40961]: purged_snaps scrub ok 2024-09-14T03:39:04.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[40961]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[40961]: osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589] boot 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[40961]: osdmap e13: 2 total, 2 up, 2 in 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[29589]: purged_snaps scrub starts 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[29589]: purged_snaps scrub ok 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[29589]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[29589]: osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589] boot 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[29589]: osdmap e13: 2 total, 2 up, 2 in 2024-09-14T03:39:04.949 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:04 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:04 smithi160 ceph-mon[32702]: purged_snaps scrub starts 2024-09-14T03:39:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:04 smithi160 ceph-mon[32702]: purged_snaps scrub ok 2024-09-14T03:39:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:04 smithi160 ceph-mon[32702]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-14T03:39:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:04 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:04 smithi160 ceph-mon[32702]: osd.1 [v2:172.21.15.114:6810/1989151589,v1:172.21.15.114:6811/1989151589] boot 2024-09-14T03:39:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:04 smithi160 ceph-mon[32702]: osdmap e13: 2 total, 2 up, 2 in 2024-09-14T03:39:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:04 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:39:05.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: osdmap e14: 2 total, 2 up, 2 in 2024-09-14T03:39:05.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:05.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: Detected new or changed devices on smithi114 2024-09-14T03:39:05.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:05.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:05.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:05.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:05.960 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi114 to 8032M 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: osdmap e14: 2 total, 2 up, 2 in 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: Detected new or changed devices on smithi114 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:05.961 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:05.962 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi114 to 8032M 2024-09-14T03:39:05.962 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:05.962 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:05.962 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:05.962 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: osdmap e14: 2 total, 2 up, 2 in 2024-09-14T03:39:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: Detected new or changed devices on smithi114 2024-09-14T03:39:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:06.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi114 to 8032M 2024-09-14T03:39:06.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:06.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:06.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:06.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:07.038 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:39:07.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-09-14T03:39:07.297 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:39:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:08 smithi160 ceph-mon[32702]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:08.447 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:08 smithi114 ceph-mon[40961]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:08.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:08 smithi114 ceph-mon[29589]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:09 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:39:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:09 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:39:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:09 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:09.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:09 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:39:09.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:09 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:39:09.491 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:09 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:09.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:09 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:39:09.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:09 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:39:09.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:09 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:10.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:10 smithi160 ceph-mon[32702]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:10.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:10 smithi160 ceph-mon[32702]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:39:10.481 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:10 smithi114 ceph-mon[29589]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:10.482 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:10 smithi114 ceph-mon[29589]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:39:10.482 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:10 smithi114 ceph-mon[40961]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:10.482 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:10 smithi114 ceph-mon[40961]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:39:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[29589]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3999510774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6787400-73ea-4edf-bb17-440ac36bd4ff"}]: dispatch 2024-09-14T03:39:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6787400-73ea-4edf-bb17-440ac36bd4ff"}]: dispatch 2024-09-14T03:39:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6787400-73ea-4edf-bb17-440ac36bd4ff"}]': finished 2024-09-14T03:39:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[29589]: osdmap e15: 3 total, 2 up, 3 in 2024-09-14T03:39:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/4102988363' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:39:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[40961]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:12.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3999510774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6787400-73ea-4edf-bb17-440ac36bd4ff"}]: dispatch 2024-09-14T03:39:12.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6787400-73ea-4edf-bb17-440ac36bd4ff"}]: dispatch 2024-09-14T03:39:12.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6787400-73ea-4edf-bb17-440ac36bd4ff"}]': finished 2024-09-14T03:39:12.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[40961]: osdmap e15: 3 total, 2 up, 3 in 2024-09-14T03:39:12.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:12.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:12 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/4102988363' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:39:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:12 smithi160 ceph-mon[32702]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:12 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3999510774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6787400-73ea-4edf-bb17-440ac36bd4ff"}]: dispatch 2024-09-14T03:39:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:12 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6787400-73ea-4edf-bb17-440ac36bd4ff"}]: dispatch 2024-09-14T03:39:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:12 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6787400-73ea-4edf-bb17-440ac36bd4ff"}]': finished 2024-09-14T03:39:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:12 smithi160 ceph-mon[32702]: osdmap e15: 3 total, 2 up, 3 in 2024-09-14T03:39:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:12 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:12 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/4102988363' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:39:14.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:14 smithi114 ceph-mon[29589]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:14.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:14 smithi114 ceph-mon[40961]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:14 smithi160 ceph-mon[32702]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:16.212 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:16 smithi114 ceph-mon[29589]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:16.213 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:16 smithi114 ceph-mon[40961]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:16 smithi160 ceph-mon[32702]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:18 smithi160 ceph-mon[32702]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-14T03:39:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:18 smithi160 ceph-mon[32702]: Deploying daemon osd.2 on smithi114 2024-09-14T03:39:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:18 smithi114 ceph-mon[29589]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-14T03:39:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:18.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:18 smithi114 ceph-mon[29589]: Deploying daemon osd.2 on smithi114 2024-09-14T03:39:18.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:18 smithi114 ceph-mon[40961]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:18.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-14T03:39:18.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:18.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:18 smithi114 ceph-mon[40961]: Deploying daemon osd.2 on smithi114 2024-09-14T03:39:20.202 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:20 smithi114 ceph-mon[29589]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:20.203 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:20 smithi114 ceph-mon[40961]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:20 smithi160 ceph-mon[32702]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:22.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:22 smithi114 ceph-mon[29589]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:22 smithi114 ceph-mon[40961]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:22 smithi160 ceph-mon[32702]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:23 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:23 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:23 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:23 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:23 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:23 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:23 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:23 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:23 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:24.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:24 smithi114 ceph-mon[29589]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:24.439 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:24 smithi114 ceph-mon[40961]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:24 smithi160 ceph-mon[32702]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:25.352 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 2 on host 'smithi114' 2024-09-14T03:39:26.155 DEBUG:teuthology.orchestra.run.smithi114:osd.2> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.2.service 2024-09-14T03:39:26.157 INFO:tasks.cephadm:Deploying osd.3 on smithi114 with /dev/vg_nvme/lv_1... 2024-09-14T03:39:26.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-14T03:39:26.479 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:39:26.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:26 smithi114 ceph-mon[40961]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:26.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:26.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:26.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:26 smithi114 ceph-mon[40961]: from='osd.2 [v2:172.21.15.114:6818/2560140870,v1:172.21.15.114:6819/2560140870]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-14T03:39:26.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:26 smithi114 ceph-mon[40961]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-14T03:39:26.601 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:26 smithi114 ceph-mon[29589]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:26.601 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:26.601 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:26.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:26 smithi114 ceph-mon[29589]: from='osd.2 [v2:172.21.15.114:6818/2560140870,v1:172.21.15.114:6819/2560140870]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-14T03:39:26.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:26 smithi114 ceph-mon[29589]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-14T03:39:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:26 smithi160 ceph-mon[32702]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:26 smithi160 ceph-mon[32702]: from='osd.2 [v2:172.21.15.114:6818/2560140870,v1:172.21.15.114:6819/2560140870]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-14T03:39:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:26 smithi160 ceph-mon[32702]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-14T03:39:27.588 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[29589]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-14T03:39:27.588 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[29589]: osdmap e16: 3 total, 2 up, 3 in 2024-09-14T03:39:27.588 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:27.589 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[29589]: from='osd.2 [v2:172.21.15.114:6818/2560140870,v1:172.21.15.114:6819/2560140870]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:27.589 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[29589]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:27.589 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:27.589 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:27.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[57613]: 2024-09-14T03:39:27.356+0000 7f9805ced640 -1 osd.2 0 waiting for initial osdmap 2024-09-14T03:39:27.589 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[57613]: 2024-09-14T03:39:27.359+0000 7f9800b24640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:39:27.589 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[40961]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-14T03:39:27.589 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[40961]: osdmap e16: 3 total, 2 up, 3 in 2024-09-14T03:39:27.590 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:27.590 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[40961]: from='osd.2 [v2:172.21.15.114:6818/2560140870,v1:172.21.15.114:6819/2560140870]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:27.590 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[40961]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:27.590 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:27.590 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:27 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:27 smithi160 ceph-mon[32702]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-14T03:39:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:27 smithi160 ceph-mon[32702]: osdmap e16: 3 total, 2 up, 3 in 2024-09-14T03:39:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:27 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:27 smithi160 ceph-mon[32702]: from='osd.2 [v2:172.21.15.114:6818/2560140870,v1:172.21.15.114:6819/2560140870]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:27 smithi160 ceph-mon[32702]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:27 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:27 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:28 smithi114 ceph-mon[29589]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:28 smithi114 ceph-mon[29589]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:39:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:28 smithi114 ceph-mon[29589]: osdmap e17: 3 total, 2 up, 3 in 2024-09-14T03:39:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:28 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:28 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:28.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:28 smithi114 ceph-mon[40961]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:28.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:28 smithi114 ceph-mon[40961]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:39:28.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:28 smithi114 ceph-mon[40961]: osdmap e17: 3 total, 2 up, 3 in 2024-09-14T03:39:28.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:28 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:28.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:28 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:28 smithi160 ceph-mon[32702]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:28 smithi160 ceph-mon[32702]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:39:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:28 smithi160 ceph-mon[32702]: osdmap e17: 3 total, 2 up, 3 in 2024-09-14T03:39:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:28 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:28 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:29.347 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:39:29.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_1 2024-09-14T03:39:29.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[29589]: purged_snaps scrub starts 2024-09-14T03:39:29.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[29589]: purged_snaps scrub ok 2024-09-14T03:39:29.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[29589]: osd.2 [v2:172.21.15.114:6818/2560140870,v1:172.21.15.114:6819/2560140870] boot 2024-09-14T03:39:29.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[29589]: osdmap e18: 3 total, 3 up, 3 in 2024-09-14T03:39:29.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:29.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-14T03:39:29.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[40961]: purged_snaps scrub starts 2024-09-14T03:39:29.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[40961]: purged_snaps scrub ok 2024-09-14T03:39:29.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[40961]: osd.2 [v2:172.21.15.114:6818/2560140870,v1:172.21.15.114:6819/2560140870] boot 2024-09-14T03:39:29.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[40961]: osdmap e18: 3 total, 3 up, 3 in 2024-09-14T03:39:29.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:29.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:29 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-14T03:39:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:29 smithi160 ceph-mon[32702]: purged_snaps scrub starts 2024-09-14T03:39:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:29 smithi160 ceph-mon[32702]: purged_snaps scrub ok 2024-09-14T03:39:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:29 smithi160 ceph-mon[32702]: osd.2 [v2:172.21.15.114:6818/2560140870,v1:172.21.15.114:6819/2560140870] boot 2024-09-14T03:39:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:29 smithi160 ceph-mon[32702]: osdmap e18: 3 total, 3 up, 3 in 2024-09-14T03:39:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:29 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:39:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:29 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-14T03:39:30.252 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:39:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:30 smithi160 ceph-mon[32702]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:30 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-14T03:39:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:30 smithi160 ceph-mon[32702]: osdmap e19: 3 total, 3 up, 3 in 2024-09-14T03:39:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:30 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:39:30.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:30 smithi114 ceph-mon[29589]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:30.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:30 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-14T03:39:30.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:30 smithi114 ceph-mon[29589]: osdmap e19: 3 total, 3 up, 3 in 2024-09-14T03:39:30.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:30 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:39:30.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:30 smithi114 ceph-mon[40961]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-14T03:39:30.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:30 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-14T03:39:30.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:30 smithi114 ceph-mon[40961]: osdmap e19: 3 total, 3 up, 3 in 2024-09-14T03:39:30.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:30 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:39:30.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:39:30 smithi114 sudo[60846]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-14T03:39:30.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:39:30 smithi114 sudo[60846]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:30.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:39:30 smithi114 sudo[60846]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:30.950 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:39:30 smithi114 sudo[60846]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:39:30 smithi114 sudo[61910]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-14T03:39:31.215 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:39:30 smithi114 sudo[61910]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:39:30 smithi114 sudo[61910]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:31.216 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:39:30 smithi114 sudo[61910]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:31.476 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:31 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:39:31.476 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:31 smithi114 ceph-mon[40961]: osdmap e20: 3 total, 3 up, 3 in 2024-09-14T03:39:31.476 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:31 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:31.476 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:31 smithi114 ceph-mon[40961]: osdmap e21: 3 total, 3 up, 3 in 2024-09-14T03:39:31.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:31 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:39:31.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:31 smithi114 ceph-mon[29589]: osdmap e20: 3 total, 3 up, 3 in 2024-09-14T03:39:31.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:31 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:31.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:31 smithi114 ceph-mon[29589]: osdmap e21: 3 total, 3 up, 3 in 2024-09-14T03:39:31.477 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61913]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-14T03:39:31.477 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61913]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:31.477 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61913]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:31.477 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61913]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:31.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:31 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:39:31.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:31 smithi160 ceph-mon[32702]: osdmap e20: 3 total, 3 up, 3 in 2024-09-14T03:39:31.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:31 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:31.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:31 smithi160 ceph-mon[32702]: osdmap e21: 3 total, 3 up, 3 in 2024-09-14T03:39:31.729 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61929]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-14T03:39:31.729 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61929]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:31.729 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61929]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:31.729 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61929]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:31.992 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61965]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-14T03:39:31.992 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61965]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:31.992 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61965]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:31.992 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[61965]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:32.247 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[62002]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-14T03:39:32.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[62002]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:32.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[62002]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:32.248 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:39:31 smithi114 sudo[62002]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:32 smithi114 sudo[62025]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-14T03:39:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:32 smithi114 sudo[62025]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:32 smithi114 sudo[62025]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:32 smithi114 sudo[62025]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:32.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:32 smithi114 sudo[62048]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-14T03:39:32.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:32 smithi114 sudo[62048]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:32.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:32 smithi114 sudo[62048]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:32.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:32 smithi114 sudo[62048]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:33.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:32 smithi160 sudo[35417]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-14T03:39:33.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:32 smithi160 sudo[35417]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:33.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:32 smithi160 sudo[35417]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:33.011 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:32 smithi160 sudo[35417]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: Detected new or changed devices on smithi114 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi114 to 5354M 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:33.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:39:33.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:39:33.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:39:33.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:33.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: Detected new or changed devices on smithi114 2024-09-14T03:39:33.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:33.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:33.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:33.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi114 to 5354M 2024-09-14T03:39:33.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:39:33.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:39:33.290 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:39:33.291 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:39:33.291 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:39:33.291 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 sudo[62199]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-14T03:39:33.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 sudo[35420]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 sudo[35420]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 sudo[35420]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 sudo[35420]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: Detected new or changed devices on smithi114 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi114 to 5354M 2024-09-14T03:39:33.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:39:33.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:39:33.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:39:33.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:39:33.546 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 sudo[62199]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:33.546 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 sudo[62199]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:33.546 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 sudo[62199]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:33.836 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 sudo[62240]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-14T03:39:33.836 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 sudo[62240]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-14T03:39:33.836 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 sudo[62240]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-14T03:39:33.836 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:33 smithi114 sudo[62240]: pam_unix(sudo:session): session closed for user root 2024-09-14T03:39:34.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[40961]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:39:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:39:34.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:34.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:34 smithi114 ceph-mon[29589]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-14T03:39:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-14T03:39:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:39:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:39:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:39:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-14T03:39:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-14T03:39:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:39:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:39:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:39:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:34 smithi160 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-14T03:39:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:35 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/44541439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ead755f3-173a-4465-a28f-af7288b0a42b"}]: dispatch 2024-09-14T03:39:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:35 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/44541439' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ead755f3-173a-4465-a28f-af7288b0a42b"}]': finished 2024-09-14T03:39:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:35 smithi114 ceph-mon[29589]: osdmap e22: 4 total, 3 up, 4 in 2024-09-14T03:39:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:35 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:35.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:35 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/44541439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ead755f3-173a-4465-a28f-af7288b0a42b"}]: dispatch 2024-09-14T03:39:35.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:35 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/44541439' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ead755f3-173a-4465-a28f-af7288b0a42b"}]': finished 2024-09-14T03:39:35.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:35 smithi114 ceph-mon[40961]: osdmap e22: 4 total, 3 up, 4 in 2024-09-14T03:39:35.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:35 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:35 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/44541439' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ead755f3-173a-4465-a28f-af7288b0a42b"}]: dispatch 2024-09-14T03:39:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:35 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/44541439' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ead755f3-173a-4465-a28f-af7288b0a42b"}]': finished 2024-09-14T03:39:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:35 smithi160 ceph-mon[32702]: osdmap e22: 4 total, 3 up, 4 in 2024-09-14T03:39:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:35 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:36.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:36 smithi114 ceph-mon[29589]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:36.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:36 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3638714752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:39:36.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:36 smithi114 ceph-mon[40961]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:36.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:36 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3638714752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:39:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:36 smithi160 ceph-mon[32702]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:36 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3638714752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:39:38.562 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:38 smithi114 ceph-mon[40961]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:38.562 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:38 smithi114 ceph-mon[29589]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:38 smithi160 ceph-mon[32702]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:40.456 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:40 smithi114 ceph-mon[29589]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:40.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:40 smithi114 ceph-mon[40961]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:40 smithi160 ceph-mon[32702]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:41.442 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:41 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-14T03:39:41.443 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:41 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:41.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:41 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-14T03:39:41.443 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:41 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:41 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-14T03:39:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:41 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:42.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:42 smithi114 ceph-mon[29589]: Deploying daemon osd.3 on smithi114 2024-09-14T03:39:42.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:42 smithi114 ceph-mon[29589]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:42 smithi114 ceph-mon[40961]: Deploying daemon osd.3 on smithi114 2024-09-14T03:39:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:42 smithi114 ceph-mon[40961]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:42 smithi160 ceph-mon[32702]: Deploying daemon osd.3 on smithi114 2024-09-14T03:39:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:42 smithi160 ceph-mon[32702]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:44.467 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:44 smithi114 ceph-mon[29589]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:44.467 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:44 smithi114 ceph-mon[40961]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:44 smithi160 ceph-mon[32702]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:46.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:46 smithi114 ceph-mon[29589]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:46.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:46.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:46.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:46.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:46 smithi114 ceph-mon[40961]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:46.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:46 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:46.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:46 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:46.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:46 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:46.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:46 smithi160 ceph-mon[32702]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:46.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:46 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:46.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:46 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:46.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:46 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:48.596 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:48 smithi114 ceph-mon[29589]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:48.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:48 smithi114 ceph-mon[40961]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:48 smithi160 ceph-mon[32702]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:49.240 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 3 on host 'smithi114' 2024-09-14T03:39:49.519 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:49 smithi114 ceph-mon[29589]: from='osd.3 [v2:172.21.15.114:6826/3048125105,v1:172.21.15.114:6827/3048125105]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-14T03:39:49.519 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:49 smithi114 ceph-mon[29589]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-14T03:39:49.520 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:49 smithi114 ceph-mon[40961]: from='osd.3 [v2:172.21.15.114:6826/3048125105,v1:172.21.15.114:6827/3048125105]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-14T03:39:49.520 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:49 smithi114 ceph-mon[40961]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-14T03:39:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:49 smithi160 ceph-mon[32702]: from='osd.3 [v2:172.21.15.114:6826/3048125105,v1:172.21.15.114:6827/3048125105]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-14T03:39:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:49 smithi160 ceph-mon[32702]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-14T03:39:49.819 DEBUG:teuthology.orchestra.run.smithi114:osd.3> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.3.service 2024-09-14T03:39:49.822 INFO:tasks.cephadm:Deploying osd.4 on smithi160 with /dev/vg_nvme/lv_4... 2024-09-14T03:39:49.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-14T03:39:50.073 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:39:50.463 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:50 smithi160 ceph-mon[32702]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:50.463 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:50 smithi160 ceph-mon[32702]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-14T03:39:50.463 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:50 smithi160 ceph-mon[32702]: osdmap e23: 4 total, 3 up, 4 in 2024-09-14T03:39:50.463 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:50 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:50.464 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:50 smithi160 ceph-mon[32702]: from='osd.3 [v2:172.21.15.114:6826/3048125105,v1:172.21.15.114:6827/3048125105]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:50.464 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:50 smithi160 ceph-mon[32702]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:50.464 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:50 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.464 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:50 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.464 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:50 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.464 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:50 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[29589]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[29589]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-14T03:39:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[29589]: osdmap e23: 4 total, 3 up, 4 in 2024-09-14T03:39:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[29589]: from='osd.3 [v2:172.21.15.114:6826/3048125105,v1:172.21.15.114:6827/3048125105]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[29589]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.622 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[64118]: 2024-09-14T03:39:50.201+0000 7f2e09557640 -1 osd.3 0 waiting for initial osdmap 2024-09-14T03:39:50.622 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[64118]: 2024-09-14T03:39:50.205+0000 7f2e0438e640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:39:50.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[40961]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:50.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[40961]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-14T03:39:50.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[40961]: osdmap e23: 4 total, 3 up, 4 in 2024-09-14T03:39:50.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:50.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[40961]: from='osd.3 [v2:172.21.15.114:6826/3048125105,v1:172.21.15.114:6827/3048125105]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:50.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[40961]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-14T03:39:50.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:50.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:50 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:51 smithi114 ceph-mon[29589]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:39:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:51 smithi114 ceph-mon[29589]: osdmap e24: 4 total, 3 up, 4 in 2024-09-14T03:39:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:51 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:51 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:51 smithi114 ceph-mon[40961]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:39:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:51 smithi114 ceph-mon[40961]: osdmap e24: 4 total, 3 up, 4 in 2024-09-14T03:39:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:51 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:51 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:51.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:51 smithi160 ceph-mon[32702]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-14T03:39:51.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:51 smithi160 ceph-mon[32702]: osdmap e24: 4 total, 3 up, 4 in 2024-09-14T03:39:51.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:51 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:51.460 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:51 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:52.447 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:52 smithi160 ceph-mon[32702]: purged_snaps scrub starts 2024-09-14T03:39:52.447 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:52 smithi160 ceph-mon[32702]: purged_snaps scrub ok 2024-09-14T03:39:52.447 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:52 smithi160 ceph-mon[32702]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:52.447 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:52 smithi160 ceph-mon[32702]: osd.3 [v2:172.21.15.114:6826/3048125105,v1:172.21.15.114:6827/3048125105] boot 2024-09-14T03:39:52.448 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:52 smithi160 ceph-mon[32702]: osdmap e25: 4 total, 4 up, 4 in 2024-09-14T03:39:52.448 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:52 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:52.452 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:39:52.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_4 2024-09-14T03:39:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[29589]: purged_snaps scrub starts 2024-09-14T03:39:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[29589]: purged_snaps scrub ok 2024-09-14T03:39:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[29589]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[29589]: osd.3 [v2:172.21.15.114:6826/3048125105,v1:172.21.15.114:6827/3048125105] boot 2024-09-14T03:39:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[29589]: osdmap e25: 4 total, 4 up, 4 in 2024-09-14T03:39:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[40961]: purged_snaps scrub starts 2024-09-14T03:39:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[40961]: purged_snaps scrub ok 2024-09-14T03:39:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[40961]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-14T03:39:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[40961]: osd.3 [v2:172.21.15.114:6826/3048125105,v1:172.21.15.114:6827/3048125105] boot 2024-09-14T03:39:52.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[40961]: osdmap e25: 4 total, 4 up, 4 in 2024-09-14T03:39:52.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:52 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:39:52.708 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:39:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:53 smithi160 ceph-mon[32702]: osdmap e26: 4 total, 4 up, 4 in 2024-09-14T03:39:53.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:53 smithi114 ceph-mon[40961]: osdmap e26: 4 total, 4 up, 4 in 2024-09-14T03:39:53.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:53 smithi114 ceph-mon[29589]: osdmap e26: 4 total, 4 up, 4 in 2024-09-14T03:39:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:39:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: Detected new or changed devices on smithi114 2024-09-14T03:39:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi114 to 4016M 2024-09-14T03:39:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: Detected new or changed devices on smithi114 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi114 to 4016M 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:54.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:54.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:54.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:39:54.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: Detected new or changed devices on smithi114 2024-09-14T03:39:54.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi114 to 4016M 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:39:54.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:39:55.862 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:55 smithi160 ceph-mon[32702]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:39:55.862 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:55 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:39:55.862 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:55 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:39:55.862 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:55 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:55.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:55 smithi114 ceph-mon[29589]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:39:55.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:55 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:39:55.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:55 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:39:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:55 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:55 smithi114 ceph-mon[40961]: from='client.24184 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:39:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:55 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:39:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:55 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:39:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:55 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:39:56.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:56 smithi114 ceph-mon[29589]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:39:56.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:56 smithi114 ceph-mon[40961]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:39:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:56 smithi160 ceph-mon[32702]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:39:57.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:57 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/561383399' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60a0aac2-e235-484f-b25f-f112c83da7f7"}]: dispatch 2024-09-14T03:39:57.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:57 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60a0aac2-e235-484f-b25f-f112c83da7f7"}]: dispatch 2024-09-14T03:39:57.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:57 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60a0aac2-e235-484f-b25f-f112c83da7f7"}]': finished 2024-09-14T03:39:57.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:57 smithi160 ceph-mon[32702]: osdmap e27: 5 total, 4 up, 5 in 2024-09-14T03:39:57.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:57 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:39:57.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:57 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/1730237795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:39:57.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/561383399' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60a0aac2-e235-484f-b25f-f112c83da7f7"}]: dispatch 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60a0aac2-e235-484f-b25f-f112c83da7f7"}]: dispatch 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60a0aac2-e235-484f-b25f-f112c83da7f7"}]': finished 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[40961]: osdmap e27: 5 total, 4 up, 5 in 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/1730237795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/561383399' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60a0aac2-e235-484f-b25f-f112c83da7f7"}]: dispatch 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60a0aac2-e235-484f-b25f-f112c83da7f7"}]: dispatch 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60a0aac2-e235-484f-b25f-f112c83da7f7"}]': finished 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[29589]: osdmap e27: 5 total, 4 up, 5 in 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:39:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:57 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/1730237795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:39:58.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:39:58 smithi114 ceph-mon[29589]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:39:58.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:39:58 smithi114 ceph-mon[40961]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:39:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:39:58 smithi160 ceph-mon[32702]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:00.756 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:00 smithi160 ceph-mon[32702]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:00.756 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:00 smithi160 ceph-mon[32702]: overall HEALTH_OK 2024-09-14T03:40:00.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:00 smithi114 ceph-mon[40961]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:00.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:00 smithi114 ceph-mon[40961]: overall HEALTH_OK 2024-09-14T03:40:00.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:00 smithi114 ceph-mon[29589]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:00.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:00 smithi114 ceph-mon[29589]: overall HEALTH_OK 2024-09-14T03:40:02.641 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:02 smithi160 ceph-mon[32702]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:02.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:02 smithi114 ceph-mon[29589]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:02 smithi114 ceph-mon[40961]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:03 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-14T03:40:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:03 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:03 smithi160 ceph-mon[32702]: Deploying daemon osd.4 on smithi160 2024-09-14T03:40:03.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:03 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-14T03:40:03.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:03 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:03.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:03 smithi114 ceph-mon[29589]: Deploying daemon osd.4 on smithi160 2024-09-14T03:40:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-14T03:40:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:03 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:03 smithi114 ceph-mon[40961]: Deploying daemon osd.4 on smithi160 2024-09-14T03:40:04.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:04 smithi160 ceph-mon[32702]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:04.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:04 smithi114 ceph-mon[29589]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:04.768 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:04 smithi114 ceph-mon[40961]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:06.844 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:06 smithi160 ceph-mon[32702]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:06.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:06 smithi114 ceph-mon[29589]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:06.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:06 smithi114 ceph-mon[40961]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:07.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:07 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:07.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:07 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:07.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:07 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:07.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:07 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:07.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:07 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:07.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:07 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:07.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:07 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:07.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:07 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:07.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:07 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:08.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:08 smithi114 ceph-mon[29589]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:08 smithi114 ceph-mon[40961]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:08 smithi160 ceph-mon[32702]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:10 smithi160 ceph-mon[32702]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:10 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:10 smithi160 ceph-mon[32702]: from='osd.4 [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-14T03:40:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:10 smithi160 ceph-mon[32702]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-14T03:40:10.564 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 4 on host 'smithi160' 2024-09-14T03:40:10.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[40961]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:10.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:10.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:10.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:10.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:10.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:10.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[40961]: from='osd.4 [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-14T03:40:10.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[40961]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-14T03:40:10.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[29589]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:10.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:10.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:10.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:10.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:10.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:10.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[29589]: from='osd.4 [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-14T03:40:10.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:10 smithi114 ceph-mon[29589]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-14T03:40:11.161 DEBUG:teuthology.orchestra.run.smithi160:osd.4> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.4.service 2024-09-14T03:40:11.163 INFO:tasks.cephadm:Deploying osd.5 on smithi160 with /dev/vg_nvme/lv_3... 2024-09-14T03:40:11.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-14T03:40:11.676 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:11 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:11.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:11 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:11.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:11 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:11.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:11 smithi160 ceph-mon[32702]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-14T03:40:11.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:11 smithi160 ceph-mon[32702]: osdmap e28: 5 total, 4 up, 5 in 2024-09-14T03:40:11.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:11 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:11.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:11 smithi160 ceph-mon[32702]: from='osd.4 [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:11.677 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:11 smithi160 ceph-mon[32702]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:11.833 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:40:11.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:11.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:11.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:11.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[29589]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-14T03:40:11.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[29589]: osdmap e28: 5 total, 4 up, 5 in 2024-09-14T03:40:11.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:11.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[29589]: from='osd.4 [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:11.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[29589]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:11.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:11.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:11.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:11.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[40961]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-14T03:40:11.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[40961]: osdmap e28: 5 total, 4 up, 5 in 2024-09-14T03:40:11.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:11.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[40961]: from='osd.4 [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:11.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:11 smithi114 ceph-mon[40961]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:12.565 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 03:40:12 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[37995]: 2024-09-14T03:40:12.318+0000 7fc174b4b640 -1 osd.4 0 waiting for initial osdmap 2024-09-14T03:40:12.565 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 03:40:12 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[37995]: 2024-09-14T03:40:12.322+0000 7fc16e95e640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:40:12.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:12 smithi160 ceph-mon[32702]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:12.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:12 smithi160 ceph-mon[32702]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:40:12.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:12 smithi160 ceph-mon[32702]: osdmap e29: 5 total, 4 up, 5 in 2024-09-14T03:40:12.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:12 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:12.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:12 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:12.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:12 smithi114 ceph-mon[29589]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:12.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:12 smithi114 ceph-mon[29589]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:40:12.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:12 smithi114 ceph-mon[29589]: osdmap e29: 5 total, 4 up, 5 in 2024-09-14T03:40:12.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:12 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:12.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:12 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:12.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:12 smithi114 ceph-mon[40961]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:12.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:12 smithi114 ceph-mon[40961]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:40:12.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:12 smithi114 ceph-mon[40961]: osdmap e29: 5 total, 4 up, 5 in 2024-09-14T03:40:12.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:12 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:12.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:12 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:14.368 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:14 smithi160 ceph-mon[32702]: purged_snaps scrub starts 2024-09-14T03:40:14.368 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:14 smithi160 ceph-mon[32702]: purged_snaps scrub ok 2024-09-14T03:40:14.368 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:14 smithi160 ceph-mon[32702]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:14.368 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:14 smithi160 ceph-mon[32702]: osd.4 [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940] boot 2024-09-14T03:40:14.368 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:14 smithi160 ceph-mon[32702]: osdmap e30: 5 total, 5 up, 5 in 2024-09-14T03:40:14.368 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:14 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:14.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[29589]: purged_snaps scrub starts 2024-09-14T03:40:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[29589]: purged_snaps scrub ok 2024-09-14T03:40:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[29589]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[29589]: osd.4 [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940] boot 2024-09-14T03:40:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[29589]: osdmap e30: 5 total, 5 up, 5 in 2024-09-14T03:40:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[40961]: purged_snaps scrub starts 2024-09-14T03:40:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[40961]: purged_snaps scrub ok 2024-09-14T03:40:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[40961]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-14T03:40:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[40961]: osd.4 [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940] boot 2024-09-14T03:40:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[40961]: osdmap e30: 5 total, 5 up, 5 in 2024-09-14T03:40:14.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:14 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:40:15.322 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: osdmap e31: 5 total, 5 up, 5 in 2024-09-14T03:40:15.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: osdmap e31: 5 total, 5 up, 5 in 2024-09-14T03:40:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: Detected new or changed devices on smithi160 2024-09-14T03:40:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:40:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi160 to 17088M 2024-09-14T03:40:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[40961]: osdmap e32: 5 total, 5 up, 5 in 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: osdmap e31: 5 total, 5 up, 5 in 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: Detected new or changed devices on smithi160 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 17088M 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:15 smithi114 ceph-mon[29589]: osdmap e32: 5 total, 5 up, 5 in 2024-09-14T03:40:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: Detected new or changed devices on smithi160 2024-09-14T03:40:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:40:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi160 to 17088M 2024-09-14T03:40:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:15.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:15 smithi160 ceph-mon[32702]: osdmap e32: 5 total, 5 up, 5 in 2024-09-14T03:40:15.730 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:40:15.750 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_3 2024-09-14T03:40:15.993 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:40:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:16 smithi160 ceph-mon[32702]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:16.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:16 smithi114 ceph-mon[29589]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:16 smithi114 ceph-mon[40961]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:18.622 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:18 smithi160 ceph-mon[32702]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:18.623 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:18 smithi160 ceph-mon[32702]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:40:18.623 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:40:18.623 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:40:18.623 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:18.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:18 smithi114 ceph-mon[29589]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:18 smithi114 ceph-mon[29589]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:40:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:40:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:40:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:18 smithi114 ceph-mon[40961]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:18 smithi114 ceph-mon[40961]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:40:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:40:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:40:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:20 smithi160 ceph-mon[32702]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:20 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/4206881195' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f49c7de-c810-4075-b731-022bb5031d2a"}]: dispatch 2024-09-14T03:40:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:20 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f49c7de-c810-4075-b731-022bb5031d2a"}]: dispatch 2024-09-14T03:40:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:20 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f49c7de-c810-4075-b731-022bb5031d2a"}]': finished 2024-09-14T03:40:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:20 smithi160 ceph-mon[32702]: osdmap e33: 6 total, 5 up, 6 in 2024-09-14T03:40:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:20 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:20 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/654037380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:40:20.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[29589]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:20.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/4206881195' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f49c7de-c810-4075-b731-022bb5031d2a"}]: dispatch 2024-09-14T03:40:20.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f49c7de-c810-4075-b731-022bb5031d2a"}]: dispatch 2024-09-14T03:40:20.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f49c7de-c810-4075-b731-022bb5031d2a"}]': finished 2024-09-14T03:40:20.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[29589]: osdmap e33: 6 total, 5 up, 6 in 2024-09-14T03:40:20.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:20.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/654037380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:40:20.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[40961]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:20.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/4206881195' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f49c7de-c810-4075-b731-022bb5031d2a"}]: dispatch 2024-09-14T03:40:20.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f49c7de-c810-4075-b731-022bb5031d2a"}]: dispatch 2024-09-14T03:40:20.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f49c7de-c810-4075-b731-022bb5031d2a"}]': finished 2024-09-14T03:40:20.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[40961]: osdmap e33: 6 total, 5 up, 6 in 2024-09-14T03:40:20.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:20.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:20 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/654037380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:40:22.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:22 smithi114 ceph-mon[29589]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-14T03:40:22.673 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:22 smithi114 ceph-mon[40961]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-14T03:40:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:22 smithi160 ceph-mon[32702]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-14T03:40:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:24 smithi160 ceph-mon[32702]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-14T03:40:24.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:24 smithi114 ceph-mon[29589]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-14T03:40:24.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:24 smithi114 ceph-mon[40961]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-14T03:40:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:26 smithi160 ceph-mon[32702]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-14T03:40:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-14T03:40:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:26 smithi160 ceph-mon[32702]: Deploying daemon osd.5 on smithi160 2024-09-14T03:40:26.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:26 smithi114 ceph-mon[29589]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-14T03:40:26.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-14T03:40:26.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:26.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:26 smithi114 ceph-mon[29589]: Deploying daemon osd.5 on smithi160 2024-09-14T03:40:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:26 smithi114 ceph-mon[40961]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-14T03:40:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-14T03:40:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:26 smithi114 ceph-mon[40961]: Deploying daemon osd.5 on smithi160 2024-09-14T03:40:28.598 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:28 smithi160 ceph-mon[32702]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:28.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:28 smithi114 ceph-mon[40961]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:28.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:28 smithi114 ceph-mon[29589]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:30.827 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:30 smithi160 ceph-mon[32702]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:30.827 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:30 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:30.828 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:30 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:30.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:30 smithi114 ceph-mon[29589]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:30.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:30 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:30.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:30 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:30.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:30 smithi114 ceph-mon[40961]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:30.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:30 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:30.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:30 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:31.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:31 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:31.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:31 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:31.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:31 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:32.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:32 smithi160 ceph-mon[32702]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:32.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:32 smithi114 ceph-mon[29589]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:32.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:32 smithi114 ceph-mon[40961]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:33.374 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 5 on host 'smithi160' 2024-09-14T03:40:34.040 DEBUG:teuthology.orchestra.run.smithi160:osd.5> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.5.service 2024-09-14T03:40:34.043 INFO:tasks.cephadm:Deploying osd.6 on smithi160 with /dev/vg_nvme/lv_2... 2024-09-14T03:40:34.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-14T03:40:34.333 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:40:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[29589]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[29589]: from='osd.5 [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-14T03:40:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[29589]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-14T03:40:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[40961]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:34.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[40961]: from='osd.5 [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-14T03:40:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[40961]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-14T03:40:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:34 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:34 smithi160 ceph-mon[32702]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:34.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:34 smithi160 ceph-mon[32702]: from='osd.5 [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-14T03:40:34.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:34 smithi160 ceph-mon[32702]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-14T03:40:34.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:34.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:34 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:35.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:35 smithi160 ceph-mon[32702]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-14T03:40:35.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:35 smithi160 ceph-mon[32702]: osdmap e34: 6 total, 5 up, 6 in 2024-09-14T03:40:35.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:35 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:35.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:35 smithi160 ceph-mon[32702]: from='osd.5 [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:35.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:35 smithi160 ceph-mon[32702]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:35.820 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 03:40:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T03:40:35.458+0000 7f4895d0d640 -1 osd.5 0 waiting for initial osdmap 2024-09-14T03:40:35.820 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 03:40:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T03:40:35.462+0000 7f4890b44640 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:40:35.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:35 smithi114 ceph-mon[29589]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-14T03:40:35.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:35 smithi114 ceph-mon[29589]: osdmap e34: 6 total, 5 up, 6 in 2024-09-14T03:40:35.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:35 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:35.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:35 smithi114 ceph-mon[29589]: from='osd.5 [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:35.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:35 smithi114 ceph-mon[29589]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:35.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:35 smithi114 ceph-mon[40961]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-14T03:40:35.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:35 smithi114 ceph-mon[40961]: osdmap e34: 6 total, 5 up, 6 in 2024-09-14T03:40:35.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:35 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:35.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:35 smithi114 ceph-mon[40961]: from='osd.5 [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:35.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:35 smithi114 ceph-mon[40961]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:36.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:36 smithi160 ceph-mon[32702]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:36.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:36 smithi160 ceph-mon[32702]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:40:36.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:36 smithi160 ceph-mon[32702]: osdmap e35: 6 total, 5 up, 6 in 2024-09-14T03:40:36.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:36 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:36.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:36 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:36 smithi114 ceph-mon[29589]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:36 smithi114 ceph-mon[29589]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:40:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:36 smithi114 ceph-mon[29589]: osdmap e35: 6 total, 5 up, 6 in 2024-09-14T03:40:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:36 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:36 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:36.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:36 smithi114 ceph-mon[40961]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-14T03:40:36.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:36 smithi114 ceph-mon[40961]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:40:36.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:36 smithi114 ceph-mon[40961]: osdmap e35: 6 total, 5 up, 6 in 2024-09-14T03:40:36.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:36 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:36.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:36 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:37.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:37 smithi160 ceph-mon[32702]: purged_snaps scrub starts 2024-09-14T03:40:37.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:37 smithi160 ceph-mon[32702]: purged_snaps scrub ok 2024-09-14T03:40:37.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:37 smithi160 ceph-mon[32702]: osd.5 [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003] boot 2024-09-14T03:40:37.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:37 smithi160 ceph-mon[32702]: osdmap e36: 6 total, 6 up, 6 in 2024-09-14T03:40:37.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:37 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:37.819 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:37 smithi160 ceph-mon[32702]: osdmap e37: 6 total, 6 up, 6 in 2024-09-14T03:40:37.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[29589]: purged_snaps scrub starts 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[29589]: purged_snaps scrub ok 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[29589]: osd.5 [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003] boot 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[29589]: osdmap e36: 6 total, 6 up, 6 in 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[29589]: osdmap e37: 6 total, 6 up, 6 in 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[40961]: purged_snaps scrub starts 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[40961]: purged_snaps scrub ok 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[40961]: osd.5 [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003] boot 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[40961]: osdmap e36: 6 total, 6 up, 6 in 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:40:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:37 smithi114 ceph-mon[40961]: osdmap e37: 6 total, 6 up, 6 in 2024-09-14T03:40:38.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:38.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-14T03:40:38.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:40:38.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:40:38.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:38.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:38.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:38.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:38.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:38.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:38 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:38.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:38.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:38.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:40:38.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:38 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:39.399 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:40:39.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_2 2024-09-14T03:40:39.663 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:40:39.784 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:39 smithi160 ceph-mon[32702]: Detected new or changed devices on smithi160 2024-09-14T03:40:39.785 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:39 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi160 to 8544M 2024-09-14T03:40:39.785 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:39 smithi160 ceph-mon[32702]: osdmap e38: 6 total, 6 up, 6 in 2024-09-14T03:40:39.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:39 smithi114 ceph-mon[29589]: Detected new or changed devices on smithi160 2024-09-14T03:40:39.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:39 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 8544M 2024-09-14T03:40:39.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:39 smithi114 ceph-mon[29589]: osdmap e38: 6 total, 6 up, 6 in 2024-09-14T03:40:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:39 smithi114 ceph-mon[40961]: Detected new or changed devices on smithi160 2024-09-14T03:40:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:39 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi160 to 8544M 2024-09-14T03:40:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:39 smithi114 ceph-mon[40961]: osdmap e38: 6 total, 6 up, 6 in 2024-09-14T03:40:40.715 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:40 smithi160 ceph-mon[32702]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:40.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:40 smithi114 ceph-mon[29589]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:40.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:40 smithi114 ceph-mon[40961]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:41.750 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:41 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:40:41.750 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:41 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:40:41.750 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:41 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:41.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:41 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:40:41.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:41 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:40:41.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:41 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:41 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:40:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:41 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:40:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:41 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:42.649 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:42 smithi160 ceph-mon[32702]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:42.649 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:42 smithi160 ceph-mon[32702]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:40:42.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:42 smithi114 ceph-mon[29589]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:42 smithi114 ceph-mon[29589]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:40:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:42 smithi114 ceph-mon[40961]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:42 smithi114 ceph-mon[40961]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:40:43.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[40961]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-14T03:40:43.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[40961]: Cluster is now healthy 2024-09-14T03:40:43.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/4250081884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "455de335-7290-4242-b743-6b0c446683ad"}]: dispatch 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "455de335-7290-4242-b743-6b0c446683ad"}]: dispatch 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "455de335-7290-4242-b743-6b0c446683ad"}]': finished 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[40961]: osdmap e39: 7 total, 6 up, 7 in 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[29589]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[29589]: Cluster is now healthy 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/4250081884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "455de335-7290-4242-b743-6b0c446683ad"}]: dispatch 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "455de335-7290-4242-b743-6b0c446683ad"}]: dispatch 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "455de335-7290-4242-b743-6b0c446683ad"}]': finished 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[29589]: osdmap e39: 7 total, 6 up, 7 in 2024-09-14T03:40:43.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:43 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:40:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:43 smithi160 ceph-mon[32702]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-14T03:40:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:43 smithi160 ceph-mon[32702]: Cluster is now healthy 2024-09-14T03:40:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:43 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/4250081884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "455de335-7290-4242-b743-6b0c446683ad"}]: dispatch 2024-09-14T03:40:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:43 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "455de335-7290-4242-b743-6b0c446683ad"}]: dispatch 2024-09-14T03:40:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:43 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "455de335-7290-4242-b743-6b0c446683ad"}]': finished 2024-09-14T03:40:43.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:43 smithi160 ceph-mon[32702]: osdmap e39: 7 total, 6 up, 7 in 2024-09-14T03:40:43.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:43 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:40:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:44 smithi114 ceph-mon[40961]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-14T03:40:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:44 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/1674446139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:40:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:44 smithi114 ceph-mon[29589]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-14T03:40:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:44 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/1674446139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:40:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:44 smithi160 ceph-mon[32702]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-14T03:40:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:44 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/1674446139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:40:46.816 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:46 smithi160 ceph-mon[32702]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-14T03:40:46.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:46 smithi114 ceph-mon[29589]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-14T03:40:46.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:46 smithi114 ceph-mon[40961]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-14T03:40:48.771 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:48 smithi160 ceph-mon[32702]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-14T03:40:48.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:48 smithi114 ceph-mon[40961]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-14T03:40:48.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:48 smithi114 ceph-mon[29589]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-14T03:40:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:49 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-14T03:40:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:49 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:49 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-14T03:40:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:49 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:49 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-14T03:40:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:49 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:40:50.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:50 smithi114 ceph-mon[29589]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:50.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:50 smithi114 ceph-mon[29589]: Deploying daemon osd.6 on smithi160 2024-09-14T03:40:50.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:50 smithi114 ceph-mon[40961]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:50.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:50 smithi114 ceph-mon[40961]: Deploying daemon osd.6 on smithi160 2024-09-14T03:40:50.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:50 smithi160 ceph-mon[32702]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:50 smithi160 ceph-mon[32702]: Deploying daemon osd.6 on smithi160 2024-09-14T03:40:52.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:52 smithi114 ceph-mon[29589]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:52.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:52 smithi114 ceph-mon[40961]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:52 smithi160 ceph-mon[32702]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-14T03:40:54.730 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:54 smithi160 ceph-mon[32702]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:54.730 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:54.730 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:54.730 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:54 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:54.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:54 smithi114 ceph-mon[29589]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:54 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:54.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:54 smithi114 ceph-mon[40961]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:54.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:40:54.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:54.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:54 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:56.817 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:56 smithi160 ceph-mon[32702]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:56.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:56 smithi114 ceph-mon[29589]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:56.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:56 smithi114 ceph-mon[40961]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:57.102 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 6 on host 'smithi160' 2024-09-14T03:40:57.749 DEBUG:teuthology.orchestra.run.smithi160:osd.6> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.6.service 2024-09-14T03:40:57.751 INFO:tasks.cephadm:Deploying osd.7 on smithi160 with /dev/vg_nvme/lv_1... 2024-09-14T03:40:57.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-14T03:40:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[29589]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[29589]: from='osd.6 [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-14T03:40:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[29589]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-14T03:40:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[40961]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:58.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[40961]: from='osd.6 [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-14T03:40:58.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[40961]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-14T03:40:58.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:58 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:58 smithi160 ceph-mon[32702]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:40:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:58 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:58 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:58 smithi160 ceph-mon[32702]: from='osd.6 [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-14T03:40:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:58 smithi160 ceph-mon[32702]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-14T03:40:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:58 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:58 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:40:58.565 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:40:59.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:59 smithi114 ceph-mon[29589]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-14T03:40:59.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:59 smithi114 ceph-mon[29589]: osdmap e40: 7 total, 6 up, 7 in 2024-09-14T03:40:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:59 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:40:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:59 smithi114 ceph-mon[29589]: from='osd.6 [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:40:59 smithi114 ceph-mon[29589]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:59 smithi114 ceph-mon[40961]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-14T03:40:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:59 smithi114 ceph-mon[40961]: osdmap e40: 7 total, 6 up, 7 in 2024-09-14T03:40:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:59 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:40:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:59 smithi114 ceph-mon[40961]: from='osd.6 [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:40:59 smithi114 ceph-mon[40961]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:59 smithi160 ceph-mon[32702]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-14T03:40:59.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:59 smithi160 ceph-mon[32702]: osdmap e40: 7 total, 6 up, 7 in 2024-09-14T03:40:59.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:59 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:40:59.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:59 smithi160 ceph-mon[32702]: from='osd.6 [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:59.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:40:59 smithi160 ceph-mon[32702]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:40:59.409 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 03:40:59 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T03:40:59.111+0000 7fd3e2df2640 -1 osd.6 0 waiting for initial osdmap 2024-09-14T03:40:59.409 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 03:40:59 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T03:40:59.118+0000 7fd3dc404640 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:41:00.228 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:00 smithi160 ceph-mon[32702]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:41:00.228 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:00 smithi160 ceph-mon[32702]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:41:00.228 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:00 smithi160 ceph-mon[32702]: osdmap e41: 7 total, 6 up, 7 in 2024-09-14T03:41:00.228 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:00 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:00.229 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:00 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:00.229 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:00 smithi160 ceph-mon[32702]: osd.6 [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982] boot 2024-09-14T03:41:00.229 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:00 smithi160 ceph-mon[32702]: osdmap e42: 7 total, 7 up, 7 in 2024-09-14T03:41:00.229 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:00 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:00.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[29589]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:41:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[29589]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:41:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[29589]: osdmap e41: 7 total, 6 up, 7 in 2024-09-14T03:41:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[29589]: osd.6 [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982] boot 2024-09-14T03:41:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[29589]: osdmap e42: 7 total, 7 up, 7 in 2024-09-14T03:41:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[40961]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-14T03:41:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[40961]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:41:00.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[40961]: osdmap e41: 7 total, 6 up, 7 in 2024-09-14T03:41:00.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:00.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:00.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[40961]: osd.6 [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982] boot 2024-09-14T03:41:00.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[40961]: osdmap e42: 7 total, 7 up, 7 in 2024-09-14T03:41:00.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:00 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:01.365 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:01 smithi160 ceph-mon[32702]: purged_snaps scrub starts 2024-09-14T03:41:01.365 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:01 smithi160 ceph-mon[32702]: purged_snaps scrub ok 2024-09-14T03:41:01.366 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:01 smithi160 ceph-mon[32702]: osdmap e43: 7 total, 7 up, 7 in 2024-09-14T03:41:01.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:01 smithi114 ceph-mon[29589]: purged_snaps scrub starts 2024-09-14T03:41:01.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:01 smithi114 ceph-mon[29589]: purged_snaps scrub ok 2024-09-14T03:41:01.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:01 smithi114 ceph-mon[29589]: osdmap e43: 7 total, 7 up, 7 in 2024-09-14T03:41:01.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:01 smithi114 ceph-mon[40961]: purged_snaps scrub starts 2024-09-14T03:41:01.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:01 smithi114 ceph-mon[40961]: purged_snaps scrub ok 2024-09-14T03:41:01.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:01 smithi114 ceph-mon[40961]: osdmap e43: 7 total, 7 up, 7 in 2024-09-14T03:41:02.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: Detected new or changed devices on smithi160 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi160 to 5696M 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:02.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:02.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:02.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:02.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:02 smithi160 ceph-mon[32702]: osdmap e44: 7 total, 7 up, 7 in 2024-09-14T03:41:02.770 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-14T03:41:02.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_1 2024-09-14T03:41:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: Detected new or changed devices on smithi160 2024-09-14T03:41:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 5696M 2024-09-14T03:41:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[29589]: osdmap e44: 7 total, 7 up, 7 in 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: Detected new or changed devices on smithi160 2024-09-14T03:41:02.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi160 to 5696M 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:02.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:02.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:02 smithi114 ceph-mon[40961]: osdmap e44: 7 total, 7 up, 7 in 2024-09-14T03:41:03.031 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:41:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:04 smithi160 ceph-mon[32702]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:04.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:04 smithi114 ceph-mon[29589]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:04 smithi114 ceph-mon[40961]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:05 smithi160 ceph-mon[32702]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:41:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:41:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:05 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:05.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:05 smithi114 ceph-mon[29589]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:41:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:41:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:05 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:05 smithi114 ceph-mon[40961]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-14T03:41:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-14T03:41:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:05 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:06 smithi160 ceph-mon[32702]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:06.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:06 smithi114 ceph-mon[40961]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:06.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:06 smithi114 ceph-mon[29589]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:07 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/1467762565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d8b5297-b1db-41b1-be35-073e7780128e"}]: dispatch 2024-09-14T03:41:07.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:07 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d8b5297-b1db-41b1-be35-073e7780128e"}]: dispatch 2024-09-14T03:41:07.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:07 smithi160 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d8b5297-b1db-41b1-be35-073e7780128e"}]': finished 2024-09-14T03:41:07.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:07 smithi160 ceph-mon[32702]: osdmap e45: 8 total, 7 up, 8 in 2024-09-14T03:41:07.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:07 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:07.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:07 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/1467762565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d8b5297-b1db-41b1-be35-073e7780128e"}]: dispatch 2024-09-14T03:41:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:07 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d8b5297-b1db-41b1-be35-073e7780128e"}]: dispatch 2024-09-14T03:41:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:07 smithi114 ceph-mon[29589]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d8b5297-b1db-41b1-be35-073e7780128e"}]': finished 2024-09-14T03:41:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:07 smithi114 ceph-mon[29589]: osdmap e45: 8 total, 7 up, 8 in 2024-09-14T03:41:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:07 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:07 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/1467762565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d8b5297-b1db-41b1-be35-073e7780128e"}]: dispatch 2024-09-14T03:41:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:07 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d8b5297-b1db-41b1-be35-073e7780128e"}]: dispatch 2024-09-14T03:41:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:07 smithi114 ceph-mon[40961]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d8b5297-b1db-41b1-be35-073e7780128e"}]': finished 2024-09-14T03:41:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:07 smithi114 ceph-mon[40961]: osdmap e45: 8 total, 7 up, 8 in 2024-09-14T03:41:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:07 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:08.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:08 smithi114 ceph-mon[29589]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:08 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/88988755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:41:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:08 smithi114 ceph-mon[40961]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:08 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/88988755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:41:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:08 smithi160 ceph-mon[32702]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:08 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/88988755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-14T03:41:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:10 smithi160 ceph-mon[32702]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:10.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:10 smithi114 ceph-mon[40961]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:10.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:10 smithi114 ceph-mon[29589]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:12 smithi160 ceph-mon[32702]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:12.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:12 smithi114 ceph-mon[29589]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:12.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:12 smithi114 ceph-mon[40961]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:13.644 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:13 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-14T03:41:13.644 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:13 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:13.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:13 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-14T03:41:13.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:13 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:13 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-14T03:41:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:13 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:14.684 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:14 smithi160 ceph-mon[32702]: Deploying daemon osd.7 on smithi160 2024-09-14T03:41:14.684 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:14 smithi160 ceph-mon[32702]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:14.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:14 smithi114 ceph-mon[29589]: Deploying daemon osd.7 on smithi160 2024-09-14T03:41:14.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:14 smithi114 ceph-mon[29589]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:14.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:14 smithi114 ceph-mon[40961]: Deploying daemon osd.7 on smithi160 2024-09-14T03:41:14.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:14 smithi114 ceph-mon[40961]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:16.684 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:16 smithi160 ceph-mon[32702]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:16.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:16 smithi114 ceph-mon[29589]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:16 smithi114 ceph-mon[40961]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:18 smithi160 ceph-mon[32702]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:18 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:18 smithi114 ceph-mon[29589]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:18 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:18 smithi114 ceph-mon[40961]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:18 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:20.549 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:20 smithi160 ceph-mon[32702]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:20.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:20 smithi114 ceph-mon[29589]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:20.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:20 smithi114 ceph-mon[40961]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:21.136 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 7 on host 'smithi160' 2024-09-14T03:41:21.784 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:21 smithi160 ceph-mon[32702]: from='osd.7 [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-14T03:41:21.784 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:21 smithi160 ceph-mon[32702]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-14T03:41:21.784 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:21 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:21.785 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:21 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:21.810 DEBUG:teuthology.orchestra.run.smithi160:osd.7> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.7.service 2024-09-14T03:41:21.812 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-14T03:41:21.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-14T03:41:21.841 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:21 smithi114 ceph-mon[40961]: from='osd.7 [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-14T03:41:21.841 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:21 smithi114 ceph-mon[40961]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-14T03:41:21.841 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:21 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:21.841 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:21 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:21.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:21 smithi114 ceph-mon[29589]: from='osd.7 [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-14T03:41:21.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:21 smithi114 ceph-mon[29589]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-14T03:41:21.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:21.842 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:21 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:22.072 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:41:22.702 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:22 smithi160 ceph-mon[32702]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:22.702 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:22 smithi160 ceph-mon[32702]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-14T03:41:22.702 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:22 smithi160 ceph-mon[32702]: osdmap e46: 8 total, 7 up, 8 in 2024-09-14T03:41:22.702 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:22 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:22.702 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:22 smithi160 ceph-mon[32702]: from='osd.7 [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:41:22.702 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:22 smithi160 ceph-mon[32702]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:41:22.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:22 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:22.703 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:22 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:22.703 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 03:41:22 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T03:41:22.444+0000 7faa80dae640 -1 osd.7 0 waiting for initial osdmap 2024-09-14T03:41:22.703 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 03:41:22 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T03:41:22.452+0000 7faa7a3c0640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:41:22.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[29589]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:22.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[29589]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-14T03:41:22.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[29589]: osdmap e46: 8 total, 7 up, 8 in 2024-09-14T03:41:22.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:22.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[29589]: from='osd.7 [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:41:22.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[29589]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:41:22.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:22.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:22.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[40961]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:22.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[40961]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-14T03:41:22.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[40961]: osdmap e46: 8 total, 7 up, 8 in 2024-09-14T03:41:22.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:22.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[40961]: from='osd.7 [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:41:22.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[40961]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-09-14T03:41:22.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:22.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:22 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:23.557 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:23 smithi160 ceph-mon[32702]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:41:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:23 smithi160 ceph-mon[32702]: osdmap e47: 8 total, 7 up, 8 in 2024-09-14T03:41:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:23 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:23 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:23 smithi160 ceph-mon[32702]: osd.7 [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841] boot 2024-09-14T03:41:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:23 smithi160 ceph-mon[32702]: osdmap e48: 8 total, 8 up, 8 in 2024-09-14T03:41:23.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:23 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:23.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[40961]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:41:23.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[40961]: osdmap e47: 8 total, 7 up, 8 in 2024-09-14T03:41:23.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:23.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:23.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[40961]: osd.7 [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841] boot 2024-09-14T03:41:23.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[40961]: osdmap e48: 8 total, 8 up, 8 in 2024-09-14T03:41:23.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:23.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[29589]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-09-14T03:41:23.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[29589]: osdmap e47: 8 total, 7 up, 8 in 2024-09-14T03:41:23.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:23.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:23.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[29589]: osd.7 [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841] boot 2024-09-14T03:41:23.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[29589]: osdmap e48: 8 total, 8 up, 8 in 2024-09-14T03:41:23.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:23 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:23.945 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:41:24.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:24 smithi114 ceph-mon[29589]: purged_snaps scrub starts 2024-09-14T03:41:24.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:24 smithi114 ceph-mon[29589]: purged_snaps scrub ok 2024-09-14T03:41:24.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:24 smithi114 ceph-mon[29589]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:24.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:24 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2091968733' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-14T03:41:24.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:24 smithi114 ceph-mon[29589]: osdmap e49: 8 total, 8 up, 8 in 2024-09-14T03:41:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:24 smithi114 ceph-mon[40961]: purged_snaps scrub starts 2024-09-14T03:41:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:24 smithi114 ceph-mon[40961]: purged_snaps scrub ok 2024-09-14T03:41:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:24 smithi114 ceph-mon[40961]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:24 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2091968733' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-14T03:41:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:24 smithi114 ceph-mon[40961]: osdmap e49: 8 total, 8 up, 8 in 2024-09-14T03:41:24.672 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":48,"num_osds":8,"num_up_osds":8,"osd_up_since":1726285283,"num_in_osds":8,"osd_in_since":1726285266,"num_remapped_pgs":1} 2024-09-14T03:41:24.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-14T03:41:24.906 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:24 smithi160 ceph-mon[32702]: purged_snaps scrub starts 2024-09-14T03:41:24.906 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:24 smithi160 ceph-mon[32702]: purged_snaps scrub ok 2024-09-14T03:41:24.906 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:24 smithi160 ceph-mon[32702]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-14T03:41:24.906 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:24 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2091968733' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-14T03:41:24.906 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:24 smithi160 ceph-mon[32702]: osdmap e49: 8 total, 8 up, 8 in 2024-09-14T03:41:24.956 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:41:26.485 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:26.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: Detected new or changed devices on smithi160 2024-09-14T03:41:26.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 4272M 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[29589]: osdmap e50: 8 total, 8 up, 8 in 2024-09-14T03:41:26.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: Detected new or changed devices on smithi160 2024-09-14T03:41:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi160 to 4272M 2024-09-14T03:41:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:26 smithi114 ceph-mon[40961]: osdmap e50: 8 total, 8 up, 8 in 2024-09-14T03:41:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: Detected new or changed devices on smithi160 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi160 to 4272M 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:26.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:26.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:26.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:26.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:26 smithi160 ceph-mon[32702]: osdmap e50: 8 total, 8 up, 8 in 2024-09-14T03:41:26.911 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:41:26.912 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":50,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","created":"2024-09-14T03:36:30.445371+0000","modified":"2024-09-14T03:41:25.439488+0000","last_up_change":"2024-09-14T03:41:23.437053+0000","last_in_change":"2024-09-14T03:41:06.908282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T03:39:29.348427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0a021a3a-45d2-4825-a701-d7c3793ef6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6803","nonce":2918802523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6805","nonce":2918802523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6809","nonce":2918802523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6807","nonce":2918802523}]},"public_addr":"172.21.15.114:6803/2918802523","cluster_addr":"172.21.15.114:6805/2918802523","heartbeat_back_addr":"172.21.15.114:6809/2918802523","heartbeat_front_addr":"172.21.15.114:6807/2918802523","state":["exists","up"]},{"osd":1,"uuid":"076291c4-5927-456d-bf74-2ab79be4c756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6811","nonce":1989151589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6813","nonce":1989151589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6817","nonce":1989151589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6815","nonce":1989151589}]},"public_addr":"172.21.15.114:6811/1989151589","cluster_addr":"172.21.15.114:6813/1989151589","heartbeat_back_addr":"172.21.15.114:6817/1989151589","heartbeat_front_addr":"172.21.15.114:6815/1989151589","state":["exists","up"]},{"osd":2,"uuid":"d6787400-73ea-4edf-bb17-440ac36bd4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6819","nonce":2560140870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6821","nonce":2560140870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6825","nonce":2560140870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6823","nonce":2560140870}]},"public_addr":"172.21.15.114:6819/2560140870","cluster_addr":"172.21.15.114:6821/2560140870","heartbeat_back_addr":"172.21.15.114:6825/2560140870","heartbeat_front_addr":"172.21.15.114:6823/2560140870","state":["exists","up"]},{"osd":3,"uuid":"ead755f3-173a-4465-a28f-af7288b0a42b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6827","nonce":3048125105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6829","nonce":3048125105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6833","nonce":3048125105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6831","nonce":3048125105}]},"public_addr":"172.21.15.114:6827/3048125105","cluster_addr":"172.21.15.114:6829/3048125105","heartbeat_back_addr":"172.21.15.114:6833/3048125105","heartbeat_front_addr":"172.21.15.114:6831/3048125105","state":["exists","up"]},{"osd":4,"uuid":"60a0aac2-e235-484f-b25f-f112c83da7f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6801","nonce":825361940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6803","nonce":825361940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6807","nonce":825361940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6805","nonce":825361940}]},"public_addr":"172.21.15.160:6801/825361940","cluster_addr":"172.21.15.160:6803/825361940","heartbeat_back_addr":"172.21.15.160:6807/825361940","heartbeat_front_addr":"172.21.15.160:6805/825361940","state":["exists","up"]},{"osd":5,"uuid":"8f49c7de-c810-4075-b731-022bb5031d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6809","nonce":3752160003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6811","nonce":3752160003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6815","nonce":3752160003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6813","nonce":3752160003}]},"public_addr":"172.21.15.160:6809/3752160003","cluster_addr":"172.21.15.160:6811/3752160003","heartbeat_back_addr":"172.21.15.160:6815/3752160003","heartbeat_front_addr":"172.21.15.160:6813/3752160003","state":["exists","up"]},{"osd":6,"uuid":"455de335-7290-4242-b743-6b0c446683ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716932982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716932982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716932982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716932982}]},"public_addr":"172.21.15.160:6817/3716932982","cluster_addr":"172.21.15.160:6819/3716932982","heartbeat_back_addr":"172.21.15.160:6823/3716932982","heartbeat_front_addr":"172.21.15.160:6821/3716932982","state":["exists","up"]},{"osd":7,"uuid":"7d8b5297-b1db-41b1-be35-073e7780128e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6825","nonce":1323253841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6827","nonce":1323253841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6831","nonce":1323253841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6829","nonce":1323253841}]},"public_addr":"172.21.15.160:6825/1323253841","cluster_addr":"172.21.15.160:6827/1323253841","heartbeat_back_addr":"172.21.15.160:6831/1323253841","heartbeat_front_addr":"172.21.15.160:6829/1323253841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:38:38.363862+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:02.363463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:26.353194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:50.132880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:11.402862+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:34.539123+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:58.191287+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:41:22.159510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/2095546075":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2155247703":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2067882043":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6801/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1089490408":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2694023561":"2024-09-15T03:36:58.525006+0000","172.21.15.114:6801/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2536880143":"2024-09-15T03:36:58.525006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T03:41:27.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:27 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1964840432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:41:27.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:27 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1964840432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:41:27.648 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-14T03:39:29.348427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-14T03:41:27.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-14T03:41:27.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:27 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1964840432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:41:27.889 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:41:28.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:28 smithi114 ceph-mon[29589]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:28.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:28 smithi114 ceph-mon[40961]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:28 smithi160 ceph-mon[32702]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:29.703 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-09-14T03:41:30.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:30 smithi114 ceph-mon[29589]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:30.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:30 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/94904194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-14T03:41:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:30 smithi114 ceph-mon[40961]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:30 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/94904194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-14T03:41:30.444 INFO:tasks.cephadm:Adding prometheus.a on smithi160 2024-09-14T03:41:30.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi160=a' 2024-09-14T03:41:30.473 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:30 smithi160 ceph-mon[32702]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:30.473 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:30 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/94904194' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-14T03:41:30.712 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:41:32.405 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled prometheus update... 2024-09-14T03:41:32.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:32 smithi114 ceph-mon[29589]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:32.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:32 smithi114 ceph-mon[40961]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:32 smithi160 ceph-mon[32702]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:32.998 DEBUG:teuthology.orchestra.run.smithi160:prometheus.a> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-14T03:41:33.040 INFO:tasks.cephadm:Adding node-exporter.a on smithi114 2024-09-14T03:41:33.040 INFO:tasks.cephadm:Adding node-exporter.b on smithi160 2024-09-14T03:41:33.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi114=a;smithi160=b' 2024-09-14T03:41:33.328 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:41:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:33 smithi160 ceph-mon[32702]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi160=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:33 smithi160 ceph-mon[32702]: Saving service prometheus spec with placement smithi160=a;count:1 2024-09-14T03:41:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:33 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:33 smithi160 ceph-mon[32702]: Deploying daemon prometheus.a on smithi160 2024-09-14T03:41:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[29589]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi160=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[29589]: Saving service prometheus spec with placement smithi160=a;count:1 2024-09-14T03:41:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:33.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:33.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:33.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:33.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:33.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[29589]: Deploying daemon prometheus.a on smithi160 2024-09-14T03:41:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[40961]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi160=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[40961]: Saving service prometheus spec with placement smithi160=a;count:1 2024-09-14T03:41:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:33.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:33.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:33.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:33 smithi114 ceph-mon[40961]: Deploying daemon prometheus.a on smithi160 2024-09-14T03:41:34.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:34 smithi114 ceph-mon[29589]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:34.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:34 smithi114 ceph-mon[40961]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:34 smithi160 ceph-mon[32702]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:36.310 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled node-exporter update... 2024-09-14T03:41:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:36 smithi160 ceph-mon[32702]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:36 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:36.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:36 smithi114 ceph-mon[29589]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:36 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:36.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:36 smithi114 ceph-mon[40961]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:36.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:36 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:37.497 DEBUG:teuthology.orchestra.run.smithi114:node-exporter.a> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-14T03:41:37.501 DEBUG:teuthology.orchestra.run.smithi160:node-exporter.b> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-14T03:41:37.503 INFO:tasks.cephadm:Adding alertmanager.a on smithi114 2024-09-14T03:41:37.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi114=a' 2024-09-14T03:41:37.781 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:41:37.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:37 smithi114 ceph-mon[40961]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi114=a;smithi160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:37 smithi114 ceph-mon[40961]: Saving service node-exporter spec with placement smithi114=a;smithi160=b;count:2 2024-09-14T03:41:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:37 smithi114 ceph-mon[29589]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi114=a;smithi160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:37 smithi114 ceph-mon[29589]: Saving service node-exporter spec with placement smithi114=a;smithi160=b;count:2 2024-09-14T03:41:37.902 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:37 smithi160 ceph-mon[32702]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi114=a;smithi160=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:37.902 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:37 smithi160 ceph-mon[32702]: Saving service node-exporter spec with placement smithi114=a;smithi160=b;count:2 2024-09-14T03:41:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:38 smithi160 ceph-mon[32702]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:38.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:38 smithi114 ceph-mon[29589]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:38 smithi114 ceph-mon[40961]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:40.562 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled alertmanager update... 2024-09-14T03:41:40.803 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:40 smithi160 ceph-mon[32702]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:40.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:40 smithi114 ceph-mon[29589]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:40.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:40 smithi114 ceph-mon[40961]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:41.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:41 smithi114 ceph-mon[29589]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:41.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:41 smithi114 ceph-mon[29589]: Saving service alertmanager spec with placement smithi114=a;count:1 2024-09-14T03:41:41.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:41 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:41 smithi114 ceph-mon[40961]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:41 smithi114 ceph-mon[40961]: Saving service alertmanager spec with placement smithi114=a;count:1 2024-09-14T03:41:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:41 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:41 smithi160 ceph-mon[32702]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:41:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:41 smithi160 ceph-mon[32702]: Saving service alertmanager spec with placement smithi114=a;count:1 2024-09-14T03:41:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:41 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:42.869 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:42 smithi160 ceph-mon[32702]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:42.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:42 smithi114 ceph-mon[29589]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:42.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:42 smithi114 ceph-mon[40961]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:42.892 DEBUG:teuthology.orchestra.run.smithi114:alertmanager.a> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-14T03:41:42.895 INFO:tasks.cephadm:Adding grafana.a on smithi160 2024-09-14T03:41:42.895 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi160=a' 2024-09-14T03:41:43.787 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:41:44.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:44 smithi114 ceph-mon[29589]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:44 smithi114 ceph-mon[40961]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:44 smithi160 ceph-mon[32702]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:45.267 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 podman[62052]: 2024-09-14 03:41:45.145919834 +0000 UTC m=+0.166942953 container create b32e2d1cd0b0a2fcc11f1ba1c8e7bdff4a24ecb51f7fb82375807c3869f4489f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:41:45.605 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 podman[62052]: 2024-09-14 03:41:45.552536018 +0000 UTC m=+0.573559092 container init b32e2d1cd0b0a2fcc11f1ba1c8e7bdff4a24ecb51f7fb82375807c3869f4489f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:41:45.605 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 podman[62052]: 2024-09-14 03:41:45.557518915 +0000 UTC m=+0.578541991 container start b32e2d1cd0b0a2fcc11f1ba1c8e7bdff4a24ecb51f7fb82375807c3869f4489f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:41:45.605 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.581Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-14T03:41:45.605 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.581Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-14T03:41:45.605 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.581Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi160 (none))" 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.581Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.581Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.583Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.584Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.585Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.585Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.586Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.586Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.826µs 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.586Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.586Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.586Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.845µs wal_replay_duration=175.791µs wbl_replay_duration=89ns total_replay_duration=218.039µs 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.588Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-14T03:41:45.606 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.588Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-14T03:41:45.607 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.588Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-14T03:41:45.607 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.604Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.027218ms db_storage=685ns remote_storage=1.09µs web_handler=211ns query_engine=424ns scrape=375.565µs scrape_sd=15.399µs notify=685ns notify_sd=2.225µs rules=15.441607ms tracing=6.132µs 2024-09-14T03:41:45.607 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.604Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-14T03:41:45.607 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:41:45.604Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-14T03:41:45.870 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 bash[62052]: b32e2d1cd0b0a2fcc11f1ba1c8e7bdff4a24ecb51f7fb82375807c3869f4489f 2024-09-14T03:41:45.871 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:41:45 smithi160 systemd[1]: Started Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:41:46.049 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled grafana update... 2024-09-14T03:41:46.640 DEBUG:teuthology.orchestra.run.smithi160:grafana.a> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@grafana.a.service 2024-09-14T03:41:46.643 INFO:tasks.cephadm:Setting up client nodes... 2024-09-14T03:41:46.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-14T03:41:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-mon[32702]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-14T03:41:46.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.909 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ignoring --setuser ceph since I am not root 2024-09-14T03:41:46.909 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ignoring --setgroup ceph since I am not root 2024-09-14T03:41:46.909 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-mgr[34767]: -- 172.21.15.160:0/2571965888 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55845b52e340 con 0x55845b53ac00 2024-09-14T03:41:46.909 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:46.876+0000 7fdeb6f79ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-14T03:41:46.912 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:41:46.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[29589]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:46.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-14T03:41:46.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.952 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ignoring --setuser ceph since I am not root 2024-09-14T03:41:46.952 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ignoring --setgroup ceph since I am not root 2024-09-14T03:41:46.952 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mgr[29861]: -- 172.21.15.114:0/142229359 <== mon.2 v2:172.21.15.114:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f42ba9c340 con 0x55f42baa8800 2024-09-14T03:41:46.952 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:46.888+0000 7eff49370ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-14T03:41:46.952 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:46.948+0000 7eff49370ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-14T03:41:46.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[40961]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:46.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.953 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-14T03:41:46.953 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:46.953 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:46 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' 2024-09-14T03:41:47.260 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:46.932+0000 7fdeb6f79ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-14T03:41:47.260 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:46.996+0000 7fdeb6f79ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-14T03:41:47.260 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:47.119+0000 7fdeb6f79ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:41:47.260 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:47.257+0000 7fdeb6f79ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-14T03:41:47.284 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:47.014+0000 7eff49370ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-14T03:41:47.284 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:47.141+0000 7eff49370ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:41:47.285 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:47.282+0000 7eff49370ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-14T03:41:47.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:47 smithi114 ceph-mon[40961]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-14T03:41:47.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:47 smithi114 ceph-mon[40961]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-14T03:41:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:47 smithi114 ceph-mon[29589]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-14T03:41:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:47 smithi114 ceph-mon[29589]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-14T03:41:47.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:47.542+0000 7eff49370ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-14T03:41:47.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:47.603+0000 7eff49370ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-14T03:41:47.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:47.769+0000 7eff49370ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-14T03:41:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:47 smithi160 ceph-mon[32702]: from='mgr.14150 172.21.15.114:0/682357665' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-14T03:41:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:47 smithi160 ceph-mon[32702]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-14T03:41:47.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:47.513+0000 7fdeb6f79ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-14T03:41:47.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:47.574+0000 7fdeb6f79ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-14T03:41:47.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:47.736+0000 7fdeb6f79ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-14T03:41:48.708 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:48 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:48.636+0000 7eff49370ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-14T03:41:48.893 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-09-14T03:41:48.894 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD8BeVm1voFNRAAdpnKI2IGW1bNeBdo1fp5mA== 2024-09-14T03:41:48.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:48.576+0000 7fdeb6f79ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-14T03:41:48.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:48.640+0000 7fdeb6f79ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-14T03:41:49.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:48 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:48.705+0000 7eff49370ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-14T03:41:49.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:48 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/311744163' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-14T03:41:49.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:48 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/311744163' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-14T03:41:49.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:48 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/311744163' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-14T03:41:49.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:48 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/311744163' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-14T03:41:49.349 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:48 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/311744163' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-14T03:41:49.349 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:48 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/311744163' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-14T03:41:49.491 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:49.423+0000 7eff49370ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-14T03:41:49.492 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T03:41:49.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-14T03:41:49.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-14T03:41:49.531 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-14T03:41:49.645 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:49.346+0000 7fdeb6f79ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-14T03:41:49.645 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:49.456+0000 7fdeb6f79ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-14T03:41:49.645 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:49.579+0000 7fdeb6f79ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:41:49.645 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:49.642+0000 7fdeb6f79ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-14T03:41:49.786 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T03:41:49.848 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:49.540+0000 7eff49370ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-14T03:41:49.848 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:49.661+0000 7eff49370ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:41:49.848 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:49.720+0000 7eff49370ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-14T03:41:49.848 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:49.844+0000 7eff49370ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-14T03:41:49.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:49.783+0000 7fdeb6f79ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-14T03:41:49.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:49.854+0000 7fdeb6f79ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-14T03:41:50.104 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:49.911+0000 7eff49370ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-14T03:41:50.104 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:49.968+0000 7eff49370ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-14T03:41:50.104 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:50.101+0000 7eff49370ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-14T03:41:50.189 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:49.912+0000 7fdeb6f79ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-14T03:41:50.189 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:50.052+0000 7fdeb6f79ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-14T03:41:50.189 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:50.121+0000 7fdeb6f79ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-14T03:41:50.189 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:50.186+0000 7fdeb6f79ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-14T03:41:50.367 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:50.165+0000 7eff49370ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-14T03:41:50.368 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:50.229+0000 7eff49370ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-14T03:41:50.368 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:50.308+0000 7eff49370ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-14T03:41:50.368 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:50.365+0000 7eff49370ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-14T03:41:50.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:50.422+0000 7eff49370ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-14T03:41:50.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:50.272+0000 7fdeb6f79ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-14T03:41:50.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:50.330+0000 7fdeb6f79ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-14T03:41:50.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:50.388+0000 7fdeb6f79ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-14T03:41:51.326 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:51.154+0000 7fdeb6f79ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-14T03:41:51.326 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: [14/Sep/2024:03:41:51] ENGINE Bus STARTING 2024-09-14T03:41:51.326 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: CherryPy Checker: 2024-09-14T03:41:51.326 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: The Application mounted at '' has an empty config. 2024-09-14T03:41:51.326 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: 2024-09-14T03:41:51.326 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: [14/Sep/2024:03:41:51] ENGINE Serving on http://:::9283 2024-09-14T03:41:51.326 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: [14/Sep/2024:03:41:51] ENGINE Bus STARTED 2024-09-14T03:41:51.327 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: Standby manager daemon x restarted 2024-09-14T03:41:51.327 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: Standby manager daemon x started 2024-09-14T03:41:51.327 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:41:51.327 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:41:51.327 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:41:51.327 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:41:51.327 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: Active manager daemon y restarted 2024-09-14T03:41:51.327 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: Activating manager daemon y 2024-09-14T03:41:51.327 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: osdmap e51: 8 total, 8 up, 8 in 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: mgrmap e17: y(active, starting, since 0.00669684s), standbys: x 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:41:51.328 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:51.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:51.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:41:51.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:41:51.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:41:51.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: Manager daemon y is now available 2024-09-14T03:41:51.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:51.431 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:51.159+0000 7eff49370ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-14T03:41:51.432 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:41:51] ENGINE Bus STARTING 2024-09-14T03:41:51.432 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: CherryPy Checker: 2024-09-14T03:41:51.432 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: The Application mounted at '' has an empty config. 2024-09-14T03:41:51.432 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:51.432 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:41:51] ENGINE Serving on http://:::9283 2024-09-14T03:41:51.432 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:41:51] ENGINE Bus STARTED 2024-09-14T03:41:51.432 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: Standby manager daemon x restarted 2024-09-14T03:41:51.432 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: Standby manager daemon x started 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: Active manager daemon y restarted 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: Activating manager daemon y 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: osdmap e51: 8 total, 8 up, 8 in 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: mgrmap e17: y(active, starting, since 0.00669684s), standbys: x 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:41:51.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: Manager daemon y is now available 2024-09-14T03:41:51.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: Standby manager daemon x restarted 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: Standby manager daemon x started 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/451097789' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: Active manager daemon y restarted 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: Activating manager daemon y 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: osdmap e51: 8 total, 8 up, 8 in 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: mgrmap e17: y(active, starting, since 0.00669684s), standbys: x 2024-09-14T03:41:51.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:41:51.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:41:51.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:41:51.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: Manager daemon y is now available 2024-09-14T03:41:51.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:41:51.650 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:41:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:41:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:41:51.650 INFO:teuthology.orchestra.run.smithi160.stdout:[client.1] 2024-09-14T03:41:51.651 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQD/BeVmgllrJhAAqoMUEhmBBH9jan5Lte8Avg== 2024-09-14T03:41:52.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:52 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:52.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:52 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:41:52.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:52 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:41:52.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:52 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:41:52.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:52 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:41:52.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:52 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:41:52.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:52 smithi160 ceph-mon[32702]: from='client.? 172.21.15.160:0/3182223054' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-14T03:41:52.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:52 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-14T03:41:52.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:52 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-14T03:41:52.272 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:52 smithi160 ceph-mon[32702]: mgrmap e18: y(active, since 1.01167s), standbys: x 2024-09-14T03:41:52.295 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-14T03:41:52.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-14T03:41:52.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-14T03:41:52.332 INFO:tasks.cephadm:Creating RBD pool 2024-09-14T03:41:52.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create rbd 8 2024-09-14T03:41:52.355 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:52.355 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:41:52.355 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:41:52.355 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:41:52.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:41:52.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:41:52.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[40961]: from='client.? 172.21.15.160:0/3182223054' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-14T03:41:52.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-14T03:41:52.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-14T03:41:52.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[40961]: mgrmap e18: y(active, since 1.01167s), standbys: x 2024-09-14T03:41:52.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:52.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:41:52.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:41:52.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:41:52.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:41:52.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:41:52.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[29589]: from='client.? 172.21.15.160:0/3182223054' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-14T03:41:52.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-14T03:41:52.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-14T03:41:52.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:52 smithi114 ceph-mon[29589]: mgrmap e18: y(active, since 1.01167s), standbys: x 2024-09-14T03:41:52.611 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:41:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-mon[29589]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:53.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-mon[40961]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:53.644 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:53 smithi160 ceph-mon[32702]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:53.889 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:53.632+0000 7efee812b640 -1 log_channel(cephadm) log [ERR] : [14/Sep/2024:03:41:53] ENGINE Error in HTTPServer.serve 2024-09-14T03:41:53.889 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: Traceback (most recent call last): 2024-09-14T03:41:53.889 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:41:53.889 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self._connections.run(self.expiration_interval) 2024-09-14T03:41:53.889 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:41:53.889 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self._run(expiration_interval) 2024-09-14T03:41:53.889 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:41:53.889 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: s = self.context.wrap_socket( 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: return self.sslsocket_class._create( 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self.do_handshake() 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: self._sslobj.do_handshake() 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:41:53.890 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:41:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:41:54.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[29589]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-14T03:41:54.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[29589]: [14/Sep/2024:03:41:53] ENGINE Bus STARTING 2024-09-14T03:41:54.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[29589]: [14/Sep/2024:03:41:53] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:41:54.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[29589]: [14/Sep/2024:03:41:53] ENGINE Bus STARTED 2024-09-14T03:41:54.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[29589]: [14/Sep/2024:03:41:53] ENGINE Error in HTTPServer.serve 2024-09-14T03:41:54.622 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-14T03:41:54.622 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:41:54.622 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:41:54.622 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:41:54.622 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-14T03:41:54.623 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:41:54.623 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:41:54.623 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:41:54.623 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:41:54.623 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:41:54.623 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-14T03:41:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:41:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-14T03:41:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:41:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-14T03:41:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:41:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-14T03:41:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:41:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.626 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[40961]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-14T03:41:54.626 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.626 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.626 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[40961]: [14/Sep/2024:03:41:53] ENGINE Bus STARTING 2024-09-14T03:41:54.626 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[40961]: [14/Sep/2024:03:41:53] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:41:54.627 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[40961]: [14/Sep/2024:03:41:53] ENGINE Bus STARTED 2024-09-14T03:41:54.627 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[40961]: [14/Sep/2024:03:41:53] ENGINE Error in HTTPServer.serve 2024-09-14T03:41:54.627 INFO:journalctl@ceph.mon.c.smithi114.stdout: Traceback (most recent call last): 2024-09-14T03:41:54.627 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:41:54.627 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:41:54.627 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:41:54.627 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._run(expiration_interval) 2024-09-14T03:41:54.627 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:41:54.628 INFO:journalctl@ceph.mon.c.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:41:54.628 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:41:54.628 INFO:journalctl@ceph.mon.c.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:41:54.628 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:41:54.628 INFO:journalctl@ceph.mon.c.smithi114.stdout: s = self.context.wrap_socket( 2024-09-14T03:41:54.628 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:41:54.628 INFO:journalctl@ceph.mon.c.smithi114.stdout: return self.sslsocket_class._create( 2024-09-14T03:41:54.628 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:41:54.629 INFO:journalctl@ceph.mon.c.smithi114.stdout: self.do_handshake() 2024-09-14T03:41:54.629 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:41:54.629 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._sslobj.do_handshake() 2024-09-14T03:41:54.629 INFO:journalctl@ceph.mon.c.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:41:54.629 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.629 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.629 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:54 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:54 smithi160 ceph-mon[32702]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-14T03:41:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:54 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:54 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:54 smithi160 ceph-mon[32702]: [14/Sep/2024:03:41:53] ENGINE Bus STARTING 2024-09-14T03:41:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:54 smithi160 ceph-mon[32702]: [14/Sep/2024:03:41:53] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:41:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:54 smithi160 ceph-mon[32702]: [14/Sep/2024:03:41:53] ENGINE Bus STARTED 2024-09-14T03:41:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:54 smithi160 ceph-mon[32702]: [14/Sep/2024:03:41:53] ENGINE Error in HTTPServer.serve 2024-09-14T03:41:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: Traceback (most recent call last): 2024-09-14T03:41:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:41:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:41:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:41:54.660 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._run(expiration_interval) 2024-09-14T03:41:54.660 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:41:54.660 INFO:journalctl@ceph.mon.b.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:41:54.660 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:41:54.660 INFO:journalctl@ceph.mon.b.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:41:54.660 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:41:54.660 INFO:journalctl@ceph.mon.b.smithi160.stdout: s = self.context.wrap_socket( 2024-09-14T03:41:54.660 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:41:54.661 INFO:journalctl@ceph.mon.b.smithi160.stdout: return self.sslsocket_class._create( 2024-09-14T03:41:54.661 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:41:54.661 INFO:journalctl@ceph.mon.b.smithi160.stdout: self.do_handshake() 2024-09-14T03:41:54.661 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:41:54.661 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._sslobj.do_handshake() 2024-09-14T03:41:54.662 INFO:journalctl@ceph.mon.b.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:41:54.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:54 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:54 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:54.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:54 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.425 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'rbd' created 2024-09-14T03:41:55.789 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1059342385' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi160 to 4016M 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[40961]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1059342385' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-14T03:41:55.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 4016M 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:55.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:55.793 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:41:55.793 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:55 smithi114 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:41:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1059342385' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-14T03:41:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi160 to 4016M 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:41:55.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:55 smithi160 ceph-mon[32702]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:41:56.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- rbd pool init rbd 2024-09-14T03:41:56.550 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:41:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:41:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:41:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:41:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:41:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:41:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:41:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1059342385' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-14T03:41:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: osdmap e52: 8 total, 8 up, 8 in 2024-09-14T03:41:56.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: Deploying daemon alertmanager.a on smithi114 2024-09-14T03:41:56.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[29589]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-14T03:41:56.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:41:56.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1059342385' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: osdmap e52: 8 total, 8 up, 8 in 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: Deploying daemon alertmanager.a on smithi114 2024-09-14T03:41:56.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:56 smithi114 ceph-mon[40961]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-14T03:41:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:41:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:41:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1059342385' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: osdmap e52: 8 total, 8 up, 8 in 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: Deploying daemon alertmanager.a on smithi114 2024-09-14T03:41:56.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:56 smithi160 ceph-mon[32702]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-14T03:41:57.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:57 smithi114 ceph-mon[29589]: osdmap e53: 8 total, 8 up, 8 in 2024-09-14T03:41:57.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:57 smithi114 ceph-mon[40961]: osdmap e53: 8 total, 8 up, 8 in 2024-09-14T03:41:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:57 smithi160 ceph-mon[32702]: osdmap e53: 8 total, 8 up, 8 in 2024-09-14T03:41:58.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:58 smithi114 ceph-mon[29589]: pgmap v8: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:58.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:58 smithi114 ceph-mon[29589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:41:58.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:58 smithi114 ceph-mon[29589]: osdmap e54: 8 total, 8 up, 8 in 2024-09-14T03:41:58.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:58 smithi114 ceph-mon[40961]: pgmap v8: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:58.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:58 smithi114 ceph-mon[40961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:41:58.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:58 smithi114 ceph-mon[40961]: osdmap e54: 8 total, 8 up, 8 in 2024-09-14T03:41:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:58 smithi160 ceph-mon[32702]: pgmap v8: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:41:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:58 smithi160 ceph-mon[32702]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:41:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:58 smithi160 ceph-mon[32702]: osdmap e54: 8 total, 8 up, 8 in 2024-09-14T03:41:59.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:59 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1605818283' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-14T03:41:59.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:41:59 smithi114 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-14T03:41:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:59 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1605818283' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-14T03:41:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:41:59 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-14T03:41:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:59 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1605818283' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-14T03:41:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:41:59 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-14T03:42:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:00 smithi114 ceph-mon[29589]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-14T03:42:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:00 smithi114 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-14T03:42:00.739 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:00 smithi114 ceph-mon[29589]: osdmap e55: 8 total, 8 up, 8 in 2024-09-14T03:42:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:00 smithi114 ceph-mon[40961]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-14T03:42:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:00 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-14T03:42:00.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:00 smithi114 ceph-mon[40961]: osdmap e55: 8 total, 8 up, 8 in 2024-09-14T03:42:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:00 smithi160 ceph-mon[32702]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-14T03:42:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:00 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-14T03:42:00.810 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:00 smithi160 ceph-mon[32702]: osdmap e55: 8 total, 8 up, 8 in 2024-09-14T03:42:01.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:01 smithi114 ceph-mon[29589]: osdmap e56: 8 total, 8 up, 8 in 2024-09-14T03:42:01.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:01 smithi114 ceph-mon[40961]: osdmap e56: 8 total, 8 up, 8 in 2024-09-14T03:42:01.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:01 smithi160 ceph-mon[32702]: osdmap e56: 8 total, 8 up, 8 in 2024-09-14T03:42:01.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:42:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:42:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:42:02.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-14T03:42:02.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:02 smithi114 ceph-mon[29589]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-14T03:42:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:02 smithi114 ceph-mon[29589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:42:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:02 smithi114 ceph-mon[29589]: Cluster is now healthy 2024-09-14T03:42:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:02 smithi114 ceph-mon[29589]: osdmap e57: 8 total, 8 up, 8 in 2024-09-14T03:42:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:02 smithi114 ceph-mon[40961]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-14T03:42:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:02 smithi114 ceph-mon[40961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:42:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:02 smithi114 ceph-mon[40961]: Cluster is now healthy 2024-09-14T03:42:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:02 smithi114 ceph-mon[40961]: osdmap e57: 8 total, 8 up, 8 in 2024-09-14T03:42:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:02 smithi160 ceph-mon[32702]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-14T03:42:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:02 smithi160 ceph-mon[32702]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:42:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:02 smithi160 ceph-mon[32702]: Cluster is now healthy 2024-09-14T03:42:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:02 smithi160 ceph-mon[32702]: osdmap e57: 8 total, 8 up, 8 in 2024-09-14T03:42:04.290 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:04.757 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:04 smithi114 ceph-mon[29589]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 357 B/s wr, 0 op/s 2024-09-14T03:42:04.757 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:04 smithi114 ceph-mon[40961]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 357 B/s wr, 0 op/s 2024-09-14T03:42:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:04 smithi160 ceph-mon[32702]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 357 B/s wr, 0 op/s 2024-09-14T03:42:05.794 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:05 smithi114 podman[71930]: 2024-09-14 03:42:05.710231618 +0000 UTC m=+0.121535551 volume create a470b79eb821d548f35d4351d88103c5cb035f686693b29a305986b98296fcba 2024-09-14T03:42:06.071 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:05 smithi114 podman[71930]: 2024-09-14 03:42:05.793925104 +0000 UTC m=+0.205228995 container create 4a0609b0f113d5f8b8a2f0172548136e319e75792d4d2d2b43f8d594d7b8112b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:42:06.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-mon[29589]: osdmap e58: 8 total, 8 up, 8 in 2024-09-14T03:42:06.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-mon[29589]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 357 B/s wr, 0 op/s 2024-09-14T03:42:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-mon[40961]: osdmap e58: 8 total, 8 up, 8 in 2024-09-14T03:42:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-mon[40961]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 357 B/s wr, 0 op/s 2024-09-14T03:42:06.371 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 podman[71930]: 2024-09-14 03:42:06.113448781 +0000 UTC m=+0.524752676 container init 4a0609b0f113d5f8b8a2f0172548136e319e75792d4d2d2b43f8d594d7b8112b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:42:06.371 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 podman[71930]: 2024-09-14 03:42:06.118196603 +0000 UTC m=+0.529500495 container start 4a0609b0f113d5f8b8a2f0172548136e319e75792d4d2d2b43f8d594d7b8112b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:42:06.371 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:42:06.130Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-14T03:42:06.371 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:42:06.130Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-14T03:42:06.371 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:42:06.131Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.114 port=9094 2024-09-14T03:42:06.371 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:42:06.132Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-14T03:42:06.372 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:42:06.154Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-14T03:42:06.372 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:42:06.155Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-14T03:42:06.372 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:42:06.157Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-14T03:42:06.372 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:42:06.157Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-14T03:42:06.372 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 bash[71930]: 4a0609b0f113d5f8b8a2f0172548136e319e75792d4d2d2b43f8d594d7b8112b 2024-09-14T03:42:06.372 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:06 smithi114 systemd[1]: Started Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:42:06.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:06 smithi160 ceph-mon[32702]: osdmap e58: 8 total, 8 up, 8 in 2024-09-14T03:42:06.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:06 smithi160 ceph-mon[32702]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 357 B/s wr, 0 op/s 2024-09-14T03:42:06.651 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:42:06] ENGINE Bus STOPPING 2024-09-14T03:42:06.897 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:42:06.907 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:42:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-14T03:42:06.907 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:42:06] ENGINE Bus STOPPED 2024-09-14T03:42:06.907 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:42:06] ENGINE Bus STARTING 2024-09-14T03:42:07.212 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:42:06] ENGINE Serving on http://:::9283 2024-09-14T03:42:07.212 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:42:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:42:06] ENGINE Bus STARTED 2024-09-14T03:42:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: Deploying daemon grafana.a on smithi160 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/869065565' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-14T03:42:07.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-14T03:42:07.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: Deploying daemon grafana.a on smithi160 2024-09-14T03:42:07.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:07 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/869065565' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-14T03:42:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-14T03:42:07.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-14T03:42:07.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:07.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: Deploying daemon grafana.a on smithi160 2024-09-14T03:42:07.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:07 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/869065565' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-14T03:42:07.754 INFO:teuthology.orchestra.run.smithi114.stdout:[{"version":19,"timestamp":"2024-09-14T03:42:06.365437+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/GRAFANA_API_SSL_VERIFY","new_value":"false"}]}] 2024-09-14T03:42:07.754 INFO:tasks.ceph_manager:config epoch is 19 2024-09-14T03:42:07.754 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-14T03:42:07.754 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-14T03:42:07.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-14T03:42:08.003 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:08.497 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:08 smithi114 ceph-mon[29589]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 304 B/s wr, 0 op/s 2024-09-14T03:42:08.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:08 smithi114 ceph-mon[40961]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 304 B/s wr, 0 op/s 2024-09-14T03:42:08.497 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:08 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:42:08.132Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000146442s 2024-09-14T03:42:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:08 smithi160 ceph-mon[32702]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 304 B/s wr, 0 op/s 2024-09-14T03:42:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:42:10.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:10 smithi114 ceph-mon[29589]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-14T03:42:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:10 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3334632421' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-14T03:42:10.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:10 smithi114 ceph-mon[40961]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-14T03:42:10.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:10 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3334632421' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-14T03:42:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:10 smithi160 ceph-mon[32702]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-14T03:42:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:10 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3334632421' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-14T03:42:10.732 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"flags":0,"active_gid":14448,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":1727106713},{"type":"v1","addr":"172.21.15.114:6801","nonce":1727106713}]},"active_addr":"172.21.15.114:6801/1727106713","active_change":"2024-09-14T03:41:51.162981+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14445,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.114:8443/","prometheus":"http://172.21.15.114:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":51,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4089637122}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":546762471}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":3853151361}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2995702914}]}]} 2024-09-14T03:42:10.738 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-14T03:42:10.738 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-14T03:42:10.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-14T03:42:10.980 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:11.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:42:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:42:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:42:12.327 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:12 smithi114 ceph-mon[29589]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 210 B/s wr, 0 op/s 2024-09-14T03:42:12.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:12 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:12.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:12 smithi114 ceph-mon[40961]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 210 B/s wr, 0 op/s 2024-09-14T03:42:12.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:12 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:12 smithi160 ceph-mon[32702]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 210 B/s wr, 0 op/s 2024-09-14T03:42:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:12 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:12.716 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:42:12.716 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":58,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","created":"2024-09-14T03:36:30.445371+0000","modified":"2024-09-14T03:42:05.064493+0000","last_up_change":"2024-09-14T03:41:23.437053+0000","last_in_change":"2024-09-14T03:41:06.908282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T03:39:29.348427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T03:41:54.688727+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0a021a3a-45d2-4825-a701-d7c3793ef6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6803","nonce":2918802523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6805","nonce":2918802523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6809","nonce":2918802523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6807","nonce":2918802523}]},"public_addr":"172.21.15.114:6803/2918802523","cluster_addr":"172.21.15.114:6805/2918802523","heartbeat_back_addr":"172.21.15.114:6809/2918802523","heartbeat_front_addr":"172.21.15.114:6807/2918802523","state":["exists","up"]},{"osd":1,"uuid":"076291c4-5927-456d-bf74-2ab79be4c756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6811","nonce":1989151589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6813","nonce":1989151589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6817","nonce":1989151589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6815","nonce":1989151589}]},"public_addr":"172.21.15.114:6811/1989151589","cluster_addr":"172.21.15.114:6813/1989151589","heartbeat_back_addr":"172.21.15.114:6817/1989151589","heartbeat_front_addr":"172.21.15.114:6815/1989151589","state":["exists","up"]},{"osd":2,"uuid":"d6787400-73ea-4edf-bb17-440ac36bd4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6819","nonce":2560140870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6821","nonce":2560140870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6825","nonce":2560140870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6823","nonce":2560140870}]},"public_addr":"172.21.15.114:6819/2560140870","cluster_addr":"172.21.15.114:6821/2560140870","heartbeat_back_addr":"172.21.15.114:6825/2560140870","heartbeat_front_addr":"172.21.15.114:6823/2560140870","state":["exists","up"]},{"osd":3,"uuid":"ead755f3-173a-4465-a28f-af7288b0a42b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6827","nonce":3048125105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6829","nonce":3048125105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6833","nonce":3048125105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6831","nonce":3048125105}]},"public_addr":"172.21.15.114:6827/3048125105","cluster_addr":"172.21.15.114:6829/3048125105","heartbeat_back_addr":"172.21.15.114:6833/3048125105","heartbeat_front_addr":"172.21.15.114:6831/3048125105","state":["exists","up"]},{"osd":4,"uuid":"60a0aac2-e235-484f-b25f-f112c83da7f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6801","nonce":825361940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6803","nonce":825361940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6807","nonce":825361940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6805","nonce":825361940}]},"public_addr":"172.21.15.160:6801/825361940","cluster_addr":"172.21.15.160:6803/825361940","heartbeat_back_addr":"172.21.15.160:6807/825361940","heartbeat_front_addr":"172.21.15.160:6805/825361940","state":["exists","up"]},{"osd":5,"uuid":"8f49c7de-c810-4075-b731-022bb5031d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6809","nonce":3752160003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6811","nonce":3752160003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6815","nonce":3752160003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6813","nonce":3752160003}]},"public_addr":"172.21.15.160:6809/3752160003","cluster_addr":"172.21.15.160:6811/3752160003","heartbeat_back_addr":"172.21.15.160:6815/3752160003","heartbeat_front_addr":"172.21.15.160:6813/3752160003","state":["exists","up"]},{"osd":6,"uuid":"455de335-7290-4242-b743-6b0c446683ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716932982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716932982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716932982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716932982}]},"public_addr":"172.21.15.160:6817/3716932982","cluster_addr":"172.21.15.160:6819/3716932982","heartbeat_back_addr":"172.21.15.160:6823/3716932982","heartbeat_front_addr":"172.21.15.160:6821/3716932982","state":["exists","up"]},{"osd":7,"uuid":"7d8b5297-b1db-41b1-be35-073e7780128e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6825","nonce":1323253841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6827","nonce":1323253841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6831","nonce":1323253841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6829","nonce":1323253841}]},"public_addr":"172.21.15.160:6825/1323253841","cluster_addr":"172.21.15.160:6827/1323253841","heartbeat_back_addr":"172.21.15.160:6831/1323253841","heartbeat_front_addr":"172.21.15.160:6829/1323253841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:38:38.363862+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:02.363463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:26.353194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:50.132880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:11.402862+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:34.539123+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:58.191287+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:41:22.159510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/767494479":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6800/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1784685209":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6801/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6801/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/426398557":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/2694023561":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1089490408":"2024-09-15T03:36:58.525006+0000","172.21.15.114:6801/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2536880143":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2067882043":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2155247703":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2095546075":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/3344792350":"2024-09-15T03:41:51.162793+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T03:42:13.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:13 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1693656059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:42:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:13 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1693656059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:42:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:13 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1693656059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:42:13.698 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-14T03:42:13.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-14T03:42:13.944 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:14.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:14 smithi114 ceph-mon[29589]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:14.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:14 smithi114 ceph-mon[40961]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:14.548 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:14 smithi160 ceph-mon[32702]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:15.867 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:42:15.867 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":58,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","created":"2024-09-14T03:36:30.445371+0000","modified":"2024-09-14T03:42:05.064493+0000","last_up_change":"2024-09-14T03:41:23.437053+0000","last_in_change":"2024-09-14T03:41:06.908282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T03:39:29.348427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T03:41:54.688727+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"0a021a3a-45d2-4825-a701-d7c3793ef6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6803","nonce":2918802523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6805","nonce":2918802523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6809","nonce":2918802523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6807","nonce":2918802523}]},"public_addr":"172.21.15.114:6803/2918802523","cluster_addr":"172.21.15.114:6805/2918802523","heartbeat_back_addr":"172.21.15.114:6809/2918802523","heartbeat_front_addr":"172.21.15.114:6807/2918802523","state":["exists","up"]},{"osd":1,"uuid":"076291c4-5927-456d-bf74-2ab79be4c756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6811","nonce":1989151589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6813","nonce":1989151589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6817","nonce":1989151589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6815","nonce":1989151589}]},"public_addr":"172.21.15.114:6811/1989151589","cluster_addr":"172.21.15.114:6813/1989151589","heartbeat_back_addr":"172.21.15.114:6817/1989151589","heartbeat_front_addr":"172.21.15.114:6815/1989151589","state":["exists","up"]},{"osd":2,"uuid":"d6787400-73ea-4edf-bb17-440ac36bd4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6819","nonce":2560140870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6821","nonce":2560140870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6825","nonce":2560140870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6823","nonce":2560140870}]},"public_addr":"172.21.15.114:6819/2560140870","cluster_addr":"172.21.15.114:6821/2560140870","heartbeat_back_addr":"172.21.15.114:6825/2560140870","heartbeat_front_addr":"172.21.15.114:6823/2560140870","state":["exists","up"]},{"osd":3,"uuid":"ead755f3-173a-4465-a28f-af7288b0a42b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6827","nonce":3048125105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6829","nonce":3048125105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6833","nonce":3048125105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6831","nonce":3048125105}]},"public_addr":"172.21.15.114:6827/3048125105","cluster_addr":"172.21.15.114:6829/3048125105","heartbeat_back_addr":"172.21.15.114:6833/3048125105","heartbeat_front_addr":"172.21.15.114:6831/3048125105","state":["exists","up"]},{"osd":4,"uuid":"60a0aac2-e235-484f-b25f-f112c83da7f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6801","nonce":825361940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6803","nonce":825361940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6807","nonce":825361940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6805","nonce":825361940}]},"public_addr":"172.21.15.160:6801/825361940","cluster_addr":"172.21.15.160:6803/825361940","heartbeat_back_addr":"172.21.15.160:6807/825361940","heartbeat_front_addr":"172.21.15.160:6805/825361940","state":["exists","up"]},{"osd":5,"uuid":"8f49c7de-c810-4075-b731-022bb5031d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6809","nonce":3752160003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6811","nonce":3752160003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6815","nonce":3752160003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6813","nonce":3752160003}]},"public_addr":"172.21.15.160:6809/3752160003","cluster_addr":"172.21.15.160:6811/3752160003","heartbeat_back_addr":"172.21.15.160:6815/3752160003","heartbeat_front_addr":"172.21.15.160:6813/3752160003","state":["exists","up"]},{"osd":6,"uuid":"455de335-7290-4242-b743-6b0c446683ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716932982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716932982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716932982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716932982}]},"public_addr":"172.21.15.160:6817/3716932982","cluster_addr":"172.21.15.160:6819/3716932982","heartbeat_back_addr":"172.21.15.160:6823/3716932982","heartbeat_front_addr":"172.21.15.160:6821/3716932982","state":["exists","up"]},{"osd":7,"uuid":"7d8b5297-b1db-41b1-be35-073e7780128e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6825","nonce":1323253841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6827","nonce":1323253841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6831","nonce":1323253841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6829","nonce":1323253841}]},"public_addr":"172.21.15.160:6825/1323253841","cluster_addr":"172.21.15.160:6827/1323253841","heartbeat_back_addr":"172.21.15.160:6831/1323253841","heartbeat_front_addr":"172.21.15.160:6829/1323253841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:38:38.363862+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:02.363463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:26.353194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:50.132880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:11.402862+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:34.539123+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:58.191287+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:41:22.159510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/767494479":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6800/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1784685209":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6801/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6801/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/426398557":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/2694023561":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1089490408":"2024-09-15T03:36:58.525006+0000","172.21.15.114:6801/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2536880143":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2067882043":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2155247703":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2095546075":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/3344792350":"2024-09-15T03:41:51.162793+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T03:42:16.494 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:16 smithi160 ceph-mon[32702]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:16.494 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:16 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1529510257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:42:16.542 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:16 smithi114 ceph-mon[40961]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:16.542 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:16 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1529510257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:42:16.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:16 smithi114 ceph-mon[29589]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:16.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:16 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1529510257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:42:16.543 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:42:16 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:42:16.134Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002596551s 2024-09-14T03:42:16.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-14T03:42:16.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-14T03:42:16.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-14T03:42:16.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-14T03:42:16.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-14T03:42:16.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-14T03:42:16.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-14T03:42:16.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-14T03:42:17.072 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:17.078 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:17.078 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:17.079 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:17.085 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:17.088 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:17.091 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:17.158 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 podman[64256]: 2024-09-14 03:42:16.984052597 +0000 UTC m=+0.014887867 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-14T03:42:17.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 podman[64256]: 2024-09-14 03:42:17.264010194 +0000 UTC m=+0.294845464 container init cb3fc2419525356e2a30c1dc0253b4907e3a9318c688fadbd15a6dd1eaec3001 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.tags=base rhel8, distribution-scope=public, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-14T03:42:17.515 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 podman[64256]: 2024-09-14 03:42:17.26888555 +0000 UTC m=+0.299720809 container start cb3fc2419525356e2a30c1dc0253b4907e3a9318c688fadbd15a6dd1eaec3001 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, architecture=x86_64, distribution-scope=public, version=8.5, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, io.openshift.tags=base rhel8, io.buildah.version=1.32.0, maintainer=Paul Cuzner , vcs-type=git, build-date=2022-04-29T11:48:11.055897) 2024-09-14T03:42:17.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 bash[64256]: cb3fc2419525356e2a30c1dc0253b4907e3a9318c688fadbd15a6dd1eaec3001 2024-09-14T03:42:17.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 systemd[1]: Started Ceph grafana.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:42:17.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428315931Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-14T03:42:17.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428531532Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-14T03:42:17.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428556256Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-14T03:42:17.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428567869Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-14T03:42:17.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428576945Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-14T03:42:17.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428586524Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-14T03:42:17.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428596242Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-14T03:42:17.516 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428606428Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428620261Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428631875Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428641484Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428650967Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.42866072Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=settings t=2024-09-14T03:42:17.428671646Z level=info msg="App mode production" 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=sqlstore t=2024-09-14T03:42:17.428730206Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=sqlstore t=2024-09-14T03:42:17.428754498Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.429638729Z level=info msg="Starting DB migrations" 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.429738224Z level=info msg="Executing migration" id="create migration_log table" 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.43033845Z level=info msg="Executing migration" id="create user table" 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.430779605Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-14T03:42:17.517 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.431277904Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-14T03:42:17.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.431713468Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-14T03:42:17.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.432251384Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-14T03:42:17.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.43264151Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-14T03:42:17.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.433457985Z level=info msg="Executing migration" id="create user table v2" 2024-09-14T03:42:17.518 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.433872937Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-14T03:42:17.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.434299916Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-14T03:42:17.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.434746913Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-14T03:42:17.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.435128722Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-14T03:42:17.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.435486832Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-14T03:42:17.519 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.436008893Z level=info msg="Executing migration" id="Update user table charset" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.436354047Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.436734037Z level=info msg="Executing migration" id="Add missing user data" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.437091558Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.437517752Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.437886727Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.438336495Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.439458421Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.439935475Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.440385659Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.441246077Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-14T03:42:17.520 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.442075199Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.44249122Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.445172674Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.44557861Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.448045807Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.448556544Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.451163609Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.453840827Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.454319905Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.457120658Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.457636858Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.459935124Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.462700703Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.46308158Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-14T03:42:17.521 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.466024021Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.468643436Z level=info msg="Executing migration" id="create star table" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.471377724Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.474012487Z level=info msg="Executing migration" id="create org table v1" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.476893096Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.479628449Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.482302734Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.484930726Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.48770709Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.490187712Z level=info msg="Executing migration" id="Update org table charset" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.492949394Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.495570671Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.498306906Z level=info msg="Executing migration" id="create dashboard table" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.500999662Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-14T03:42:17.522 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.503621864Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-14T03:42:17.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.50630608Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-14T03:42:17.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.509123715Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-14T03:42:17.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.511885266Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-14T03:42:17.523 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.514588502Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-14T03:42:17.766 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.517204321Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.520038072Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.522563997Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.525295582Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.526214502Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.52692278Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.527880386Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.528512744Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.529444392Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.530109916Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.531025558Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-14T03:42:17.767 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.531706146Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-14T03:42:17.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.532631133Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-14T03:42:17.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.533328146Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-14T03:42:17.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.534199888Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-14T03:42:17.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.535046403Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-14T03:42:17.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.535866314Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-14T03:42:17.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.536637117Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-14T03:42:17.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.537508863Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-14T03:42:17.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.538248722Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-14T03:42:17.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.539078089Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-14T03:42:17.768 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.540022053Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.540888248Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.5415933Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.542603156Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.543302225Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.544253071Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.545045353Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.545929421Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.546787263Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.5476509Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.548486058Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.549337317Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-14T03:42:17.769 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.550190157Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-14T03:42:17.770 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.551038338Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-14T03:42:17.770 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.5518929Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-14T03:42:17.770 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.552695244Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-14T03:42:17.770 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.55369319Z level=info msg="Executing migration" id="create data_source table" 2024-09-14T03:42:17.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.554391789Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-14T03:42:17.771 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.557200391Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.560064066Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.562880857Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.565377814Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.568218072Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.570645956Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.573327668Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.576087713Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.57892817Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.581721082Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.58455134Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.587220357Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-14T03:42:17.772 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.590147953Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.592965204Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.595688858Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.598364366Z level=info msg="Executing migration" id="Add uid column" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.601193841Z level=info msg="Executing migration" id="Update uid value" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.604109868Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.606646381Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.609373102Z level=info msg="Executing migration" id="create api_key table" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.612012736Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.614741831Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.617457696Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.619959397Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.622628662Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-14T03:42:17.773 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.625378452Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.627965067Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.629015384Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.631746618Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.634461692Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.637301561Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.637734886Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.640404688Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.642937068Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.645630474Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.646243424Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.647074927Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.647870933Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.648647884Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-14T03:42:17.774 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.649202359Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.649973372Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.650899075Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.651531146Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.652038001Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.652846409Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.653626427Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.65442274Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.655233963Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.656010603Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.656771525Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.657735754Z level=info msg="Executing migration" id="create quota table v1" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.658393721Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-14T03:42:17.775 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.661309797Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.664015903Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.664458372Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.664898683Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.667488952Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.670155371Z level=info msg="Executing migration" id="create session table" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.670650477Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.670982018Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.671285025Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.671676452Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.674238218Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.674583299Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.674876468Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-14T03:42:17.776 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.675209032Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.675511186Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.675924295Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.676241192Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.67860327Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.678980807Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.681421513Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.684200026Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.6845763Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.686826322Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.6892527Z level=info msg="Executing migration" id="create alert table v1" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.691944472Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.694767875Z level=info msg="Executing migration" id="add index alert state" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.695225121Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-14T03:42:17.777 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.695666095Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-14T03:42:17.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.69607247Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-14T03:42:17.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.696566433Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-14T03:42:17.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.697013167Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-14T03:42:17.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.698435866Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-14T03:42:17.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.698851544Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-14T03:42:17.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.699296119Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-14T03:42:17.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.699614537Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-14T03:42:17.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.70002555Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-14T03:42:17.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.700440983Z level=info msg="Executing migration" id="Add column is_default" 2024-09-14T03:42:17.778 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.701083824Z level=info msg="Executing migration" id="Add column frequency" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.701726732Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.702393639Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.703050771Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.703529599Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.703840615Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.704171451Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.704588641Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.705043524Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.705454588Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.705861493Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-14T03:42:17.779 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.706352727Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.707037424Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.707689944Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.70806359Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.708518559Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.708912745Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.709561201Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.709868595Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.710324915Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.710770309Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.711129173Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.711570965Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.714209537Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.714656458Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-14T03:42:17.780 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.715105653Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.715571097Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.71828177Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.718955446Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.719653401Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.720110691Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.720957016Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.721851768Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.72268159Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.723133427Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.724736202Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.725588369Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.726411383Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-14T03:42:17.781 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.726758303Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.727621359Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.728468111Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.729290926Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.730027223Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.730870862Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.731719665Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.732571678Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.733304801Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.734218711Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.735033145Z level=info msg="Executing migration" id="Move region to single row" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.736001299Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.736575576Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-14T03:42:17.782 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.737535373Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.738320002Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.739171664Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.739640864Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.740499839Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.741377402Z level=info msg="Executing migration" id="create test_data table" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.741837728Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.742318071Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.743133684Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.743953855Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.744797483Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.74562299Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-14T03:42:17.783 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.746441858Z level=info msg="Executing migration" id="create team table" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.746886723Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.747372206Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.750430929Z level=info msg="Executing migration" id="create team member table" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.7509221Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.753727235Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.75659526Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.759198706Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.759992118Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.762984515Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.76554732Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.768404068Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-14T03:42:17.784 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.769003455Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-14T03:42:18.018 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.771722716Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-14T03:42:18.019 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.772205517Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-14T03:42:18.019 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.774917375Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-14T03:42:18.019 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.775384267Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-14T03:42:18.019 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.77808352Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-14T03:42:18.019 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.781025094Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-14T03:42:18.019 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.783609778Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-14T03:42:18.019 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.786149055Z level=info msg="Executing migration" id="create tag table" 2024-09-14T03:42:18.019 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.788918798Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-14T03:42:18.019 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.791743837Z level=info msg="Executing migration" id="create login attempt table" 2024-09-14T03:42:18.020 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.794447535Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-14T03:42:18.020 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.796972535Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-14T03:42:18.020 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.799759392Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-14T03:42:18.020 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.802662434Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-14T03:42:18.020 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.804855323Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-14T03:42:18.020 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.807643788Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-14T03:42:18.020 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.810380749Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-14T03:42:18.020 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.813185792Z level=info msg="Executing migration" id="create user auth table" 2024-09-14T03:42:18.020 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.815558404Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-14T03:42:18.021 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.818392717Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-14T03:42:18.021 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.82098362Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.823647318Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.826330029Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.82710293Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.827914678Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.828926412Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.829800383Z level=info msg="Executing migration" id="create server_lock table" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.83067925Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.831683539Z level=info msg="Executing migration" id="create user auth token table" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.832360448Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.833336521Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.834029942Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.835019895Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.835797756Z level=info msg="Executing migration" id="create cache_data table" 2024-09-14T03:42:18.022 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.836668572Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.837677854Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.838378413Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.839233112Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.840047321Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.841035795Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.841780948Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.842601657Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.843589873Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.844294136Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.845173805Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.847945392Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.85064675Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-14T03:42:18.023 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.853469415Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.856200292Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.858904728Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.861539819Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.864210665Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.866951238Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.869838375Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.87250144Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.875202537Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.878061133Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.880850704Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.883475016Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.886205393Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-14T03:42:18.024 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.888942341Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.891687007Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.895171308Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.898480225Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.901367434Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.903865458Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.906503633Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.90911941Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.911727815Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.914505078Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.917049169Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.91984745Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.922594005Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.925212951Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-14T03:42:18.025 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.927929042Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.930665735Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.933312763Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.93609056Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.938873348Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.941593755Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.942446234Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.943294151Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.944234044Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.944922892Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.945719053Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.946488491Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.947326984Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.94820217Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-14T03:42:18.026 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.949035384Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.949900923Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.950730117Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.951574725Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.952423805Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.953264242Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.954133744Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.954926904Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.955853471Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.95665372Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.95753555Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.958365052Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-14T03:42:18.027 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.959252882Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.960140044Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.96110548Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.96180922Z level=info msg="Executing migration" id="create alert_image table" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.962744511Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.963461841Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.96437709Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.964900249Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.965378195Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.965463034Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.966168633Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.966590447Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.967048922Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.967177988Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-14T03:42:18.028 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.967296425Z level=info msg="alerts found to migrate" alerts=0 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.967914262Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.968777595Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.969641685Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.972525027Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.975138747Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.977942468Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.980576075Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.983368724Z level=info msg="Executing migration" id="create data_keys table" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.985943121Z level=info msg="Executing migration" id="create secrets table" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.988681354Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.992794186Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.9955275Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:17 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:17.998440072Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-14T03:42:18.029 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.00277621Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-14T03:42:18.030 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.006874672Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-14T03:42:18.030 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.009646571Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-14T03:42:18.030 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.012418189Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-14T03:42:18.030 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.015082013Z level=info msg="Executing migration" id="create permission table" 2024-09-14T03:42:18.030 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.017996135Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-14T03:42:18.030 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.020685077Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-14T03:42:18.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.023603782Z level=info msg="Executing migration" id="create role table" 2024-09-14T03:42:18.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.026178945Z level=info msg="Executing migration" id="add column display_name" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.029025925Z level=info msg="Executing migration" id="add column group_name" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.031681996Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.034309138Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.036986836Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.039463127Z level=info msg="Executing migration" id="create team role table" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.042312235Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.044881827Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.047565246Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.050118439Z level=info msg="Executing migration" id="create user role table" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.05296959Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.055674348Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-14T03:42:18.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.058386369Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.060964185Z level=info msg="Executing migration" id="create builtin role table" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.063641432Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.066362629Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.069062102Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.070094366Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.07094558Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.071721175Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.072189522Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.072654671Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.073478059Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.073968304Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.074933445Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-14T03:42:18.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.075350537Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.075797367Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.076104552Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.076494667Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.076968515Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.077345632Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.077633776Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.078011851Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.078425048Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.078871533Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.07983562Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-14T03:42:18.411 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.080141219Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-14T03:42:18.412 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.080545082Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-14T03:42:18.412 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.080976815Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-14T03:42:18.412 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.081408453Z level=info msg="Executing migration" id="add correlation config column" 2024-09-14T03:42:18.412 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.082413654Z level=info msg="Executing migration" id="create entity_events table" 2024-09-14T03:42:18.412 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.082843967Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-14T03:42:18.412 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.083631559Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-14T03:42:18.412 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.083710616Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.084054587Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.084160261Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.084461847Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.084878661Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.08530262Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.085758999Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.08622135Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.086631148Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.087075224Z level=info msg="Executing migration" id="Drop public config table" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.087506863Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.087905514Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.088355346Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.088802634Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-14T03:42:18.413 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.089299576Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.092310777Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.093322382Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.094372858Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.094717266Z level=info msg="Executing migration" id="add share column" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.095714881Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.096184036Z level=info msg="Executing migration" id="create file table" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.09662829Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.09741647Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.09813108Z level=info msg="Executing migration" id="create file_meta table" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.0989689Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-14T03:42:18.414 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.09984524Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.100757539Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.101540097Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.102352984Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.102828046Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.104017709Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.104869008Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.105862598Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.106708241Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.107473136Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.108386025Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.109618822Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.110638158Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-14T03:42:18.415 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.111594143Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-14T03:42:18.416 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.121739402Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-14T03:42:18.416 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.122591239Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-14T03:42:18.416 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.123447635Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-14T03:42:18.416 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.126924959Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-14T03:42:18.416 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.127756114Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-14T03:42:18.416 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.128742098Z level=info msg="Executing migration" id="create folder table" 2024-09-14T03:42:18.417 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.131185567Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-14T03:42:18.417 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.134007807Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-14T03:42:18.417 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.136863824Z level=info msg="Executing migration" id="Update folder title length" 2024-09-14T03:42:18.417 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.139681452Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-14T03:42:18.418 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=migrator t=2024-09-14T03:42:18.14245804Z level=info msg="migrations completed" performed=485 skipped=0 duration=712.748258ms 2024-09-14T03:42:18.418 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=sqlstore t=2024-09-14T03:42:18.14286806Z level=info msg="Created default organization" 2024-09-14T03:42:18.418 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=plugin.loader t=2024-09-14T03:42:18.202411714Z level=info msg="Plugin registered" pluginID=input 2024-09-14T03:42:18.418 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=plugin.loader t=2024-09-14T03:42:18.230405325Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-14T03:42:18.418 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=plugin.loader t=2024-09-14T03:42:18.230427111Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-14T03:42:18.418 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=secrets t=2024-09-14T03:42:18.230478893Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-14T03:42:18.418 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=query_data t=2024-09-14T03:42:18.232188771Z level=info msg="Query Service initialization" 2024-09-14T03:42:18.418 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=live.push_http t=2024-09-14T03:42:18.234546037Z level=info msg="Live Push Gateway initialization" 2024-09-14T03:42:18.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[29589]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:18.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[29589]: Deploying daemon node-exporter.a on smithi114 2024-09-14T03:42:18.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[40961]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:18.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:18 smithi114 ceph-mon[40961]: Deploying daemon node-exporter.a on smithi114 2024-09-14T03:42:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-mon[32702]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:18.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-mon[32702]: Deploying daemon node-exporter.a on smithi114 2024-09-14T03:42:19.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=infra.usagestats.collector t=2024-09-14T03:42:18.978005864Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:18 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=provisioning.datasources t=2024-09-14T03:42:18.978359375Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=provisioning.datasources t=2024-09-14T03:42:19.010803894Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=provisioning.datasources t=2024-09-14T03:42:19.042844567Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=provisioning.alerting t=2024-09-14T03:42:19.0645756Z level=info msg="starting to provision alerting" 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=provisioning.alerting t=2024-09-14T03:42:19.064613381Z level=info msg="finished to provision alerting" 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=grafanaStorageLogger t=2024-09-14T03:42:19.064776464Z level=info msg="storage starting" 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=ngalert.state.manager t=2024-09-14T03:42:19.064762024Z level=info msg="Warming state cache for startup" 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=http.server t=2024-09-14T03:42:19.066505672Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=sqlstore.transactions t=2024-09-14T03:42:19.075326564Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=ngalert.state.manager t=2024-09-14T03:42:19.075582331Z level=info msg="State cache has been initialized" states=0 duration=10.817082ms 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=ticker t=2024-09-14T03:42:19.075708479Z level=info msg=starting first_tick=2024-09-14T03:42:20Z 2024-09-14T03:42:19.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=ngalert.multiorg.alertmanager t=2024-09-14T03:42:19.075719742Z level=info msg="starting MultiOrg Alertmanager" 2024-09-14T03:42:19.410 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:42:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[64270]: logger=sqlstore.transactions t=2024-09-14T03:42:19.122007917Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-14T03:42:20.575 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:20 smithi114 ceph-mon[29589]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:20.576 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:20 smithi114 ceph-mon[40961]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:20.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:20 smithi160 ceph-mon[32702]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:21.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:42:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:42:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:42:22.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:22 smithi114 ceph-mon[29589]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:22.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:22 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:22.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:22 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:22 smithi114 ceph-mon[40961]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:22 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:22 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:22 smithi160 ceph-mon[32702]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:22 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:22 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:24.593 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:24 smithi114 ceph-mon[40961]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:24.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:24 smithi114 ceph-mon[29589]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:24 smithi160 ceph-mon[32702]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:26.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:26 smithi114 ceph-mon[29589]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:26.557 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:26 smithi114 ceph-mon[40961]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:26 smithi160 ceph-mon[32702]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:28.055 INFO:teuthology.orchestra.run.smithi114.stdout:77309411366 2024-09-14T03:42:28.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-14T03:42:28.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:28 smithi114 ceph-mon[29589]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:28.315 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:28 smithi114 ceph-mon[40961]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:28 smithi160 ceph-mon[32702]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:28.751 INFO:teuthology.orchestra.run.smithi114.stdout:180388626451 2024-09-14T03:42:28.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-14T03:42:30.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:30 smithi114 ceph-mon[29589]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:30.255 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:30 smithi114 ceph-mon[40961]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:30.488 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:30 smithi160 ceph-mon[32702]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:30.819 INFO:teuthology.orchestra.run.smithi114.stdout:107374182433 2024-09-14T03:42:30.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-14T03:42:31.698 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:42:31 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:42:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:42:32.342 INFO:teuthology.orchestra.run.smithi114.stdout:34359738415 2024-09-14T03:42:32.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-14T03:42:32.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:32 smithi114 ceph-mon[29589]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:32.540 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:32 smithi114 ceph-mon[40961]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:32 smithi160 ceph-mon[32702]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:33.370 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:33 smithi114 podman[74246]: 2024-09-14 03:42:33.012880094 +0000 UTC m=+4.785643593 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-14T03:42:33.792 INFO:teuthology.orchestra.run.smithi114.stdout:154618822681 2024-09-14T03:42:33.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-14T03:42:34.342 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:34 smithi114 ceph-mon[29589]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:34.342 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:34 smithi114 ceph-mon[40961]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:34.343 INFO:teuthology.orchestra.run.smithi114.stdout:55834574892 2024-09-14T03:42:34.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-14T03:42:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:34 smithi160 ceph-mon[32702]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:35.043 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:34 smithi114 podman[74246]: 2024-09-14 03:42:34.672291468 +0000 UTC m=+6.445054963 container create 40522b00d68c4486465a77ac08abd8faa62f2bbd5c27cc09be07663e1a352e83 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 podman[74246]: 2024-09-14 03:42:35.042207072 +0000 UTC m=+6.814970566 container init 40522b00d68c4486465a77ac08abd8faa62f2bbd5c27cc09be07663e1a352e83 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 podman[74246]: 2024-09-14 03:42:35.04700392 +0000 UTC m=+6.819767415 container start 40522b00d68c4486465a77ac08abd8faa62f2bbd5c27cc09be07663e1a352e83 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.051Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.051Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=arp 2024-09-14T03:42:35.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-14T03:42:35.372 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-14T03:42:35.372 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-14T03:42:35.372 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-14T03:42:35.372 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-14T03:42:35.372 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-14T03:42:35.372 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=edac 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-14T03:42:35.373 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=os 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-14T03:42:35.374 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=stat 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=time 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=uname 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.052Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[74859]: ts=2024-09-14T03:42:35.053Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 bash[74246]: 40522b00d68c4486465a77ac08abd8faa62f2bbd5c27cc09be07663e1a352e83 2024-09-14T03:42:35.375 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:42:35 smithi114 systemd[1]: Started Ceph node-exporter.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:42:35.553 INFO:teuthology.orchestra.run.smithi114.stdout:206158430224 2024-09-14T03:42:35.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-14T03:42:35.601 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:36.056 INFO:teuthology.orchestra.run.smithi114.stdout:128849018908 2024-09-14T03:42:36.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-14T03:42:36.074 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:36.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:36 smithi160 bash[64550]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-14T03:42:36.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:36 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:36.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:36 smithi114 ceph-mon[29589]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:36.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:36 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:36.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:36 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:36.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:36 smithi114 ceph-mon[29589]: Deploying daemon node-exporter.b on smithi160 2024-09-14T03:42:36.286 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:36 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:36.286 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:36 smithi114 ceph-mon[40961]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:36.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:36 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:36.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:36 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:36.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:36 smithi114 ceph-mon[40961]: Deploying daemon node-exporter.b on smithi160 2024-09-14T03:42:36.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:36 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:36.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:36 smithi160 ceph-mon[32702]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:36.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:36 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:36.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:36 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:36.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:36 smithi160 ceph-mon[32702]: Deploying daemon node-exporter.b on smithi160 2024-09-14T03:42:36.908 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:36 smithi160 bash[64550]: Getting image source signatures 2024-09-14T03:42:36.908 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:36 smithi160 bash[64550]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-14T03:42:36.908 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:36 smithi160 bash[64550]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-14T03:42:36.908 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:36 smithi160 bash[64550]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-14T03:42:37.006 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:37.007 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:37.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:37 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:37.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:37 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:37 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:37.903 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:37.950 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:37.950 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:38.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:37 smithi160 bash[64550]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-14T03:42:38.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:37 smithi160 bash[64550]: Writing manifest to image destination 2024-09-14T03:42:38.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:38 smithi114 ceph-mon[29589]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:38.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:38 smithi114 ceph-mon[40961]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:38.452 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-mon[32702]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:38.452 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 podman[64550]: 2024-09-14 03:42:38.142932425 +0000 UTC m=+2.182165869 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-14T03:42:38.452 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 podman[64550]: 2024-09-14 03:42:38.249973172 +0000 UTC m=+2.289206652 container create a97cccb21ef2351615d919194581484a99cd187bfcfac45877fec8f23f17f2b0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T03:42:38.664 INFO:teuthology.orchestra.run.smithi114.stdout:180388626454 2024-09-14T03:42:38.908 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 podman[64550]: 2024-09-14 03:42:38.451724899 +0000 UTC m=+2.490958334 container init a97cccb21ef2351615d919194581484a99cd187bfcfac45877fec8f23f17f2b0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T03:42:38.909 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 podman[64550]: 2024-09-14 03:42:38.456429924 +0000 UTC m=+2.495663358 container start a97cccb21ef2351615d919194581484a99cd187bfcfac45877fec8f23f17f2b0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T03:42:38.909 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.460Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-14T03:42:38.909 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.460Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-14T03:42:38.909 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.460Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-14T03:42:38.909 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.460Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-14T03:42:38.909 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.460Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-14T03:42:38.909 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.460Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-14T03:42:38.909 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-14T03:42:38.909 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=arp 2024-09-14T03:42:38.909 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-14T03:42:38.910 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-14T03:42:38.910 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-14T03:42:38.910 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-14T03:42:38.910 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-14T03:42:38.910 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-14T03:42:38.910 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-14T03:42:38.910 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-14T03:42:38.910 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=edac 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-14T03:42:38.911 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=os 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-14T03:42:38.912 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=stat 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=time 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=uname 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-14T03:42:38.913 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[64626]: ts=2024-09-14T03:42:38.461Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-14T03:42:38.914 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 bash[64550]: a97cccb21ef2351615d919194581484a99cd187bfcfac45877fec8f23f17f2b0 2024-09-14T03:42:38.914 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:42:38 smithi160 systemd[1]: Started Ceph node-exporter.b for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:42:39.444 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626451 got 180388626454 for osd.6 2024-09-14T03:42:39.445 DEBUG:teuthology.parallel:result is None 2024-09-14T03:42:39.673 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1272461241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1272461241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-14T03:42:39.675 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:39 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:42:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:39 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:39 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:39 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:39 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:39 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:39 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1272461241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-14T03:42:39.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:39 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:42:40.979 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:40 smithi114 ceph-mon[29589]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:40.980 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:40 smithi114 ceph-mon[40961]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:41.028 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:40 smithi160 ceph-mon[32702]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:41.731 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:42:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:42:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:42:42.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:42 smithi114 ceph-mon[40961]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:42 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:42 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:42 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:42.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:42 smithi114 ceph-mon[29589]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:42.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:42 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:42.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:42 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:42.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:42 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:42 smithi160 ceph-mon[32702]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:42 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:42 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:42 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:44.395 INFO:teuthology.orchestra.run.smithi114.stdout:34359738419 2024-09-14T03:42:44.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:44 smithi114 ceph-mon[29589]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:44.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:44 smithi114 ceph-mon[40961]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:44 smithi160 ceph-mon[32702]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:45.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:45 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/4114470213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-14T03:42:45.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:45 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/4114470213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-14T03:42:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:45 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/4114470213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-14T03:42:45.902 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738415 got 34359738419 for osd.0 2024-09-14T03:42:45.903 DEBUG:teuthology.parallel:result is None 2024-09-14T03:42:46.539 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:46 smithi114 ceph-mon[29589]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:46.540 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:46 smithi114 ceph-mon[40961]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:46 smithi160 ceph-mon[32702]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:47.021 INFO:teuthology.orchestra.run.smithi114.stdout:77309411370 2024-09-14T03:42:47.396 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:47 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3616922347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-14T03:42:47.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:47 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3616922347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-14T03:42:47.465 INFO:teuthology.orchestra.run.smithi114.stdout:107374182437 2024-09-14T03:42:47.551 INFO:teuthology.orchestra.run.smithi114.stdout:128849018912 2024-09-14T03:42:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:47 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3616922347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-14T03:42:48.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:48 smithi114 ceph-mon[29589]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:48.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:48 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/494629675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-14T03:42:48.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:48 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/4046781249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-14T03:42:48.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:48 smithi114 ceph-mon[40961]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:48.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:48 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/494629675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-14T03:42:48.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:48 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/4046781249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-14T03:42:48.326 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018908 got 128849018912 for osd.4 2024-09-14T03:42:48.327 DEBUG:teuthology.parallel:result is None 2024-09-14T03:42:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:48 smithi160 ceph-mon[32702]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:48 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/494629675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-14T03:42:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:48 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/4046781249' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-14T03:42:48.735 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182433 got 107374182437 for osd.3 2024-09-14T03:42:48.735 DEBUG:teuthology.parallel:result is None 2024-09-14T03:42:49.114 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:49.525 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411370 for osd.2 2024-09-14T03:42:49.525 DEBUG:teuthology.parallel:result is None 2024-09-14T03:42:49.873 INFO:teuthology.orchestra.run.smithi114.stdout:55834574896 2024-09-14T03:42:50.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:50 smithi114 ceph-mon[40961]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:50.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:50 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2723533758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-14T03:42:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:50 smithi114 ceph-mon[29589]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:50 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2723533758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-14T03:42:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:50 smithi160 ceph-mon[32702]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:50 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2723533758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-14T03:42:50.944 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574896 for osd.1 2024-09-14T03:42:50.945 DEBUG:teuthology.parallel:result is None 2024-09-14T03:42:51.536 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:51.537 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:42:51.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:42:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:42:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:42:52.203 INFO:teuthology.orchestra.run.smithi114.stdout:206158430227 2024-09-14T03:42:52.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:52 smithi114 ceph-mon[29589]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:52.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:52 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1380416029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-14T03:42:52.476 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:52 smithi114 ceph-mon[40961]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:52.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:52 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1380416029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-14T03:42:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:52 smithi160 ceph-mon[32702]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:52 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1380416029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-14T03:42:53.329 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430224 got 206158430227 for osd.7 2024-09-14T03:42:53.330 DEBUG:teuthology.parallel:result is None 2024-09-14T03:42:53.633 INFO:teuthology.orchestra.run.smithi114.stdout:154618822685 2024-09-14T03:42:54.250 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822681 got 154618822685 for osd.5 2024-09-14T03:42:54.251 DEBUG:teuthology.parallel:result is None 2024-09-14T03:42:54.251 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-14T03:42:54.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T03:42:54.588 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:54 smithi114 ceph-mon[40961]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:54.588 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:54 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1251131787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-14T03:42:54.589 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:54 smithi114 ceph-mon[29589]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:54.589 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:54 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1251131787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-14T03:42:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:54 smithi160 ceph-mon[32702]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:54 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1251131787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-14T03:42:54.988 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:56.578 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:56 smithi114 ceph-mon[40961]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:56.578 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:56 smithi114 ceph-mon[29589]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:56 smithi160 ceph-mon[32702]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:57.021 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:42:57.022 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T03:42:57.824 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-09-14T03:42:55.184062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385088,"kb_used_data":6240,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747576128,"statfs":{"total":767960285184,"available":765517955072,"internally_reserved":0,"allocated":6389760,"data_stored":3715423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001781"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.200976+0000","last_change":"2024-09-14T03:42:01.466959+0000","last_active":"2024-09-14T03:42:05.200976+0000","last_peered":"2024-09-14T03:42:05.200976+0000","last_clean":"2024-09-14T03:42:05.200976+0000","last_became_active":"2024-09-14T03:41:56.432058+0000","last_became_peered":"2024-09-14T03:41:56.432058+0000","last_unstale":"2024-09-14T03:42:05.200976+0000","last_undegraded":"2024-09-14T03:42:05.200976+0000","last_fullsized":"2024-09-14T03:42:05.200976+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:58:30.284511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.072382+0000","last_change":"2024-09-14T03:42:01.467292+0000","last_active":"2024-09-14T03:42:05.072382+0000","last_peered":"2024-09-14T03:42:05.072382+0000","last_clean":"2024-09-14T03:42:05.072382+0000","last_became_active":"2024-09-14T03:41:56.770678+0000","last_became_peered":"2024-09-14T03:41:56.770678+0000","last_unstale":"2024-09-14T03:42:05.072382+0000","last_undegraded":"2024-09-14T03:42:05.072382+0000","last_fullsized":"2024-09-14T03:42:05.072382+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:31:37.700007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040100900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.160273+0000","last_change":"2024-09-14T03:42:02.106658+0000","last_active":"2024-09-14T03:42:05.160273+0000","last_peered":"2024-09-14T03:42:05.160273+0000","last_clean":"2024-09-14T03:42:05.160273+0000","last_became_active":"2024-09-14T03:41:56.814735+0000","last_became_peered":"2024-09-14T03:41:56.814735+0000","last_unstale":"2024-09-14T03:42:05.160273+0000","last_undegraded":"2024-09-14T03:42:05.160273+0000","last_fullsized":"2024-09-14T03:42:05.160273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:02:06.281098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.072365+0000","last_change":"2024-09-14T03:42:01.467194+0000","last_active":"2024-09-14T03:42:05.072365+0000","last_peered":"2024-09-14T03:42:05.072365+0000","last_clean":"2024-09-14T03:42:05.072365+0000","last_became_active":"2024-09-14T03:41:56.814429+0000","last_became_peered":"2024-09-14T03:41:56.814429+0000","last_unstale":"2024-09-14T03:42:05.072365+0000","last_undegraded":"2024-09-14T03:42:05.072365+0000","last_fullsized":"2024-09-14T03:42:05.072365+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:00:09.573444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":21,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.493628+0000","last_change":"2024-09-14T03:42:01.466148+0000","last_active":"2024-09-14T03:42:05.493628+0000","last_peered":"2024-09-14T03:42:05.493628+0000","last_clean":"2024-09-14T03:42:05.493628+0000","last_became_active":"2024-09-14T03:41:56.432336+0000","last_became_peered":"2024-09-14T03:41:56.432336+0000","last_unstale":"2024-09-14T03:42:05.493628+0000","last_undegraded":"2024-09-14T03:42:05.493628+0000","last_fullsized":"2024-09-14T03:42:05.493628+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:33:46.232955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.330522+0000","last_change":"2024-09-14T03:42:01.461806+0000","last_active":"2024-09-14T03:42:05.330522+0000","last_peered":"2024-09-14T03:42:05.330522+0000","last_clean":"2024-09-14T03:42:05.330522+0000","last_became_active":"2024-09-14T03:41:56.814735+0000","last_became_peered":"2024-09-14T03:41:56.814735+0000","last_unstale":"2024-09-14T03:42:05.330522+0000","last_undegraded":"2024-09-14T03:42:05.330522+0000","last_fullsized":"2024-09-14T03:42:05.330522+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:09:02.739141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.160273+0000","last_change":"2024-09-14T03:42:02.106608+0000","last_active":"2024-09-14T03:42:05.160273+0000","last_peered":"2024-09-14T03:42:05.160273+0000","last_clean":"2024-09-14T03:42:05.160273+0000","last_became_active":"2024-09-14T03:41:56.814765+0000","last_became_peered":"2024-09-14T03:41:56.814765+0000","last_unstale":"2024-09-14T03:42:05.160273+0000","last_undegraded":"2024-09-14T03:42:05.160273+0000","last_fullsized":"2024-09-14T03:42:05.160273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:15:23.921629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.493631+0000","last_change":"2024-09-14T03:42:01.465792+0000","last_active":"2024-09-14T03:42:05.493631+0000","last_peered":"2024-09-14T03:42:05.493631+0000","last_clean":"2024-09-14T03:42:05.493631+0000","last_became_active":"2024-09-14T03:41:56.433826+0000","last_became_peered":"2024-09-14T03:41:56.433826+0000","last_unstale":"2024-09-14T03:42:05.493631+0000","last_undegraded":"2024-09-14T03:42:05.493631+0000","last_fullsized":"2024-09-14T03:42:05.493631+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:45:32.233869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"51'79","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-14T03:42:02.106382+0000","last_change":"2024-09-14T03:41:25.453993+0000","last_active":"2024-09-14T03:42:02.106382+0000","last_peered":"2024-09-14T03:42:02.106382+0000","last_clean":"2024-09-14T03:42:02.106382+0000","last_became_active":"2024-09-14T03:41:25.446786+0000","last_became_peered":"2024-09-14T03:41:25.446786+0000","last_unstale":"2024-09-14T03:42:02.106382+0000","last_undegraded":"2024-09-14T03:42:02.106382+0000","last_fullsized":"2024-09-14T03:42:02.106382+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:50:17.999993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430228,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298500,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446652,"statfs":{"total":95995035648,"available":95689371648,"internally_reserved":0,"allocated":1171456,"data_stored":833637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.66100000000000003}]},{"osd":1,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.379},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.42799999999999999}]},{"osd":6,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46100000000000002}]}]},{"osd":6,"up_from":42,"seq":180388626457,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298500,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446652,"statfs":{"total":95995035648,"available":95689371648,"internally_reserved":0,"allocated":1171456,"data_stored":833341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.70599999999999996}]},{"osd":2,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.61599999999999999}]},{"osd":4,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.58999999999999997}]},{"osd":5,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64500000000000002}]},{"osd":7,"last update":"Sat Sep 14 03:42:27 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.73099999999999998}]}]},{"osd":5,"up_from":36,"seq":154618822685,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":242658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.61499999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.64300000000000002}]},{"osd":1,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67300000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.40600000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.39000000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.625}]},{"osd":3,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.433,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.67900000000000005,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.72499999999999998}]},{"osd":4,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.41099999999999998,"15min":0.40600000000000003},"min":{"1min":0.32800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.40999999999999998,"15min":0.39700000000000002},"min":{"1min":0.31900000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Sep 14 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.755}]},{"osd":7,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.69199999999999995}]}]},{"osd":4,"up_from":30,"seq":128849018913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":242954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.36399999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51400000000000001,"15min":0.51900000000000002},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.59699999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.53100000000000003,"15min":0.53700000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.53800000000000003,"15min":0.54500000000000004},"min":{"1min":0.35899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.66200000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.60399999999999998}]},{"osd":2,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53100000000000003,"15min":0.53900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.626,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71999999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.497}]},{"osd":3,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68000000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.66600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.625}]},{"osd":5,"last update":"Sat Sep 14 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.34499999999999997,"15min":0.33800000000000002},"min":{"1min":0.28499999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33000000000000002},"min":{"1min":0.26400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.35499999999999998}]},{"osd":6,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.44400000000000001}]},{"osd":7,"last update":"Sat Sep 14 03:42:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.38600000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":243269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:41:56 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.374,"15min":0.374},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.40999999999999998,"15min":0.40500000000000003},"min":{"1min":0.29199999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.40200000000000002}]},{"osd":1,"last update":"Sat Sep 14 03:41:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.40799999999999997,"15min":0.40600000000000003},"min":{"1min":0.255,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.307},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42799999999999999,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.58499999999999996,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.45200000000000001}]},{"osd":2,"last update":"Sat Sep 14 03:41:56 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.32800000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.56599999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.39700000000000002,"15min":0.39600000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.59699999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.47899999999999998}]},{"osd":4,"last update":"Sat Sep 14 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.52100000000000002,"15min":0.51100000000000001},"min":{"1min":0.41199999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.69899999999999995}]},{"osd":5,"last update":"Sat Sep 14 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.81200000000000006,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77600000000000002,"5min":0.876,"15min":0.876},"last":0.59599999999999997}]},{"osd":6,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.63200000000000001}]},{"osd":7,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.67100000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":242954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.374,"15min":0.36499999999999999},"min":{"1min":0.24099999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.66600000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42799999999999999}]},{"osd":1,"last update":"Sat Sep 14 03:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.26100000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.52600000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39300000000000002,"15min":0.39100000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61499999999999999,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.41099999999999998}]},{"osd":3,"last update":"Sat Sep 14 03:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.41799999999999998,"15min":0.41599999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63500000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.499}]},{"osd":4,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.68400000000000005}]},{"osd":5,"last update":"Sat Sep 14 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.436,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.77400000000000002,"5min":1,"15min":1},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.622}]},{"osd":6,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.67700000000000005}]},{"osd":7,"last update":"Sat Sep 14 03:42:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.755}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297924,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447228,"statfs":{"total":95995035648,"available":95689961472,"internally_reserved":0,"allocated":581632,"data_stored":243288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34000000000000002,"15min":0.33900000000000002},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.47099999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.371,"5min":0.35799999999999998,"15min":0.34300000000000003},"min":{"1min":0.23100000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.52900000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.36299999999999999}]},{"osd":2,"last update":"Sat Sep 14 03:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.36199999999999999,"15min":0.35399999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36499999999999999,"15min":0.35699999999999998},"min":{"1min":0.26700000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55700000000000005,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.29399999999999998}]},{"osd":3,"last update":"Sat Sep 14 03:41:54 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.40799999999999997,"15min":0.41099999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52100000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.378}]},{"osd":4,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.57699999999999996}]},{"osd":5,"last update":"Sat Sep 14 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57199999999999995,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.64000000000000001}]},{"osd":6,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.502},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60699999999999998}]},{"osd":7,"last update":"Sat Sep 14 03:42:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298496,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446656,"statfs":{"total":95995035648,"available":95689375744,"internally_reserved":0,"allocated":1167360,"data_stored":833322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 14 03:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.34100000000000003,"15min":0.32700000000000001},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.34599999999999997,"15min":0.32000000000000001},"min":{"1min":0.28899999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Sep 14 03:42:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36899999999999999,"15min":0.36399999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56399999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.38300000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.41499999999999998}]},{"osd":3,"last update":"Sat Sep 14 03:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41299999999999998,"15min":0.40999999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.434,"15min":0.42999999999999999},"min":{"1min":0.308,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.54300000000000004,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Sep 14 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.746},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71199999999999997,"5min":0.754,"15min":0.754},"last":0.73499999999999999}]},{"osd":5,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68200000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.625},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.71299999999999997}]},{"osd":6,"last update":"Sat Sep 14 03:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.60399999999999998}]},{"osd":7,"last update":"Sat Sep 14 03:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T03:42:57.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T03:42:58.084 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:42:58.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[29589]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:42:58.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[29589]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:58.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:58.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:58.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:42:58.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:42:58.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:58.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[40961]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:42:58.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[40961]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:58.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:58.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:58.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:42:58.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:42:58.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:58 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:58 smithi160 ceph-mon[32702]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:42:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:58 smithi160 ceph-mon[32702]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:42:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:58 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:58 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:58 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:42:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:58 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:42:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:58 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:42:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:59 smithi114 ceph-mon[29589]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-14T03:42:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:42:59 smithi114 ceph-mon[29589]: Reconfiguring daemon alertmanager.a on smithi114 2024-09-14T03:42:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:59 smithi114 ceph-mon[40961]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-14T03:42:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:42:59 smithi114 ceph-mon[40961]: Reconfiguring daemon alertmanager.a on smithi114 2024-09-14T03:42:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:59 smithi160 ceph-mon[32702]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-14T03:42:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:42:59 smithi160 ceph-mon[32702]: Reconfiguring daemon alertmanager.a on smithi114 2024-09-14T03:43:00.463 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:00 smithi114 ceph-mon[29589]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:00.464 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:00 smithi114 ceph-mon[40961]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:00 smithi160 ceph-mon[32702]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:01.778 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:43:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:43:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:43:02.227 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:02 smithi114 systemd[1]: Stopping Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:43:02.573 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:02 smithi114 ceph-mon[29589]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:02.574 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:02 smithi114 ceph-mon[40961]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:02.588 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:43:02.588 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T03:43:02.589 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:02 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[71961]: ts=2024-09-14T03:43:02.572Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-14T03:43:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:02 smithi160 ceph-mon[32702]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:02.870 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:02 smithi114 podman[77563]: 2024-09-14 03:43:02.5928816 +0000 UTC m=+0.120907069 container died 4a0609b0f113d5f8b8a2f0172548136e319e75792d4d2d2b43f8d594d7b8112b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:03.266 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:02 smithi114 podman[77563]: 2024-09-14 03:43:02.987423894 +0000 UTC m=+0.515449344 container cleanup 4a0609b0f113d5f8b8a2f0172548136e319e75792d4d2d2b43f8d594d7b8112b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:03.266 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:02 smithi114 bash[77563]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a 2024-09-14T03:43:03.542 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:03 smithi114 ceph-mon[29589]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:43:03.543 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:03 smithi114 ceph-mon[40961]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:43:03.543 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:03 smithi114 podman[77582]: 2024-09-14 03:43:03.448032529 +0000 UTC m=+0.853287114 container remove 4a0609b0f113d5f8b8a2f0172548136e319e75792d4d2d2b43f8d594d7b8112b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:03 smithi160 ceph-mon[32702]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:43:03.870 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:03 smithi114 podman[77582]: 2024-09-14 03:43:03.542039725 +0000 UTC m=+0.947294316 volume remove a470b79eb821d548f35d4351d88103c5cb035f686693b29a305986b98296fcba 2024-09-14T03:43:03.933 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-14T03:43:01.184924+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385088,"kb_used_data":6240,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747576128,"statfs":{"total":767960285184,"available":765517955072,"internally_reserved":0,"allocated":6389760,"data_stored":3715423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001756"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.200976+0000","last_change":"2024-09-14T03:42:01.466959+0000","last_active":"2024-09-14T03:42:05.200976+0000","last_peered":"2024-09-14T03:42:05.200976+0000","last_clean":"2024-09-14T03:42:05.200976+0000","last_became_active":"2024-09-14T03:41:56.432058+0000","last_became_peered":"2024-09-14T03:41:56.432058+0000","last_unstale":"2024-09-14T03:42:05.200976+0000","last_undegraded":"2024-09-14T03:42:05.200976+0000","last_fullsized":"2024-09-14T03:42:05.200976+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:58:30.284511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.072382+0000","last_change":"2024-09-14T03:42:01.467292+0000","last_active":"2024-09-14T03:42:05.072382+0000","last_peered":"2024-09-14T03:42:05.072382+0000","last_clean":"2024-09-14T03:42:05.072382+0000","last_became_active":"2024-09-14T03:41:56.770678+0000","last_became_peered":"2024-09-14T03:41:56.770678+0000","last_unstale":"2024-09-14T03:42:05.072382+0000","last_undegraded":"2024-09-14T03:42:05.072382+0000","last_fullsized":"2024-09-14T03:42:05.072382+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:31:37.700007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040100900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.160273+0000","last_change":"2024-09-14T03:42:02.106658+0000","last_active":"2024-09-14T03:42:05.160273+0000","last_peered":"2024-09-14T03:42:05.160273+0000","last_clean":"2024-09-14T03:42:05.160273+0000","last_became_active":"2024-09-14T03:41:56.814735+0000","last_became_peered":"2024-09-14T03:41:56.814735+0000","last_unstale":"2024-09-14T03:42:05.160273+0000","last_undegraded":"2024-09-14T03:42:05.160273+0000","last_fullsized":"2024-09-14T03:42:05.160273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:02:06.281098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.072365+0000","last_change":"2024-09-14T03:42:01.467194+0000","last_active":"2024-09-14T03:42:05.072365+0000","last_peered":"2024-09-14T03:42:05.072365+0000","last_clean":"2024-09-14T03:42:05.072365+0000","last_became_active":"2024-09-14T03:41:56.814429+0000","last_became_peered":"2024-09-14T03:41:56.814429+0000","last_unstale":"2024-09-14T03:42:05.072365+0000","last_undegraded":"2024-09-14T03:42:05.072365+0000","last_fullsized":"2024-09-14T03:42:05.072365+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:00:09.573444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":21,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.493628+0000","last_change":"2024-09-14T03:42:01.466148+0000","last_active":"2024-09-14T03:42:05.493628+0000","last_peered":"2024-09-14T03:42:05.493628+0000","last_clean":"2024-09-14T03:42:05.493628+0000","last_became_active":"2024-09-14T03:41:56.432336+0000","last_became_peered":"2024-09-14T03:41:56.432336+0000","last_unstale":"2024-09-14T03:42:05.493628+0000","last_undegraded":"2024-09-14T03:42:05.493628+0000","last_fullsized":"2024-09-14T03:42:05.493628+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:33:46.232955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.330522+0000","last_change":"2024-09-14T03:42:01.461806+0000","last_active":"2024-09-14T03:42:05.330522+0000","last_peered":"2024-09-14T03:42:05.330522+0000","last_clean":"2024-09-14T03:42:05.330522+0000","last_became_active":"2024-09-14T03:41:56.814735+0000","last_became_peered":"2024-09-14T03:41:56.814735+0000","last_unstale":"2024-09-14T03:42:05.330522+0000","last_undegraded":"2024-09-14T03:42:05.330522+0000","last_fullsized":"2024-09-14T03:42:05.330522+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:09:02.739141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.160273+0000","last_change":"2024-09-14T03:42:02.106608+0000","last_active":"2024-09-14T03:42:05.160273+0000","last_peered":"2024-09-14T03:42:05.160273+0000","last_clean":"2024-09-14T03:42:05.160273+0000","last_became_active":"2024-09-14T03:41:56.814765+0000","last_became_peered":"2024-09-14T03:41:56.814765+0000","last_unstale":"2024-09-14T03:42:05.160273+0000","last_undegraded":"2024-09-14T03:42:05.160273+0000","last_fullsized":"2024-09-14T03:42:05.160273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:15:23.921629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-14T03:42:05.493631+0000","last_change":"2024-09-14T03:42:01.465792+0000","last_active":"2024-09-14T03:42:05.493631+0000","last_peered":"2024-09-14T03:42:05.493631+0000","last_clean":"2024-09-14T03:42:05.493631+0000","last_became_active":"2024-09-14T03:41:56.433826+0000","last_became_peered":"2024-09-14T03:41:56.433826+0000","last_unstale":"2024-09-14T03:42:05.493631+0000","last_undegraded":"2024-09-14T03:42:05.493631+0000","last_fullsized":"2024-09-14T03:42:05.493631+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:45:32.233869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"51'79","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-14T03:42:02.106382+0000","last_change":"2024-09-14T03:41:25.453993+0000","last_active":"2024-09-14T03:42:02.106382+0000","last_peered":"2024-09-14T03:42:02.106382+0000","last_clean":"2024-09-14T03:42:02.106382+0000","last_became_active":"2024-09-14T03:41:25.446786+0000","last_became_peered":"2024-09-14T03:41:25.446786+0000","last_unstale":"2024-09-14T03:42:02.106382+0000","last_undegraded":"2024-09-14T03:42:02.106382+0000","last_fullsized":"2024-09-14T03:42:02.106382+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:50:17.999993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430229,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298500,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446652,"statfs":{"total":95995035648,"available":95689371648,"internally_reserved":0,"allocated":1171456,"data_stored":833637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.60599999999999998}]},{"osd":1,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51900000000000002}]},{"osd":2,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.54700000000000004}]},{"osd":3,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.49399999999999999}]},{"osd":4,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.42399999999999999}]},{"osd":5,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.56200000000000006}]},{"osd":6,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53800000000000003}]}]},{"osd":6,"up_from":42,"seq":180388626458,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298500,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446652,"statfs":{"total":95995035648,"available":95689371648,"internally_reserved":0,"allocated":1171456,"data_stored":833341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.50700000000000001}]},{"osd":1,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60799999999999998}]},{"osd":2,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.52000000000000002}]},{"osd":3,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.47299999999999998}]},{"osd":4,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.34699999999999998}]},{"osd":5,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.376},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.39200000000000002}]},{"osd":7,"last update":"Sat Sep 14 03:42:27 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42999999999999999}]}]},{"osd":5,"up_from":36,"seq":154618822686,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":242658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.61499999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.51600000000000001}]},{"osd":1,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67300000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.40600000000000003,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51600000000000001,"15min":0.51200000000000001},"min":{"1min":0.39000000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.433,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.67900000000000005,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.50600000000000001}]},{"osd":4,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.41099999999999998,"15min":0.40600000000000003},"min":{"1min":0.32800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.40999999999999998,"15min":0.39700000000000002},"min":{"1min":0.31900000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Sep 14 03:42:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.626}]},{"osd":7,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.58699999999999997}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":242954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.36399999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70399999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51400000000000001,"15min":0.51900000000000002},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.59699999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42799999999999999}]},{"osd":1,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.53100000000000003,"15min":0.53700000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.62},{"interface":"front","average":{"1min":0.495,"5min":0.53800000000000003,"15min":0.54500000000000004},"min":{"1min":0.35899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.66200000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53100000000000003,"15min":0.53900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.626,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.54700000000000004,"15min":0.55500000000000005},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.71999999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.45000000000000001}]},{"osd":3,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68000000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.66600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.53900000000000003}]},{"osd":5,"last update":"Sat Sep 14 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.34499999999999997,"15min":0.33800000000000002},"min":{"1min":0.28499999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33000000000000002},"min":{"1min":0.26400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.51300000000000001}]},{"osd":6,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.55600000000000005}]},{"osd":7,"last update":"Sat Sep 14 03:42:27 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.65000000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":243269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:41:56 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.374,"15min":0.374},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.31},{"interface":"front","average":{"1min":0.442,"5min":0.40999999999999998,"15min":0.40500000000000003},"min":{"1min":0.29199999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Sep 14 03:41:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.40799999999999997,"15min":0.40600000000000003},"min":{"1min":0.255,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42799999999999999,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.58499999999999996,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Sep 14 03:41:56 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.32800000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.56599999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.39700000000000002,"15min":0.39600000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.59699999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.34000000000000002}]},{"osd":4,"last update":"Sat Sep 14 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.52100000000000002,"15min":0.51100000000000001},"min":{"1min":0.41199999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.75600000000000001}]},{"osd":5,"last update":"Sat Sep 14 03:42:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.81200000000000006,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77600000000000002,"5min":0.876,"15min":0.876},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.59599999999999997}]},{"osd":7,"last update":"Sat Sep 14 03:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.70099999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":242954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.247},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.374,"15min":0.36499999999999999},"min":{"1min":0.24099999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.66600000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.32600000000000001}]},{"osd":1,"last update":"Sat Sep 14 03:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.379,"15min":0.38300000000000001},"min":{"1min":0.26100000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.52600000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39300000000000002,"15min":0.39100000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61499999999999999,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Sep 14 03:42:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.41299999999999998,"15min":0.41399999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.54700000000000004,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.42599999999999999,"15min":0.433},"min":{"1min":0.28999999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.51500000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.313}]},{"osd":4,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.48199999999999998}]},{"osd":5,"last update":"Sat Sep 14 03:42:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.436,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.77400000000000002,"5min":1,"15min":1},"last":0.441},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.496}]},{"osd":6,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.504}]},{"osd":7,"last update":"Sat Sep 14 03:42:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.41899999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297924,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447228,"statfs":{"total":95995035648,"available":95689961472,"internally_reserved":0,"allocated":581632,"data_stored":243288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:42:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34000000000000002,"15min":0.33900000000000002},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.47099999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.35799999999999998,"15min":0.34300000000000003},"min":{"1min":0.23100000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.52900000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.247}]},{"osd":2,"last update":"Sat Sep 14 03:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.36199999999999999,"15min":0.35399999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.36499999999999999,"15min":0.35699999999999998},"min":{"1min":0.26700000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55700000000000005,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.34300000000000003}]},{"osd":3,"last update":"Sat Sep 14 03:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.39400000000000002,"15min":0.40000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.39900000000000002,"15min":0.40799999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.54200000000000004,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.32400000000000001}]},{"osd":4,"last update":"Sat Sep 14 03:42:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56299999999999994}]},{"osd":5,"last update":"Sat Sep 14 03:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.441},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57199999999999995,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.46999999999999997}]},{"osd":6,"last update":"Sat Sep 14 03:42:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.45000000000000001}]},{"osd":7,"last update":"Sat Sep 14 03:42:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.52800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298496,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446656,"statfs":{"total":95995035648,"available":95689375744,"internally_reserved":0,"allocated":1167360,"data_stored":833322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 14 03:42:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.34100000000000003,"15min":0.32700000000000001},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.34599999999999997,"15min":0.32000000000000001},"min":{"1min":0.28899999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Sep 14 03:42:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.36899999999999999,"15min":0.36399999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56399999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.441},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.38300000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Sep 14 03:42:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.40000000000000002,"15min":0.40600000000000003},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.48799999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42599999999999999,"15min":0.42799999999999999},"min":{"1min":0.27400000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.48899999999999999}]},{"osd":4,"last update":"Sat Sep 14 03:42:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71199999999999997,"5min":0.754,"15min":0.754},"last":0.63300000000000001}]},{"osd":5,"last update":"Sat Sep 14 03:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68200000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.624}]},{"osd":6,"last update":"Sat Sep 14 03:42:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58299999999999996}]},{"osd":7,"last update":"Sat Sep 14 03:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.60299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T03:43:03.935 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-14T03:43:03.935 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-14T03:43:03.935 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-14T03:43:03.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-14T03:43:04.267 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-14T03:43:04.267 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 systemd[1]: Stopped Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:43:04.267 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 systemd[1]: Starting Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:43:04.562 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:04 smithi114 ceph-mon[29589]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:04.562 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:04 smithi114 ceph-mon[40961]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:04.563 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 podman[77740]: 2024-09-14 03:43:04.269042842 +0000 UTC m=+0.107815796 volume create 2f8f51a23eee7f9117165f53d552234d7d05da556ec12094f5a0a3faaf0e502e 2024-09-14T03:43:04.563 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 podman[77740]: 2024-09-14 03:43:04.181382533 +0000 UTC m=+0.020155498 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-14T03:43:04.563 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 podman[77740]: 2024-09-14 03:43:04.353141731 +0000 UTC m=+0.191914683 container create 80f53fb9f5983b665626832a2f84e6951abdd2835c6fe723f9a38588db8c1ca3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:04 smithi160 ceph-mon[32702]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:04.914 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:05.036 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 podman[77740]: 2024-09-14 03:43:04.697655113 +0000 UTC m=+0.536428073 container init 80f53fb9f5983b665626832a2f84e6951abdd2835c6fe723f9a38588db8c1ca3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:05.036 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 podman[77740]: 2024-09-14 03:43:04.702777352 +0000 UTC m=+0.541550307 container start 80f53fb9f5983b665626832a2f84e6951abdd2835c6fe723f9a38588db8c1ca3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:05.036 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:43:04.715Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-14T03:43:05.036 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:43:04.715Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-14T03:43:05.036 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:43:04.716Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.114 port=9094 2024-09-14T03:43:05.036 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:43:04.717Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-14T03:43:05.036 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:43:04.740Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-14T03:43:05.037 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:43:04.740Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-14T03:43:05.037 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:43:04.743Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-14T03:43:05.037 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:43:04.743Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-14T03:43:05.037 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 bash[77740]: 80f53fb9f5983b665626832a2f84e6951abdd2835c6fe723f9a38588db8c1ca3 2024-09-14T03:43:05.037 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:04 smithi114 systemd[1]: Started Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:43:06.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:05 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:05 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:05 smithi114 ceph-mon[40961]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-14T03:43:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:05 smithi114 ceph-mon[40961]: Reconfiguring daemon prometheus.a on smithi160 2024-09-14T03:43:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:05 smithi114 ceph-mon[40961]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:05 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:05 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:05 smithi114 ceph-mon[29589]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-14T03:43:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:05 smithi114 ceph-mon[29589]: Reconfiguring daemon prometheus.a on smithi160 2024-09-14T03:43:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:05 smithi114 ceph-mon[29589]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:06.137 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:05 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:06.137 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:05 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:06.137 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:05 smithi160 ceph-mon[32702]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-14T03:43:06.137 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:05 smithi160 ceph-mon[32702]: Reconfiguring daemon prometheus.a on smithi160 2024-09-14T03:43:06.137 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:05 smithi160 ceph-mon[32702]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:06.860 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:43:06.860 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-14T03:43:07.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:06 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:07.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:06 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:07.121 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:43:06.717Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000197285s 2024-09-14T03:43:07.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:06 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:07.482 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-14T03:43:07.482 INFO:tasks.cephadm:Setup complete, yielding 2024-09-14T03:43:07.482 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-14T03:43:07.494 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-14T03:43:07.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- bash -c 'ceph fs volume create foo' 2024-09-14T03:43:07.738 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:08.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:07 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/191590662' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-14T03:43:08.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:07 smithi114 ceph-mon[40961]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:07 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/191590662' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-14T03:43:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:07 smithi114 ceph-mon[29589]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:08.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:07 smithi160 systemd[1]: Stopping Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:43:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:07 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/191590662' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-14T03:43:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:07 smithi160 ceph-mon[32702]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:08.508 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.241Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-14T03:43:08.508 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.241Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-14T03:43:08.508 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.241Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-14T03:43:08.508 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.241Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-14T03:43:08.508 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.241Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-14T03:43:08.508 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.241Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-14T03:43:08.508 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.241Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-14T03:43:08.509 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.241Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-14T03:43:08.509 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.241Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-14T03:43:08.509 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.242Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-14T03:43:08.509 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.242Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-14T03:43:08.509 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[62084]: ts=2024-09-14T03:43:08.242Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-14T03:43:08.509 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 podman[65510]: 2024-09-14 03:43:08.251773646 +0000 UTC m=+0.149846306 container died b32e2d1cd0b0a2fcc11f1ba1c8e7bdff4a24ecb51f7fb82375807c3869f4489f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:08.865 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 podman[65510]: 2024-09-14 03:43:08.6803101 +0000 UTC m=+0.578382759 container cleanup b32e2d1cd0b0a2fcc11f1ba1c8e7bdff4a24ecb51f7fb82375807c3869f4489f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:08.865 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 bash[65510]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a 2024-09-14T03:43:09.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:08 smithi160 podman[65528]: 2024-09-14 03:43:08.864839342 +0000 UTC m=+0.615426425 container remove b32e2d1cd0b0a2fcc11f1ba1c8e7bdff4a24ecb51f7fb82375807c3869f4489f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:09.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-14T03:43:09.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 systemd[1]: Stopped Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:43:09.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.098s CPU time. 2024-09-14T03:43:09.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 systemd[1]: Starting Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:43:09.562 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 podman[65627]: 2024-09-14 03:43:09.205385382 +0000 UTC m=+0.016886953 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-14T03:43:09.562 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 podman[65627]: 2024-09-14 03:43:09.326474858 +0000 UTC m=+0.137976414 container create d05492e182231bf1ad8d4d51c2c6b70679b74993a5e248d69fbb68bafd49f77e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:09.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 podman[65627]: 2024-09-14 03:43:09.561937281 +0000 UTC m=+0.373438838 container init d05492e182231bf1ad8d4d51c2c6b70679b74993a5e248d69fbb68bafd49f77e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:09.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 podman[65627]: 2024-09-14 03:43:09.566687105 +0000 UTC m=+0.378188661 container start d05492e182231bf1ad8d4d51c2c6b70679b74993a5e248d69fbb68bafd49f77e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.590Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.590Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.590Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi160 (none))" 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.590Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.590Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.592Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.593Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.593Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.593Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.595Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-14T03:43:09.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.595Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.516µs 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.595Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.595Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.595Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.595Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.184µs wal_replay_duration=724.162µs wbl_replay_duration=110ns total_replay_duration=759.561µs 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.596Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.596Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.596Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.611Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.329472ms db_storage=843ns remote_storage=1.325µs web_handler=330ns query_engine=703ns scrape=400.017µs scrape_sd=28.16µs notify=19.001µs notify_sd=5.611µs rules=14.620214ms tracing=5.952µs 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.611Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:43:09.611Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 bash[65627]: d05492e182231bf1ad8d4d51c2c6b70679b74993a5e248d69fbb68bafd49f77e 2024-09-14T03:43:09.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:43:09 smithi160 systemd[1]: Started Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:43:10.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:09 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:09] ENGINE Bus STOPPING 2024-09-14T03:43:10.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-14T03:43:10.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE Bus STOPPED 2024-09-14T03:43:10.121 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE Bus STARTING 2024-09-14T03:43:10.435 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE Serving on http://:::9283 2024-09-14T03:43:10.435 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE Bus STARTED 2024-09-14T03:43:10.435 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE Bus STOPPING 2024-09-14T03:43:10.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:10.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-14T03:43:10.436 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:43:10.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:43:10.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-14T03:43:10.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-14T03:43:10.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:43:10.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-14T03:43:10.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:43:10.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-14T03:43:10.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.438 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:43:10.528 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:10.528 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:43:10.528 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-14T03:43:10.528 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-14T03:43:10.528 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.528 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.528 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:43:10.528 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:43:10.528 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-14T03:43:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi114.front.sepia.ceph.com:9093"}]: dispatch 2024-09-14T03:43:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:43:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-14T03:43:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:43:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-14T03:43:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:10.529 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:10 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:43:10.745 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-14T03:43:10.745 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE Bus STOPPED 2024-09-14T03:43:10.745 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE Bus STARTING 2024-09-14T03:43:10.745 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE Serving on http://:::9283 2024-09-14T03:43:10.745 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE Bus STARTED 2024-09-14T03:43:11.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:10 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:10] ENGINE Bus STOPPING 2024-09-14T03:43:11.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:11 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:43:11.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:11 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-14T03:43:11.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:11 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:43:11.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:11 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-14T03:43:11.561 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:11 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-14T03:43:11.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:11 smithi160 ceph-mon[32702]: osdmap e59: 8 total, 8 up, 8 in 2024-09-14T03:43:11.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:11 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:11.562 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:11 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:11.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:43:11.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-14T03:43:11.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:43:11.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-14T03:43:11.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-14T03:43:11.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[40961]: osdmap e59: 8 total, 8 up, 8 in 2024-09-14T03:43:11.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:11.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:11.622 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:11] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-14T03:43:11.622 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:11] ENGINE Bus STOPPED 2024-09-14T03:43:11.622 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:11] ENGINE Bus STARTING 2024-09-14T03:43:11.622 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:11] ENGINE Serving on http://:::9283 2024-09-14T03:43:11.622 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:43:11] ENGINE Bus STARTED 2024-09-14T03:43:11.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:43:11.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi160.front.sepia.ceph.com:9095"}]: dispatch 2024-09-14T03:43:11.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:43:11.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi160.front.sepia.ceph.com:3000"}]: dispatch 2024-09-14T03:43:11.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-14T03:43:11.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[29589]: osdmap e59: 8 total, 8 up, 8 in 2024-09-14T03:43:11.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:11.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:11.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[29566]: 2024-09-14T03:43:11.271+0000 7f4464dd4640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-14T03:43:12.527 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: osdmap e60: 8 total, 8 up, 8 in 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: osdmap e61: 8 total, 8 up, 8 in 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: fsmap foo:0 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: Saving service mds.foo spec with placement count:2 2024-09-14T03:43:12.528 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:12.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:12.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:43:12.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-14T03:43:12.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: osdmap e60: 8 total, 8 up, 8 in 2024-09-14T03:43:12.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:12.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:12.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-14T03:43:12.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-14T03:43:12.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: osdmap e61: 8 total, 8 up, 8 in 2024-09-14T03:43:12.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-14T03:43:12.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: fsmap foo:0 2024-09-14T03:43:12.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: Saving service mds.foo spec with placement count:2 2024-09-14T03:43:12.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:12 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:12.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-14T03:43:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:43:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-14T03:43:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: osdmap e60: 8 total, 8 up, 8 in 2024-09-14T03:43:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-14T03:43:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-14T03:43:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-14T03:43:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: osdmap e61: 8 total, 8 up, 8 in 2024-09-14T03:43:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-14T03:43:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: fsmap foo:0 2024-09-14T03:43:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: Saving service mds.foo spec with placement count:2 2024-09-14T03:43:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:12 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:13.246 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:13.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:13 smithi114 ceph-mon[40961]: osdmap e62: 8 total, 8 up, 8 in 2024-09-14T03:43:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:13 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:13.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:13 smithi114 ceph-mon[29589]: osdmap e62: 8 total, 8 up, 8 in 2024-09-14T03:43:13.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:13 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:13 smithi160 ceph-mon[32702]: osdmap e62: 8 total, 8 up, 8 in 2024-09-14T03:43:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:13 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:14.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:43:14.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:14.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:43:14.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:14.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:14.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-14T03:43:14.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:14.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[40961]: Deploying daemon mds.foo.smithi114.ebmrat on smithi114 2024-09-14T03:43:14.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:14.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:43:14.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:14.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:43:14.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:14.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:14.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-14T03:43:14.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:14.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-mon[29589]: Deploying daemon mds.foo.smithi114.ebmrat on smithi114 2024-09-14T03:43:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:43:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:14.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:43:14.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:14.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:14.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:14.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-14T03:43:14.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:14.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:14 smithi160 ceph-mon[32702]: Deploying daemon mds.foo.smithi114.ebmrat on smithi114 2024-09-14T03:43:14.781 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:43:14 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:43:14.718Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001574654s 2024-09-14T03:43:16.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-14T03:43:16.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:16 smithi114 ceph-mon[29589]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:16.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:16 smithi114 ceph-mon[40961]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:16 smithi160 ceph-mon[32702]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:16.793 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:18.345 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:18 smithi114 ceph-mon[40961]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:18.345 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:18 smithi114 ceph-mon[29589]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:18 smithi160 ceph-mon[32702]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:19.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:43:19] "GET /metrics HTTP/1.1" 200 203810 "" "Prometheus/2.43.0" 2024-09-14T03:43:19.753 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-14T03:43:19.763 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-14T03:43:19.763 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-14T03:43:19.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-14T03:43:20.008 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:20.268 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: Deploying daemon mds.foo.smithi160.ywyjae on smithi160 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: mds.? [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] up:boot 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: daemon mds.foo.smithi114.ebmrat assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-14T03:43:20.269 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-14T03:43:20.270 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: Cluster is now healthy 2024-09-14T03:43:20.270 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: fsmap foo:0 1 up:standby 2024-09-14T03:43:20.270 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"}]: dispatch 2024-09-14T03:43:20.270 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:creating} 2024-09-14T03:43:20.270 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:20 smithi160 ceph-mon[32702]: daemon mds.foo.smithi114.ebmrat is now active in filesystem foo as rank 0 2024-09-14T03:43:20.286 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: Deploying daemon mds.foo.smithi160.ywyjae on smithi160 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: mds.? [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] up:boot 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: daemon mds.foo.smithi114.ebmrat assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-14T03:43:20.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-14T03:43:20.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-14T03:43:20.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: Cluster is now healthy 2024-09-14T03:43:20.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: fsmap foo:0 1 up:standby 2024-09-14T03:43:20.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"}]: dispatch 2024-09-14T03:43:20.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:creating} 2024-09-14T03:43:20.288 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[40961]: daemon mds.foo.smithi114.ebmrat is now active in filesystem foo as rank 0 2024-09-14T03:43:20.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:20.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:20.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: Deploying daemon mds.foo.smithi160.ywyjae on smithi160 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: mds.? [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] up:boot 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: daemon mds.foo.smithi114.ebmrat assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: Cluster is now healthy 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: fsmap foo:0 1 up:standby 2024-09-14T03:43:20.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"}]: dispatch 2024-09-14T03:43:20.290 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:creating} 2024-09-14T03:43:20.290 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:20 smithi114 ceph-mon[29589]: daemon mds.foo.smithi114.ebmrat is now active in filesystem foo as rank 0 2024-09-14T03:43:21.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:21 smithi114 ceph-mon[40961]: mds.? [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] up:active 2024-09-14T03:43:21.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:21 smithi114 ceph-mon[40961]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 2024-09-14T03:43:21.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:21 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:21.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:21 smithi114 ceph-mon[29589]: mds.? [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] up:active 2024-09-14T03:43:21.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:21 smithi114 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 2024-09-14T03:43:21.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:21 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:21.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:43:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:43:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:43:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:21 smithi160 ceph-mon[32702]: mds.? [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] up:active 2024-09-14T03:43:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:21 smithi160 ceph-mon[32702]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 2024-09-14T03:43:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:21 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:21.992 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:43:22.759 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"flags":0,"active_gid":14448,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":1727106713},{"type":"v1","addr":"172.21.15.114:6801","nonce":1727106713}]},"active_addr":"172.21.15.114:6801/1727106713","active_change":"2024-09-14T03:41:51.162981+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14445,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.114:8443/","prometheus":"http://172.21.15.114:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":51,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4089637122}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":546762471}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":3853151361}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2995702914}]}]} 2024-09-14T03:43:22.763 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-14T03:43:22.763 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-14T03:43:22.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-14T03:43:22.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:22.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:43:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3733793991' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-14T03:43:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: mds.? [v2:172.21.15.160:6832/2654584478,v1:172.21.15.160:6833/2654584478] up:boot 2024-09-14T03:43:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"}]: dispatch 2024-09-14T03:43:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[40961]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3733793991' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: mds.? [v2:172.21.15.160:6832/2654584478,v1:172.21.15.160:6833/2654584478] up:boot 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"}]: dispatch 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:22 smithi114 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:43:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3733793991' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-14T03:43:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: mds.? [v2:172.21.15.160:6832/2654584478,v1:172.21.15.160:6833/2654584478] up:boot 2024-09-14T03:43:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"}]: dispatch 2024-09-14T03:43:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:22 smithi160 ceph-mon[32702]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:23.402 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:24.824 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:24 smithi114 ceph-mon[40961]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-09-14T03:43:24.825 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:24 smithi114 ceph-mon[40961]: mds.? [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] up:active 2024-09-14T03:43:24.825 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:24 smithi114 ceph-mon[40961]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:24.825 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:24 smithi114 ceph-mon[29589]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-09-14T03:43:24.825 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:24 smithi114 ceph-mon[29589]: mds.? [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] up:active 2024-09-14T03:43:24.825 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:24 smithi114 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:24 smithi160 ceph-mon[32702]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-09-14T03:43:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:24 smithi160 ceph-mon[32702]: mds.? [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] up:active 2024-09-14T03:43:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:24 smithi160 ceph-mon[32702]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:25.236 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:43:25.236 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":62,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","created":"2024-09-14T03:36:30.445371+0000","modified":"2024-09-14T03:43:12.274330+0000","last_up_change":"2024-09-14T03:41:23.437053+0000","last_in_change":"2024-09-14T03:41:06.908282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T03:39:29.348427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T03:41:54.688727+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-14T03:43:09.784984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-14T03:43:10.496413+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"0a021a3a-45d2-4825-a701-d7c3793ef6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6803","nonce":2918802523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6805","nonce":2918802523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6809","nonce":2918802523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6807","nonce":2918802523}]},"public_addr":"172.21.15.114:6803/2918802523","cluster_addr":"172.21.15.114:6805/2918802523","heartbeat_back_addr":"172.21.15.114:6809/2918802523","heartbeat_front_addr":"172.21.15.114:6807/2918802523","state":["exists","up"]},{"osd":1,"uuid":"076291c4-5927-456d-bf74-2ab79be4c756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6811","nonce":1989151589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6813","nonce":1989151589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6817","nonce":1989151589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6815","nonce":1989151589}]},"public_addr":"172.21.15.114:6811/1989151589","cluster_addr":"172.21.15.114:6813/1989151589","heartbeat_back_addr":"172.21.15.114:6817/1989151589","heartbeat_front_addr":"172.21.15.114:6815/1989151589","state":["exists","up"]},{"osd":2,"uuid":"d6787400-73ea-4edf-bb17-440ac36bd4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6819","nonce":2560140870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6821","nonce":2560140870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6825","nonce":2560140870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6823","nonce":2560140870}]},"public_addr":"172.21.15.114:6819/2560140870","cluster_addr":"172.21.15.114:6821/2560140870","heartbeat_back_addr":"172.21.15.114:6825/2560140870","heartbeat_front_addr":"172.21.15.114:6823/2560140870","state":["exists","up"]},{"osd":3,"uuid":"ead755f3-173a-4465-a28f-af7288b0a42b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6827","nonce":3048125105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6829","nonce":3048125105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6833","nonce":3048125105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6831","nonce":3048125105}]},"public_addr":"172.21.15.114:6827/3048125105","cluster_addr":"172.21.15.114:6829/3048125105","heartbeat_back_addr":"172.21.15.114:6833/3048125105","heartbeat_front_addr":"172.21.15.114:6831/3048125105","state":["exists","up"]},{"osd":4,"uuid":"60a0aac2-e235-484f-b25f-f112c83da7f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6801","nonce":825361940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6803","nonce":825361940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6807","nonce":825361940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6805","nonce":825361940}]},"public_addr":"172.21.15.160:6801/825361940","cluster_addr":"172.21.15.160:6803/825361940","heartbeat_back_addr":"172.21.15.160:6807/825361940","heartbeat_front_addr":"172.21.15.160:6805/825361940","state":["exists","up"]},{"osd":5,"uuid":"8f49c7de-c810-4075-b731-022bb5031d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6809","nonce":3752160003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6811","nonce":3752160003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6815","nonce":3752160003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6813","nonce":3752160003}]},"public_addr":"172.21.15.160:6809/3752160003","cluster_addr":"172.21.15.160:6811/3752160003","heartbeat_back_addr":"172.21.15.160:6815/3752160003","heartbeat_front_addr":"172.21.15.160:6813/3752160003","state":["exists","up"]},{"osd":6,"uuid":"455de335-7290-4242-b743-6b0c446683ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716932982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716932982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716932982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716932982}]},"public_addr":"172.21.15.160:6817/3716932982","cluster_addr":"172.21.15.160:6819/3716932982","heartbeat_back_addr":"172.21.15.160:6823/3716932982","heartbeat_front_addr":"172.21.15.160:6821/3716932982","state":["exists","up"]},{"osd":7,"uuid":"7d8b5297-b1db-41b1-be35-073e7780128e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6825","nonce":1323253841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6827","nonce":1323253841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6831","nonce":1323253841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6829","nonce":1323253841}]},"public_addr":"172.21.15.160:6825/1323253841","cluster_addr":"172.21.15.160:6827/1323253841","heartbeat_back_addr":"172.21.15.160:6831/1323253841","heartbeat_front_addr":"172.21.15.160:6829/1323253841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:38:38.363862+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:02.363463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:26.353194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:50.132880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:11.402862+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:34.539123+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:58.191287+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:41:22.159510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/767494479":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6800/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1784685209":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6801/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6801/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/426398557":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/2694023561":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1089490408":"2024-09-15T03:36:58.525006+0000","172.21.15.114:6801/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2536880143":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2067882043":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2155247703":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2095546075":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/3344792350":"2024-09-15T03:41:51.162793+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T03:43:25.787 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:25 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/157104638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:43:25.787 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:25 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:25.787 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:25 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:25.788 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:25 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/157104638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:43:25.788 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:25 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:25.788 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:25 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:25.814 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-14T03:43:25.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-14T03:43:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:25 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/157104638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:43:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:25 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:25 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:26.565 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:26.688 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:26 smithi114 ceph-mon[40961]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:26.688 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:26 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:26.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:26 smithi114 ceph-mon[40961]: mds.? [v2:172.21.15.160:6832/2654584478,v1:172.21.15.160:6833/2654584478] up:standby 2024-09-14T03:43:26.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:26 smithi114 ceph-mon[40961]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:26.689 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:26 smithi114 ceph-mon[29589]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:26.689 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:26 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:26.689 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:26 smithi114 ceph-mon[29589]: mds.? [v2:172.21.15.160:6832/2654584478,v1:172.21.15.160:6833/2654584478] up:standby 2024-09-14T03:43:26.689 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:26 smithi114 ceph-mon[29589]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:26 smithi160 ceph-mon[32702]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:26 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:26 smithi160 ceph-mon[32702]: mds.? [v2:172.21.15.160:6832/2654584478,v1:172.21.15.160:6833/2654584478] up:standby 2024-09-14T03:43:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:26 smithi160 ceph-mon[32702]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:43:28.825 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[40961]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:28.825 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:28.825 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[29589]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:28.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:28 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:43:28.866 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:28 smithi160 ceph-mon[32702]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:28.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:28 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:28.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:28 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:28.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:28 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:28.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:28 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:43:28.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:28 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:28.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:28 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:43:29.181 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:43:29.181 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":62,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","created":"2024-09-14T03:36:30.445371+0000","modified":"2024-09-14T03:43:12.274330+0000","last_up_change":"2024-09-14T03:41:23.437053+0000","last_in_change":"2024-09-14T03:41:06.908282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T03:39:29.348427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T03:41:54.688727+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-14T03:43:09.784984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-14T03:43:10.496413+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"0a021a3a-45d2-4825-a701-d7c3793ef6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6803","nonce":2918802523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6805","nonce":2918802523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6809","nonce":2918802523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2918802523},{"type":"v1","addr":"172.21.15.114:6807","nonce":2918802523}]},"public_addr":"172.21.15.114:6803/2918802523","cluster_addr":"172.21.15.114:6805/2918802523","heartbeat_back_addr":"172.21.15.114:6809/2918802523","heartbeat_front_addr":"172.21.15.114:6807/2918802523","state":["exists","up"]},{"osd":1,"uuid":"076291c4-5927-456d-bf74-2ab79be4c756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6811","nonce":1989151589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6813","nonce":1989151589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6817","nonce":1989151589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":1989151589},{"type":"v1","addr":"172.21.15.114:6815","nonce":1989151589}]},"public_addr":"172.21.15.114:6811/1989151589","cluster_addr":"172.21.15.114:6813/1989151589","heartbeat_back_addr":"172.21.15.114:6817/1989151589","heartbeat_front_addr":"172.21.15.114:6815/1989151589","state":["exists","up"]},{"osd":2,"uuid":"d6787400-73ea-4edf-bb17-440ac36bd4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6819","nonce":2560140870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6821","nonce":2560140870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6825","nonce":2560140870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2560140870},{"type":"v1","addr":"172.21.15.114:6823","nonce":2560140870}]},"public_addr":"172.21.15.114:6819/2560140870","cluster_addr":"172.21.15.114:6821/2560140870","heartbeat_back_addr":"172.21.15.114:6825/2560140870","heartbeat_front_addr":"172.21.15.114:6823/2560140870","state":["exists","up"]},{"osd":3,"uuid":"ead755f3-173a-4465-a28f-af7288b0a42b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6827","nonce":3048125105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6829","nonce":3048125105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6833","nonce":3048125105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3048125105},{"type":"v1","addr":"172.21.15.114:6831","nonce":3048125105}]},"public_addr":"172.21.15.114:6827/3048125105","cluster_addr":"172.21.15.114:6829/3048125105","heartbeat_back_addr":"172.21.15.114:6833/3048125105","heartbeat_front_addr":"172.21.15.114:6831/3048125105","state":["exists","up"]},{"osd":4,"uuid":"60a0aac2-e235-484f-b25f-f112c83da7f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6801","nonce":825361940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6803","nonce":825361940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6807","nonce":825361940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6805","nonce":825361940}]},"public_addr":"172.21.15.160:6801/825361940","cluster_addr":"172.21.15.160:6803/825361940","heartbeat_back_addr":"172.21.15.160:6807/825361940","heartbeat_front_addr":"172.21.15.160:6805/825361940","state":["exists","up"]},{"osd":5,"uuid":"8f49c7de-c810-4075-b731-022bb5031d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6809","nonce":3752160003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6811","nonce":3752160003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6815","nonce":3752160003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6813","nonce":3752160003}]},"public_addr":"172.21.15.160:6809/3752160003","cluster_addr":"172.21.15.160:6811/3752160003","heartbeat_back_addr":"172.21.15.160:6815/3752160003","heartbeat_front_addr":"172.21.15.160:6813/3752160003","state":["exists","up"]},{"osd":6,"uuid":"455de335-7290-4242-b743-6b0c446683ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716932982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716932982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716932982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716932982}]},"public_addr":"172.21.15.160:6817/3716932982","cluster_addr":"172.21.15.160:6819/3716932982","heartbeat_back_addr":"172.21.15.160:6823/3716932982","heartbeat_front_addr":"172.21.15.160:6821/3716932982","state":["exists","up"]},{"osd":7,"uuid":"7d8b5297-b1db-41b1-be35-073e7780128e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6825","nonce":1323253841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6827","nonce":1323253841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6831","nonce":1323253841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6829","nonce":1323253841}]},"public_addr":"172.21.15.160:6825/1323253841","cluster_addr":"172.21.15.160:6827/1323253841","heartbeat_back_addr":"172.21.15.160:6831/1323253841","heartbeat_front_addr":"172.21.15.160:6829/1323253841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:38:38.363862+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:02.363463+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:26.353194+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:50.132880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:11.402862+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:34.539123+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:58.191287+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:41:22.159510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/767494479":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6800/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1784685209":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6801/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6801/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/426398557":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/2694023561":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1089490408":"2024-09-15T03:36:58.525006+0000","172.21.15.114:6801/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2536880143":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2067882043":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2155247703":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2095546075":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/3344792350":"2024-09-15T03:41:51.162793+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T03:43:29.618 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:43:29] "GET /metrics HTTP/1.1" 200 239125 "" "Prometheus/2.43.0" 2024-09-14T03:43:29.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:29 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/99590374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:43:29.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:29 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/99590374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:43:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:29 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/99590374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-14T03:43:30.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-14T03:43:30.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-14T03:43:30.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-14T03:43:30.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-14T03:43:30.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-14T03:43:30.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-14T03:43:30.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-14T03:43:30.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-14T03:43:30.709 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:30 smithi160 ceph-mon[32702]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:30.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:30 smithi114 ceph-mon[40961]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:30.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:30 smithi114 ceph-mon[29589]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:30.958 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:30.978 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:30.984 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:31.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:43:31 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:43:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:43:32.739 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:32.740 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:32.741 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:32.741 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:33.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:32 smithi114 ceph-mon[29589]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:33.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:32 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:33.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:32 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:33.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:32 smithi114 ceph-mon[40961]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:33.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:32 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:33.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:32 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:32 smithi160 ceph-mon[32702]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:32 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:32 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:33.428 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:34.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:34 smithi114 ceph-mon[29589]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:34.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:34 smithi114 ceph-mon[40961]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:34 smithi160 ceph-mon[32702]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-14T03:43:37.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:36 smithi114 ceph-mon[40961]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-14T03:43:37.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:36 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:37.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:36 smithi114 ceph-mon[29589]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-14T03:43:37.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:36 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:36 smithi160 ceph-mon[32702]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-14T03:43:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:36 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:38.245 INFO:teuthology.orchestra.run.smithi114.stdout:77309411382 2024-09-14T03:43:38.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-14T03:43:38.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:38 smithi114 ceph-mon[29589]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-14T03:43:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:38 smithi114 ceph-mon[40961]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-14T03:43:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:38 smithi160 ceph-mon[32702]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-14T03:43:39.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:43:39] "GET /metrics HTTP/1.1" 200 239640 "" "Prometheus/2.43.0" 2024-09-14T03:43:40.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:40 smithi114 ceph-mon[29589]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:40.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:40 smithi114 ceph-mon[40961]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:40 smithi160 ceph-mon[32702]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:41.781 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:43:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:43:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:43:42.785 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:42 smithi114 ceph-mon[40961]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:42.785 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:42 smithi114 ceph-mon[29589]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:42 smithi160 ceph-mon[32702]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:43.857 INFO:teuthology.orchestra.run.smithi114.stdout:154618822696 2024-09-14T03:43:43.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-14T03:43:45.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:44 smithi114 ceph-mon[29589]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:45.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:44 smithi114 ceph-mon[40961]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:44 smithi160 ceph-mon[32702]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:46.376 INFO:teuthology.orchestra.run.smithi114.stdout:55834574907 2024-09-14T03:43:46.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-14T03:43:46.751 INFO:teuthology.orchestra.run.smithi114.stdout:34359738432 2024-09-14T03:43:46.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-14T03:43:47.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:46 smithi114 ceph-mon[40961]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:47.014 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:46 smithi114 ceph-mon[29589]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:46 smithi160 ceph-mon[32702]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:47.758 INFO:teuthology.orchestra.run.smithi114.stdout:206158430239 2024-09-14T03:43:47.758 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-14T03:43:48.511 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:48.847 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:48 smithi114 ceph-mon[40961]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:48.848 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:48 smithi114 ceph-mon[29589]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:48.849 INFO:teuthology.orchestra.run.smithi114.stdout:128849018924 2024-09-14T03:43:48.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-14T03:43:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:48 smithi160 ceph-mon[32702]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:49.344 INFO:teuthology.orchestra.run.smithi114.stdout:180388626469 2024-09-14T03:43:49.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-14T03:43:49.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:43:49] "GET /metrics HTTP/1.1" 200 239640 "" "Prometheus/2.43.0" 2024-09-14T03:43:49.753 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:49.753 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:49.754 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:49 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:49 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:49 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:49 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:43:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:49 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:49 smithi160 ceph-mon[32702]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:43:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[29589]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:50.168 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:50.168 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:50.168 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:43:50.168 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:43:50.169 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:43:50.169 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:49 smithi114 ceph-mon[40961]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:50.193 INFO:teuthology.orchestra.run.smithi114.stdout:107374182450 2024-09-14T03:43:50.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-14T03:43:50.206 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:50.237 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:51.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-14T03:43:51.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-14T03:43:51.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:51.530 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.530 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-14T03:43:51.531 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.531 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.531 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-14T03:43:51.531 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.531 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:51.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:43:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:43:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:43:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-14T03:43:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:51 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:51 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-09-14T03:43:51.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:51 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-14T03:43:51.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:43:52.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:52 smithi114 ceph-mon[29589]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:52.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:52 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-14T03:43:52.450 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:52 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-09-14T03:43:52.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:52 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-14T03:43:52.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:52 smithi114 ceph-mon[29589]: osdmap e63: 8 total, 8 up, 8 in 2024-09-14T03:43:52.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:52 smithi114 ceph-mon[40961]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:52.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:52 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-14T03:43:52.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:52 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-09-14T03:43:52.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:52 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-14T03:43:52.451 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:52 smithi114 ceph-mon[40961]: osdmap e63: 8 total, 8 up, 8 in 2024-09-14T03:43:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:52 smithi160 ceph-mon[32702]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:52 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-14T03:43:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:52 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-09-14T03:43:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:52 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-14T03:43:52.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:52 smithi160 ceph-mon[32702]: osdmap e63: 8 total, 8 up, 8 in 2024-09-14T03:43:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:53 smithi114 ceph-mon[29589]: osdmap e64: 8 total, 8 up, 8 in 2024-09-14T03:43:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:53 smithi114 ceph-mon[40961]: osdmap e64: 8 total, 8 up, 8 in 2024-09-14T03:43:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:53 smithi160 ceph-mon[32702]: osdmap e64: 8 total, 8 up, 8 in 2024-09-14T03:43:54.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:54 smithi114 ceph-mon[29589]: pgmap v77: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:54.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:54 smithi114 ceph-mon[40961]: pgmap v77: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:54 smithi160 ceph-mon[32702]: pgmap v77: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:55.999 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:43:56.573 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:56 smithi114 ceph-mon[29589]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:56.573 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:56 smithi114 ceph-mon[40961]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:56 smithi160 ceph-mon[32702]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:58.297 INFO:teuthology.orchestra.run.smithi114.stdout:77309411386 2024-09-14T03:43:58.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:58 smithi114 ceph-mon[29589]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:58.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:58 smithi114 ceph-mon[40961]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:58 smithi160 ceph-mon[32702]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:43:59.082 INFO:teuthology.orchestra.run.smithi114.stdout:55834574911 2024-09-14T03:43:59.422 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411382 got 77309411386 for osd.2 2024-09-14T03:43:59.422 DEBUG:teuthology.parallel:result is None 2024-09-14T03:43:59.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:43:59 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:43:59] "GET /metrics HTTP/1.1" 200 239614 "" "Prometheus/2.43.0" 2024-09-14T03:43:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:59 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3068865559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-14T03:43:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:43:59 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/4193306826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-14T03:43:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:59 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3068865559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-14T03:43:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:43:59 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/4193306826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-14T03:43:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:59 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3068865559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-14T03:43:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:43:59 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/4193306826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-14T03:44:00.152 INFO:teuthology.orchestra.run.smithi114.stdout:180388626471 2024-09-14T03:44:00.183 INFO:teuthology.orchestra.run.smithi114.stdout:206158430242 2024-09-14T03:44:00.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:00 smithi114 ceph-mon[29589]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:00.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:00 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2112407149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-14T03:44:00.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:00 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2361480595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-14T03:44:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:00 smithi114 ceph-mon[40961]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:00 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2112407149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-14T03:44:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:00 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2361480595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-14T03:44:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:00 smithi160 ceph-mon[32702]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:00.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:00 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2112407149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-14T03:44:00.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:00 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2361480595' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-14T03:44:00.936 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626469 got 180388626471 for osd.6 2024-09-14T03:44:00.937 DEBUG:teuthology.parallel:result is None 2024-09-14T03:44:01.436 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430239 got 206158430242 for osd.7 2024-09-14T03:44:01.436 DEBUG:teuthology.parallel:result is None 2024-09-14T03:44:01.779 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:44:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:44:01.912 INFO:teuthology.orchestra.run.smithi114.stdout:154618822699 2024-09-14T03:44:01.957 INFO:teuthology.orchestra.run.smithi114.stdout:34359738435 2024-09-14T03:44:02.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:02 smithi114 ceph-mon[29589]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:02.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:02 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/4041176592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-14T03:44:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:02 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1915792362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-14T03:44:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:02 smithi114 ceph-mon[40961]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:02 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/4041176592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-14T03:44:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:02 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1915792362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-14T03:44:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:02 smithi160 ceph-mon[32702]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:02 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/4041176592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-14T03:44:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:02 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1915792362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-14T03:44:03.262 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574907 got 55834574911 for osd.1 2024-09-14T03:44:03.262 DEBUG:teuthology.parallel:result is None 2024-09-14T03:44:03.514 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:03.952 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738432 got 34359738435 for osd.0 2024-09-14T03:44:03.953 DEBUG:teuthology.parallel:result is None 2024-09-14T03:44:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:04 smithi114 ceph-mon[29589]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:04.440 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:04 smithi114 ceph-mon[40961]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:04 smithi160 ceph-mon[32702]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:04.787 INFO:teuthology.orchestra.run.smithi114.stdout:128849018928 2024-09-14T03:44:05.191 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822696 got 154618822699 for osd.5 2024-09-14T03:44:05.191 DEBUG:teuthology.parallel:result is None 2024-09-14T03:44:05.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:05 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1153182909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-14T03:44:05.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:05 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1153182909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-14T03:44:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:05 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1153182909' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-14T03:44:06.261 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018924 got 128849018928 for osd.4 2024-09-14T03:44:06.261 DEBUG:teuthology.parallel:result is None 2024-09-14T03:44:06.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:06 smithi114 ceph-mon[40961]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:06.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:06 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:06 smithi114 ceph-mon[29589]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:06 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:06 smithi160 ceph-mon[32702]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:06 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:07.358 INFO:teuthology.orchestra.run.smithi114.stdout:107374182454 2024-09-14T03:44:07.967 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182450 got 107374182454 for osd.3 2024-09-14T03:44:07.968 DEBUG:teuthology.parallel:result is None 2024-09-14T03:44:07.968 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-14T03:44:07.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T03:44:08.216 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:08.576 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:08 smithi114 ceph-mon[40961]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:08.576 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:08 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3173416802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-14T03:44:08.576 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:08 smithi114 ceph-mon[29589]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:08.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:08 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3173416802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-14T03:44:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:08 smithi160 ceph-mon[32702]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:08 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3173416802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-14T03:44:09.659 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:09 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:09] "GET /metrics HTTP/1.1" 200 239652 "" "Prometheus/2.43.0" 2024-09-14T03:44:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:44:10.048 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T03:44:10.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:10 smithi114 ceph-mon[29589]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:10 smithi114 ceph-mon[40961]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:10 smithi160 ceph-mon[32702]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:10.720 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-14T03:44:09.198589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385764,"kb_used_data":6912,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575452,"statfs":{"total":767960285184,"available":765517262848,"internally_reserved":0,"allocated":7077888,"data_stored":4070557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002924"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239626+0000","last_change":"2024-09-14T03:43:11.909883+0000","last_active":"2024-09-14T03:43:51.239626+0000","last_peered":"2024-09-14T03:43:51.239626+0000","last_clean":"2024-09-14T03:43:51.239626+0000","last_became_active":"2024-09-14T03:43:11.909587+0000","last_became_peered":"2024-09-14T03:43:11.909587+0000","last_unstale":"2024-09-14T03:43:51.239626+0000","last_undegraded":"2024-09-14T03:43:51.239626+0000","last_fullsized":"2024-09-14T03:43:51.239626+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:30:18.120038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244211+0000","last_change":"2024-09-14T03:43:11.270513+0000","last_active":"2024-09-14T03:43:51.244211+0000","last_peered":"2024-09-14T03:43:51.244211+0000","last_clean":"2024-09-14T03:43:51.244211+0000","last_became_active":"2024-09-14T03:43:11.269672+0000","last_became_peered":"2024-09-14T03:43:11.269672+0000","last_unstale":"2024-09-14T03:43:51.244211+0000","last_undegraded":"2024-09-14T03:43:51.244211+0000","last_fullsized":"2024-09-14T03:43:51.244211+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:56.227349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241595+0000","last_change":"2024-09-14T03:43:11.910241+0000","last_active":"2024-09-14T03:43:51.241595+0000","last_peered":"2024-09-14T03:43:51.241595+0000","last_clean":"2024-09-14T03:43:51.241595+0000","last_became_active":"2024-09-14T03:43:11.909971+0000","last_became_peered":"2024-09-14T03:43:11.909971+0000","last_unstale":"2024-09-14T03:43:51.241595+0000","last_undegraded":"2024-09-14T03:43:51.241595+0000","last_fullsized":"2024-09-14T03:43:51.241595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:01:58.042085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313658+0000","last_change":"2024-09-14T03:43:11.272567+0000","last_active":"2024-09-14T03:43:51.313658+0000","last_peered":"2024-09-14T03:43:51.313658+0000","last_clean":"2024-09-14T03:43:51.313658+0000","last_became_active":"2024-09-14T03:43:11.272293+0000","last_became_peered":"2024-09-14T03:43:11.272293+0000","last_unstale":"2024-09-14T03:43:51.313658+0000","last_undegraded":"2024-09-14T03:43:51.313658+0000","last_fullsized":"2024-09-14T03:43:51.313658+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:43:39.141576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.314389+0000","last_change":"2024-09-14T03:43:11.910637+0000","last_active":"2024-09-14T03:43:51.314389+0000","last_peered":"2024-09-14T03:43:51.314389+0000","last_clean":"2024-09-14T03:43:51.314389+0000","last_became_active":"2024-09-14T03:43:11.910299+0000","last_became_peered":"2024-09-14T03:43:11.910299+0000","last_unstale":"2024-09-14T03:43:51.314389+0000","last_undegraded":"2024-09-14T03:43:51.314389+0000","last_fullsized":"2024-09-14T03:43:51.314389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:23:48.019457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244186+0000","last_change":"2024-09-14T03:43:11.269452+0000","last_active":"2024-09-14T03:43:51.244186+0000","last_peered":"2024-09-14T03:43:51.244186+0000","last_clean":"2024-09-14T03:43:51.244186+0000","last_became_active":"2024-09-14T03:43:11.269185+0000","last_became_peered":"2024-09-14T03:43:11.269185+0000","last_unstale":"2024-09-14T03:43:51.244186+0000","last_undegraded":"2024-09-14T03:43:51.244186+0000","last_fullsized":"2024-09-14T03:43:51.244186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:53:12.602272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239159+0000","last_change":"2024-09-14T03:43:11.909747+0000","last_active":"2024-09-14T03:43:51.239159+0000","last_peered":"2024-09-14T03:43:51.239159+0000","last_clean":"2024-09-14T03:43:51.239159+0000","last_became_active":"2024-09-14T03:43:11.909446+0000","last_became_peered":"2024-09-14T03:43:11.909446+0000","last_unstale":"2024-09-14T03:43:51.239159+0000","last_undegraded":"2024-09-14T03:43:51.239159+0000","last_fullsized":"2024-09-14T03:43:51.239159+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:18:01.366571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":21,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241072+0000","last_change":"2024-09-14T03:43:11.269616+0000","last_active":"2024-09-14T03:43:51.241072+0000","last_peered":"2024-09-14T03:43:51.241072+0000","last_clean":"2024-09-14T03:43:51.241072+0000","last_became_active":"2024-09-14T03:43:11.269359+0000","last_became_peered":"2024-09-14T03:43:11.269359+0000","last_unstale":"2024-09-14T03:43:51.241072+0000","last_undegraded":"2024-09-14T03:43:51.241072+0000","last_fullsized":"2024-09-14T03:43:51.241072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:33:15.404348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244945+0000","last_change":"2024-09-14T03:43:11.273602+0000","last_active":"2024-09-14T03:43:51.244945+0000","last_peered":"2024-09-14T03:43:51.244945+0000","last_clean":"2024-09-14T03:43:51.244945+0000","last_became_active":"2024-09-14T03:43:11.273307+0000","last_became_peered":"2024-09-14T03:43:11.273307+0000","last_unstale":"2024-09-14T03:43:51.244945+0000","last_undegraded":"2024-09-14T03:43:51.244945+0000","last_fullsized":"2024-09-14T03:43:51.244945+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:54:07.949045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241058+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T03:43:51.241058+0000","last_peered":"2024-09-14T03:43:51.241058+0000","last_clean":"2024-09-14T03:43:51.241058+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T03:43:51.241058+0000","last_undegraded":"2024-09-14T03:43:51.241058+0000","last_fullsized":"2024-09-14T03:43:51.241058+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244552+0000","last_change":"2024-09-14T03:43:11.273656+0000","last_active":"2024-09-14T03:43:51.244552+0000","last_peered":"2024-09-14T03:43:51.244552+0000","last_clean":"2024-09-14T03:43:51.244552+0000","last_became_active":"2024-09-14T03:43:11.273397+0000","last_became_peered":"2024-09-14T03:43:11.273397+0000","last_unstale":"2024-09-14T03:43:51.244552+0000","last_undegraded":"2024-09-14T03:43:51.244552+0000","last_fullsized":"2024-09-14T03:43:51.244552+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:54.630120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238313+0000","last_change":"2024-09-14T03:43:11.295859+0000","last_active":"2024-09-14T03:43:51.238313+0000","last_peered":"2024-09-14T03:43:51.238313+0000","last_clean":"2024-09-14T03:43:51.238313+0000","last_became_active":"2024-09-14T03:43:11.295431+0000","last_became_peered":"2024-09-14T03:43:11.295431+0000","last_unstale":"2024-09-14T03:43:51.238313+0000","last_undegraded":"2024-09-14T03:43:51.238313+0000","last_fullsized":"2024-09-14T03:43:51.238313+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:44:22.965505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244502+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T03:43:51.244502+0000","last_peered":"2024-09-14T03:43:51.244502+0000","last_clean":"2024-09-14T03:43:51.244502+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T03:43:51.244502+0000","last_undegraded":"2024-09-14T03:43:51.244502+0000","last_fullsized":"2024-09-14T03:43:51.244502+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-14T03:43:52.652451+0000","last_change":"2024-09-14T03:43:52.652451+0000","last_active":"2024-09-14T03:43:52.652451+0000","last_peered":"2024-09-14T03:43:52.652451+0000","last_clean":"2024-09-14T03:43:52.652451+0000","last_became_active":"2024-09-14T03:43:52.652081+0000","last_became_peered":"2024-09-14T03:43:52.652081+0000","last_unstale":"2024-09-14T03:43:52.652451+0000","last_undegraded":"2024-09-14T03:43:52.652451+0000","last_fullsized":"2024-09-14T03:43:52.652451+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:24:49.511684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241862+0000","last_change":"2024-09-14T03:43:11.268385+0000","last_active":"2024-09-14T03:43:51.241862+0000","last_peered":"2024-09-14T03:43:51.241862+0000","last_clean":"2024-09-14T03:43:51.241862+0000","last_became_active":"2024-09-14T03:43:11.268099+0000","last_became_peered":"2024-09-14T03:43:11.268099+0000","last_unstale":"2024-09-14T03:43:51.241862+0000","last_undegraded":"2024-09-14T03:43:51.241862+0000","last_fullsized":"2024-09-14T03:43:51.241862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:02:15.398718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313912+0000","last_change":"2024-09-14T03:43:11.290115+0000","last_active":"2024-09-14T03:43:51.313912+0000","last_peered":"2024-09-14T03:43:51.313912+0000","last_clean":"2024-09-14T03:43:51.313912+0000","last_became_active":"2024-09-14T03:43:11.289914+0000","last_became_peered":"2024-09-14T03:43:11.289914+0000","last_unstale":"2024-09-14T03:43:51.313912+0000","last_undegraded":"2024-09-14T03:43:51.313912+0000","last_fullsized":"2024-09-14T03:43:51.313912+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:55:50.945018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239241+0000","last_change":"2024-09-14T03:43:11.268760+0000","last_active":"2024-09-14T03:43:51.239241+0000","last_peered":"2024-09-14T03:43:51.239241+0000","last_clean":"2024-09-14T03:43:51.239241+0000","last_became_active":"2024-09-14T03:43:11.268339+0000","last_became_peered":"2024-09-14T03:43:11.268339+0000","last_unstale":"2024-09-14T03:43:51.239241+0000","last_undegraded":"2024-09-14T03:43:51.239241+0000","last_fullsized":"2024-09-14T03:43:51.239241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:13.840894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238588+0000","last_change":"2024-09-14T03:43:11.295695+0000","last_active":"2024-09-14T03:43:51.238588+0000","last_peered":"2024-09-14T03:43:51.238588+0000","last_clean":"2024-09-14T03:43:51.238588+0000","last_became_active":"2024-09-14T03:43:11.295441+0000","last_became_peered":"2024-09-14T03:43:11.295441+0000","last_unstale":"2024-09-14T03:43:51.238588+0000","last_undegraded":"2024-09-14T03:43:51.238588+0000","last_fullsized":"2024-09-14T03:43:51.238588+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:12.038600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.242274+0000","last_change":"2024-09-14T03:43:11.274148+0000","last_active":"2024-09-14T03:43:51.242274+0000","last_peered":"2024-09-14T03:43:51.242274+0000","last_clean":"2024-09-14T03:43:51.242274+0000","last_became_active":"2024-09-14T03:43:11.273754+0000","last_became_peered":"2024-09-14T03:43:11.273754+0000","last_unstale":"2024-09-14T03:43:51.242274+0000","last_undegraded":"2024-09-14T03:43:51.242274+0000","last_fullsized":"2024-09-14T03:43:51.242274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:16:08.201934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238749+0000","last_change":"2024-09-14T03:43:11.292860+0000","last_active":"2024-09-14T03:43:51.238749+0000","last_peered":"2024-09-14T03:43:51.238749+0000","last_clean":"2024-09-14T03:43:51.238749+0000","last_became_active":"2024-09-14T03:43:11.292629+0000","last_became_peered":"2024-09-14T03:43:11.292629+0000","last_unstale":"2024-09-14T03:43:51.238749+0000","last_undegraded":"2024-09-14T03:43:51.238749+0000","last_fullsized":"2024-09-14T03:43:51.238749+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:24:35.112097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238730+0000","last_change":"2024-09-14T03:43:11.268184+0000","last_active":"2024-09-14T03:43:51.238730+0000","last_peered":"2024-09-14T03:43:51.238730+0000","last_clean":"2024-09-14T03:43:51.238730+0000","last_became_active":"2024-09-14T03:43:11.267957+0000","last_became_peered":"2024-09-14T03:43:11.267957+0000","last_unstale":"2024-09-14T03:43:51.238730+0000","last_undegraded":"2024-09-14T03:43:51.238730+0000","last_fullsized":"2024-09-14T03:43:51.238730+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:23:59.632674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238787+0000","last_change":"2024-09-14T03:43:11.286604+0000","last_active":"2024-09-14T03:43:51.238787+0000","last_peered":"2024-09-14T03:43:51.238787+0000","last_clean":"2024-09-14T03:43:51.238787+0000","last_became_active":"2024-09-14T03:43:11.286329+0000","last_became_peered":"2024-09-14T03:43:11.286329+0000","last_unstale":"2024-09-14T03:43:51.238787+0000","last_undegraded":"2024-09-14T03:43:51.238787+0000","last_fullsized":"2024-09-14T03:43:51.238787+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:54:15.941054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238708+0000","last_change":"2024-09-14T03:43:11.275058+0000","last_active":"2024-09-14T03:43:51.238708+0000","last_peered":"2024-09-14T03:43:51.238708+0000","last_clean":"2024-09-14T03:43:51.238708+0000","last_became_active":"2024-09-14T03:43:11.273704+0000","last_became_peered":"2024-09-14T03:43:11.273704+0000","last_unstale":"2024-09-14T03:43:51.238708+0000","last_undegraded":"2024-09-14T03:43:51.238708+0000","last_fullsized":"2024-09-14T03:43:51.238708+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:58:59.923574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240503+0000","last_change":"2024-09-14T03:43:11.293157+0000","last_active":"2024-09-14T03:43:51.240503+0000","last_peered":"2024-09-14T03:43:51.240503+0000","last_clean":"2024-09-14T03:43:51.240503+0000","last_became_active":"2024-09-14T03:43:11.291638+0000","last_became_peered":"2024-09-14T03:43:11.291638+0000","last_unstale":"2024-09-14T03:43:51.240503+0000","last_undegraded":"2024-09-14T03:43:51.240503+0000","last_fullsized":"2024-09-14T03:43:51.240503+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:28:52.185952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"51'79","reported_seq":55,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244102+0000","last_change":"2024-09-14T03:41:25.453993+0000","last_active":"2024-09-14T03:43:51.244102+0000","last_peered":"2024-09-14T03:43:51.244102+0000","last_clean":"2024-09-14T03:43:51.244102+0000","last_became_active":"2024-09-14T03:41:25.446786+0000","last_became_peered":"2024-09-14T03:41:25.446786+0000","last_unstale":"2024-09-14T03:43:51.244102+0000","last_undegraded":"2024-09-14T03:43:51.244102+0000","last_fullsized":"2024-09-14T03:43:51.244102+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:50:17.999993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":26,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241721+0000","last_change":"2024-09-14T03:42:01.465792+0000","last_active":"2024-09-14T03:43:51.241721+0000","last_peered":"2024-09-14T03:43:51.241721+0000","last_clean":"2024-09-14T03:43:51.241721+0000","last_became_active":"2024-09-14T03:41:56.433826+0000","last_became_peered":"2024-09-14T03:41:56.433826+0000","last_unstale":"2024-09-14T03:43:51.241721+0000","last_undegraded":"2024-09-14T03:43:51.241721+0000","last_fullsized":"2024-09-14T03:43:51.241721+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:45:32.233869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239148+0000","last_change":"2024-09-14T03:43:11.269492+0000","last_active":"2024-09-14T03:43:51.239148+0000","last_peered":"2024-09-14T03:43:51.239148+0000","last_clean":"2024-09-14T03:43:51.239148+0000","last_became_active":"2024-09-14T03:43:11.269228+0000","last_became_peered":"2024-09-14T03:43:11.269228+0000","last_unstale":"2024-09-14T03:43:51.239148+0000","last_undegraded":"2024-09-14T03:43:51.239148+0000","last_fullsized":"2024-09-14T03:43:51.239148+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:23:01.054451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241355+0000","last_change":"2024-09-14T03:43:11.909945+0000","last_active":"2024-09-14T03:43:51.241355+0000","last_peered":"2024-09-14T03:43:51.241355+0000","last_clean":"2024-09-14T03:43:51.241355+0000","last_became_active":"2024-09-14T03:43:11.909666+0000","last_became_peered":"2024-09-14T03:43:11.909666+0000","last_unstale":"2024-09-14T03:43:51.241355+0000","last_undegraded":"2024-09-14T03:43:51.241355+0000","last_fullsized":"2024-09-14T03:43:51.241355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:13:19.629641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.243976+0000","last_change":"2024-09-14T03:42:02.106608+0000","last_active":"2024-09-14T03:43:51.243976+0000","last_peered":"2024-09-14T03:43:51.243976+0000","last_clean":"2024-09-14T03:43:51.243976+0000","last_became_active":"2024-09-14T03:41:56.814765+0000","last_became_peered":"2024-09-14T03:41:56.814765+0000","last_unstale":"2024-09-14T03:43:51.243976+0000","last_undegraded":"2024-09-14T03:43:51.243976+0000","last_fullsized":"2024-09-14T03:43:51.243976+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:15:23.921629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313693+0000","last_change":"2024-09-14T03:43:11.272354+0000","last_active":"2024-09-14T03:43:51.313693+0000","last_peered":"2024-09-14T03:43:51.313693+0000","last_clean":"2024-09-14T03:43:51.313693+0000","last_became_active":"2024-09-14T03:43:11.272092+0000","last_became_peered":"2024-09-14T03:43:11.272092+0000","last_unstale":"2024-09-14T03:43:51.313693+0000","last_undegraded":"2024-09-14T03:43:51.313693+0000","last_fullsized":"2024-09-14T03:43:51.313693+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:08.709427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313949+0000","last_change":"2024-09-14T03:43:11.289253+0000","last_active":"2024-09-14T03:43:51.313949+0000","last_peered":"2024-09-14T03:43:51.313949+0000","last_clean":"2024-09-14T03:43:51.313949+0000","last_became_active":"2024-09-14T03:43:11.288840+0000","last_became_peered":"2024-09-14T03:43:11.288840+0000","last_unstale":"2024-09-14T03:43:51.313949+0000","last_undegraded":"2024-09-14T03:43:51.313949+0000","last_fullsized":"2024-09-14T03:43:51.313949+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:48:27.761229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.243540+0000","last_change":"2024-09-14T03:42:01.461806+0000","last_active":"2024-09-14T03:43:51.243540+0000","last_peered":"2024-09-14T03:43:51.243540+0000","last_clean":"2024-09-14T03:43:51.243540+0000","last_became_active":"2024-09-14T03:41:56.814735+0000","last_became_peered":"2024-09-14T03:41:56.814735+0000","last_unstale":"2024-09-14T03:43:51.243540+0000","last_undegraded":"2024-09-14T03:43:51.243540+0000","last_fullsized":"2024-09-14T03:43:51.243540+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:09:02.739141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238925+0000","last_change":"2024-09-14T03:43:11.275584+0000","last_active":"2024-09-14T03:43:51.238925+0000","last_peered":"2024-09-14T03:43:51.238925+0000","last_clean":"2024-09-14T03:43:51.238925+0000","last_became_active":"2024-09-14T03:43:11.275351+0000","last_became_peered":"2024-09-14T03:43:11.275351+0000","last_unstale":"2024-09-14T03:43:51.238925+0000","last_undegraded":"2024-09-14T03:43:51.238925+0000","last_fullsized":"2024-09-14T03:43:51.238925+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:37:30.284825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240573+0000","last_change":"2024-09-14T03:43:11.291340+0000","last_active":"2024-09-14T03:43:51.240573+0000","last_peered":"2024-09-14T03:43:51.240573+0000","last_clean":"2024-09-14T03:43:51.240573+0000","last_became_active":"2024-09-14T03:43:11.291106+0000","last_became_peered":"2024-09-14T03:43:11.291106+0000","last_unstale":"2024-09-14T03:43:51.240573+0000","last_undegraded":"2024-09-14T03:43:51.240573+0000","last_fullsized":"2024-09-14T03:43:51.240573+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:29:13.611685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":27,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241173+0000","last_change":"2024-09-14T03:42:01.466148+0000","last_active":"2024-09-14T03:43:51.241173+0000","last_peered":"2024-09-14T03:43:51.241173+0000","last_clean":"2024-09-14T03:43:51.241173+0000","last_became_active":"2024-09-14T03:41:56.432336+0000","last_became_peered":"2024-09-14T03:41:56.432336+0000","last_unstale":"2024-09-14T03:43:51.241173+0000","last_undegraded":"2024-09-14T03:43:51.241173+0000","last_fullsized":"2024-09-14T03:43:51.241173+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:33:46.232955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238686+0000","last_change":"2024-09-14T03:43:11.273888+0000","last_active":"2024-09-14T03:43:51.238686+0000","last_peered":"2024-09-14T03:43:51.238686+0000","last_clean":"2024-09-14T03:43:51.238686+0000","last_became_active":"2024-09-14T03:43:11.271711+0000","last_became_peered":"2024-09-14T03:43:11.271711+0000","last_unstale":"2024-09-14T03:43:51.238686+0000","last_undegraded":"2024-09-14T03:43:51.238686+0000","last_fullsized":"2024-09-14T03:43:51.238686+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:23.090131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240014+0000","last_change":"2024-09-14T03:43:11.289990+0000","last_active":"2024-09-14T03:43:51.240014+0000","last_peered":"2024-09-14T03:43:51.240014+0000","last_clean":"2024-09-14T03:43:51.240014+0000","last_became_active":"2024-09-14T03:43:11.289738+0000","last_became_peered":"2024-09-14T03:43:11.289738+0000","last_unstale":"2024-09-14T03:43:51.240014+0000","last_undegraded":"2024-09-14T03:43:51.240014+0000","last_fullsized":"2024-09-14T03:43:51.240014+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:06:15.302539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241777+0000","last_change":"2024-09-14T03:43:11.268363+0000","last_active":"2024-09-14T03:43:51.241777+0000","last_peered":"2024-09-14T03:43:51.241777+0000","last_clean":"2024-09-14T03:43:51.241777+0000","last_became_active":"2024-09-14T03:43:11.268096+0000","last_became_peered":"2024-09-14T03:43:11.268096+0000","last_unstale":"2024-09-14T03:43:51.241777+0000","last_undegraded":"2024-09-14T03:43:51.241777+0000","last_fullsized":"2024-09-14T03:43:51.241777+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:52.934085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240178+0000","last_change":"2024-09-14T03:42:01.467194+0000","last_active":"2024-09-14T03:43:51.240178+0000","last_peered":"2024-09-14T03:43:51.240178+0000","last_clean":"2024-09-14T03:43:51.240178+0000","last_became_active":"2024-09-14T03:41:56.814429+0000","last_became_peered":"2024-09-14T03:41:56.814429+0000","last_unstale":"2024-09-14T03:43:51.240178+0000","last_undegraded":"2024-09-14T03:43:51.240178+0000","last_fullsized":"2024-09-14T03:43:51.240178+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:00:09.573444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239335+0000","last_change":"2024-09-14T03:43:11.293055+0000","last_active":"2024-09-14T03:43:51.239335+0000","last_peered":"2024-09-14T03:43:51.239335+0000","last_clean":"2024-09-14T03:43:51.239335+0000","last_became_active":"2024-09-14T03:43:11.292127+0000","last_became_peered":"2024-09-14T03:43:11.292127+0000","last_unstale":"2024-09-14T03:43:51.239335+0000","last_undegraded":"2024-09-14T03:43:51.239335+0000","last_fullsized":"2024-09-14T03:43:51.239335+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:17.310622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240152+0000","last_change":"2024-09-14T03:43:11.270418+0000","last_active":"2024-09-14T03:43:51.240152+0000","last_peered":"2024-09-14T03:43:51.240152+0000","last_clean":"2024-09-14T03:43:51.240152+0000","last_became_active":"2024-09-14T03:43:11.270169+0000","last_became_peered":"2024-09-14T03:43:11.270169+0000","last_unstale":"2024-09-14T03:43:51.240152+0000","last_undegraded":"2024-09-14T03:43:51.240152+0000","last_fullsized":"2024-09-14T03:43:51.240152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:40:00.047417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244617+0000","last_change":"2024-09-14T03:42:02.106658+0000","last_active":"2024-09-14T03:43:51.244617+0000","last_peered":"2024-09-14T03:43:51.244617+0000","last_clean":"2024-09-14T03:43:51.244617+0000","last_became_active":"2024-09-14T03:41:56.814735+0000","last_became_peered":"2024-09-14T03:41:56.814735+0000","last_unstale":"2024-09-14T03:43:51.244617+0000","last_undegraded":"2024-09-14T03:43:51.244617+0000","last_fullsized":"2024-09-14T03:43:51.244617+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:02:06.281098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.314283+0000","last_change":"2024-09-14T03:43:11.909914+0000","last_active":"2024-09-14T03:43:51.314283+0000","last_peered":"2024-09-14T03:43:51.314283+0000","last_clean":"2024-09-14T03:43:51.314283+0000","last_became_active":"2024-09-14T03:43:11.909624+0000","last_became_peered":"2024-09-14T03:43:11.909624+0000","last_unstale":"2024-09-14T03:43:51.314283+0000","last_undegraded":"2024-09-14T03:43:51.314283+0000","last_fullsized":"2024-09-14T03:43:51.314283+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:04.902577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239232+0000","last_change":"2024-09-14T03:43:11.268277+0000","last_active":"2024-09-14T03:43:51.239232+0000","last_peered":"2024-09-14T03:43:51.239232+0000","last_clean":"2024-09-14T03:43:51.239232+0000","last_became_active":"2024-09-14T03:43:11.267472+0000","last_became_peered":"2024-09-14T03:43:11.267472+0000","last_unstale":"2024-09-14T03:43:51.239232+0000","last_undegraded":"2024-09-14T03:43:51.239232+0000","last_fullsized":"2024-09-14T03:43:51.239232+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:00.495699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240900+0000","last_change":"2024-09-14T03:42:01.467292+0000","last_active":"2024-09-14T03:43:51.240900+0000","last_peered":"2024-09-14T03:43:51.240900+0000","last_clean":"2024-09-14T03:43:51.240900+0000","last_became_active":"2024-09-14T03:41:56.770678+0000","last_became_peered":"2024-09-14T03:41:56.770678+0000","last_unstale":"2024-09-14T03:43:51.240900+0000","last_undegraded":"2024-09-14T03:43:51.240900+0000","last_fullsized":"2024-09-14T03:43:51.240900+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:31:37.700007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040100900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238624+0000","last_change":"2024-09-14T03:43:11.910251+0000","last_active":"2024-09-14T03:43:51.238624+0000","last_peered":"2024-09-14T03:43:51.238624+0000","last_clean":"2024-09-14T03:43:51.238624+0000","last_became_active":"2024-09-14T03:43:11.909784+0000","last_became_peered":"2024-09-14T03:43:11.909784+0000","last_unstale":"2024-09-14T03:43:51.238624+0000","last_undegraded":"2024-09-14T03:43:51.238624+0000","last_fullsized":"2024-09-14T03:43:51.238624+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:47.358499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313758+0000","last_change":"2024-09-14T03:43:11.273116+0000","last_active":"2024-09-14T03:43:51.313758+0000","last_peered":"2024-09-14T03:43:51.313758+0000","last_clean":"2024-09-14T03:43:51.313758+0000","last_became_active":"2024-09-14T03:43:11.272740+0000","last_became_peered":"2024-09-14T03:43:11.272740+0000","last_unstale":"2024-09-14T03:43:51.313758+0000","last_undegraded":"2024-09-14T03:43:51.313758+0000","last_fullsized":"2024-09-14T03:43:51.313758+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:10:22.278926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241931+0000","last_change":"2024-09-14T03:42:01.466959+0000","last_active":"2024-09-14T03:43:51.241931+0000","last_peered":"2024-09-14T03:43:51.241931+0000","last_clean":"2024-09-14T03:43:51.241931+0000","last_became_active":"2024-09-14T03:41:56.432058+0000","last_became_peered":"2024-09-14T03:41:56.432058+0000","last_unstale":"2024-09-14T03:43:51.241931+0000","last_undegraded":"2024-09-14T03:43:51.241931+0000","last_fullsized":"2024-09-14T03:43:51.241931+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:58:30.284511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238511+0000","last_change":"2024-09-14T03:43:11.285521+0000","last_active":"2024-09-14T03:43:51.238511+0000","last_peered":"2024-09-14T03:43:51.238511+0000","last_clean":"2024-09-14T03:43:51.238511+0000","last_became_active":"2024-09-14T03:43:11.284778+0000","last_became_peered":"2024-09-14T03:43:11.284778+0000","last_unstale":"2024-09-14T03:43:51.238511+0000","last_undegraded":"2024-09-14T03:43:51.238511+0000","last_fullsized":"2024-09-14T03:43:51.238511+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:51:32.744643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-14T03:43:52.490179+0000","last_change":"2024-09-14T03:43:52.490179+0000","last_active":"2024-09-14T03:43:52.490179+0000","last_peered":"2024-09-14T03:43:52.490179+0000","last_clean":"2024-09-14T03:43:52.490179+0000","last_became_active":"2024-09-14T03:43:52.489786+0000","last_became_peered":"2024-09-14T03:43:52.489786+0000","last_unstale":"2024-09-14T03:43:52.490179+0000","last_undegraded":"2024-09-14T03:43:52.490179+0000","last_fullsized":"2024-09-14T03:43:52.490179+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:55.590247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"62'6","reported_seq":19,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238641+0000","last_change":"2024-09-14T03:43:11.274139+0000","last_active":"2024-09-14T03:43:51.238641+0000","last_peered":"2024-09-14T03:43:51.238641+0000","last_clean":"2024-09-14T03:43:51.238641+0000","last_became_active":"2024-09-14T03:43:11.271423+0000","last_became_peered":"2024-09-14T03:43:11.271423+0000","last_unstale":"2024-09-14T03:43:51.238641+0000","last_undegraded":"2024-09-14T03:43:51.238641+0000","last_fullsized":"2024-09-14T03:43:51.238641+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:52.045590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239478+0000","last_change":"2024-09-14T03:43:11.292973+0000","last_active":"2024-09-14T03:43:51.239478+0000","last_peered":"2024-09-14T03:43:51.239478+0000","last_clean":"2024-09-14T03:43:51.239478+0000","last_became_active":"2024-09-14T03:43:11.292708+0000","last_became_peered":"2024-09-14T03:43:11.292708+0000","last_unstale":"2024-09-14T03:43:51.239478+0000","last_undegraded":"2024-09-14T03:43:51.239478+0000","last_fullsized":"2024-09-14T03:43:51.239478+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:23:08.353783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":16,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244556+0000","last_change":"2024-09-14T03:43:11.271915+0000","last_active":"2024-09-14T03:43:51.244556+0000","last_peered":"2024-09-14T03:43:51.244556+0000","last_clean":"2024-09-14T03:43:51.244556+0000","last_became_active":"2024-09-14T03:43:11.271664+0000","last_became_peered":"2024-09-14T03:43:11.271664+0000","last_unstale":"2024-09-14T03:43:51.244556+0000","last_undegraded":"2024-09-14T03:43:51.244556+0000","last_fullsized":"2024-09-14T03:43:51.244556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:04:02.324685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241877+0000","last_change":"2024-09-14T03:43:11.272791+0000","last_active":"2024-09-14T03:43:51.241877+0000","last_peered":"2024-09-14T03:43:51.241877+0000","last_clean":"2024-09-14T03:43:51.241877+0000","last_became_active":"2024-09-14T03:43:11.272486+0000","last_became_peered":"2024-09-14T03:43:11.272486+0000","last_unstale":"2024-09-14T03:43:51.241877+0000","last_undegraded":"2024-09-14T03:43:51.241877+0000","last_fullsized":"2024-09-14T03:43:51.241877+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:24:35.175164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238686+0000","last_change":"2024-09-14T03:43:11.910804+0000","last_active":"2024-09-14T03:43:51.238686+0000","last_peered":"2024-09-14T03:43:51.238686+0000","last_clean":"2024-09-14T03:43:51.238686+0000","last_became_active":"2024-09-14T03:43:11.909452+0000","last_became_peered":"2024-09-14T03:43:11.909452+0000","last_unstale":"2024-09-14T03:43:51.238686+0000","last_undegraded":"2024-09-14T03:43:51.238686+0000","last_fullsized":"2024-09-14T03:43:51.238686+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:45:48.632425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313735+0000","last_change":"2024-09-14T03:43:11.273190+0000","last_active":"2024-09-14T03:43:51.313735+0000","last_peered":"2024-09-14T03:43:51.313735+0000","last_clean":"2024-09-14T03:43:51.313735+0000","last_became_active":"2024-09-14T03:43:11.272919+0000","last_became_peered":"2024-09-14T03:43:11.272919+0000","last_unstale":"2024-09-14T03:43:51.313735+0000","last_undegraded":"2024-09-14T03:43:51.313735+0000","last_fullsized":"2024-09-14T03:43:51.313735+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:55:53.166596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238580+0000","last_change":"2024-09-14T03:43:11.291611+0000","last_active":"2024-09-14T03:43:51.238580+0000","last_peered":"2024-09-14T03:43:51.238580+0000","last_clean":"2024-09-14T03:43:51.238580+0000","last_became_active":"2024-09-14T03:43:11.291356+0000","last_became_peered":"2024-09-14T03:43:11.291356+0000","last_unstale":"2024-09-14T03:43:51.238580+0000","last_undegraded":"2024-09-14T03:43:51.238580+0000","last_fullsized":"2024-09-14T03:43:51.238580+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:45:43.188573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-14T03:43:52.490236+0000","last_change":"2024-09-14T03:43:52.490236+0000","last_active":"2024-09-14T03:43:52.490236+0000","last_peered":"2024-09-14T03:43:52.490236+0000","last_clean":"2024-09-14T03:43:52.490236+0000","last_became_active":"2024-09-14T03:43:52.489801+0000","last_became_peered":"2024-09-14T03:43:52.489801+0000","last_unstale":"2024-09-14T03:43:52.490236+0000","last_undegraded":"2024-09-14T03:43:52.490236+0000","last_fullsized":"2024-09-14T03:43:52.490236+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:59.660861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238670+0000","last_change":"2024-09-14T03:43:11.275156+0000","last_active":"2024-09-14T03:43:51.238670+0000","last_peered":"2024-09-14T03:43:51.238670+0000","last_clean":"2024-09-14T03:43:51.238670+0000","last_became_active":"2024-09-14T03:43:11.274675+0000","last_became_peered":"2024-09-14T03:43:11.274675+0000","last_unstale":"2024-09-14T03:43:51.238670+0000","last_undegraded":"2024-09-14T03:43:51.238670+0000","last_fullsized":"2024-09-14T03:43:51.238670+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:33:13.155014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.314265+0000","last_change":"2024-09-14T03:43:11.909693+0000","last_active":"2024-09-14T03:43:51.314265+0000","last_peered":"2024-09-14T03:43:51.314265+0000","last_clean":"2024-09-14T03:43:51.314265+0000","last_became_active":"2024-09-14T03:43:11.909390+0000","last_became_peered":"2024-09-14T03:43:11.909390+0000","last_unstale":"2024-09-14T03:43:51.314265+0000","last_undegraded":"2024-09-14T03:43:51.314265+0000","last_fullsized":"2024-09-14T03:43:51.314265+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:40:10.020049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239096+0000","last_change":"2024-09-14T03:43:11.278401+0000","last_active":"2024-09-14T03:43:51.239096+0000","last_peered":"2024-09-14T03:43:51.239096+0000","last_clean":"2024-09-14T03:43:51.239096+0000","last_became_active":"2024-09-14T03:43:11.278079+0000","last_became_peered":"2024-09-14T03:43:11.278079+0000","last_unstale":"2024-09-14T03:43:51.239096+0000","last_undegraded":"2024-09-14T03:43:51.239096+0000","last_fullsized":"2024-09-14T03:43:51.239096+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:38:20.204502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239350+0000","last_change":"2024-09-14T03:43:11.292407+0000","last_active":"2024-09-14T03:43:51.239350+0000","last_peered":"2024-09-14T03:43:51.239350+0000","last_clean":"2024-09-14T03:43:51.239350+0000","last_became_active":"2024-09-14T03:43:11.292155+0000","last_became_peered":"2024-09-14T03:43:11.292155+0000","last_unstale":"2024-09-14T03:43:51.239350+0000","last_undegraded":"2024-09-14T03:43:51.239350+0000","last_fullsized":"2024-09-14T03:43:51.239350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:49:11.418541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238408+0000","last_change":"2024-09-14T03:43:11.274328+0000","last_active":"2024-09-14T03:43:51.238408+0000","last_peered":"2024-09-14T03:43:51.238408+0000","last_clean":"2024-09-14T03:43:51.238408+0000","last_became_active":"2024-09-14T03:43:11.272047+0000","last_became_peered":"2024-09-14T03:43:11.272047+0000","last_unstale":"2024-09-14T03:43:51.238408+0000","last_undegraded":"2024-09-14T03:43:51.238408+0000","last_fullsized":"2024-09-14T03:43:51.238408+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:51:17.083156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.243621+0000","last_change":"2024-09-14T03:43:11.293188+0000","last_active":"2024-09-14T03:43:51.243621+0000","last_peered":"2024-09-14T03:43:51.243621+0000","last_clean":"2024-09-14T03:43:51.243621+0000","last_became_active":"2024-09-14T03:43:11.292319+0000","last_became_peered":"2024-09-14T03:43:11.292319+0000","last_unstale":"2024-09-14T03:43:51.243621+0000","last_undegraded":"2024-09-14T03:43:51.243621+0000","last_fullsized":"2024-09-14T03:43:51.243621+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:11:13.110464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313692+0000","last_change":"2024-09-14T03:43:11.280101+0000","last_active":"2024-09-14T03:43:51.313692+0000","last_peered":"2024-09-14T03:43:51.313692+0000","last_clean":"2024-09-14T03:43:51.313692+0000","last_became_active":"2024-09-14T03:43:11.276865+0000","last_became_peered":"2024-09-14T03:43:11.276865+0000","last_unstale":"2024-09-14T03:43:51.313692+0000","last_undegraded":"2024-09-14T03:43:51.313692+0000","last_fullsized":"2024-09-14T03:43:51.313692+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:56.508197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238560+0000","last_change":"2024-09-14T03:43:11.293828+0000","last_active":"2024-09-14T03:43:51.238560+0000","last_peered":"2024-09-14T03:43:51.238560+0000","last_clean":"2024-09-14T03:43:51.238560+0000","last_became_active":"2024-09-14T03:43:11.293514+0000","last_became_peered":"2024-09-14T03:43:11.293514+0000","last_unstale":"2024-09-14T03:43:51.238560+0000","last_undegraded":"2024-09-14T03:43:51.238560+0000","last_fullsized":"2024-09-14T03:43:51.238560+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:13:01.990809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.242136+0000","last_change":"2024-09-14T03:43:11.271555+0000","last_active":"2024-09-14T03:43:51.242136+0000","last_peered":"2024-09-14T03:43:51.242136+0000","last_clean":"2024-09-14T03:43:51.242136+0000","last_became_active":"2024-09-14T03:43:11.271294+0000","last_became_peered":"2024-09-14T03:43:11.271294+0000","last_unstale":"2024-09-14T03:43:51.242136+0000","last_undegraded":"2024-09-14T03:43:51.242136+0000","last_fullsized":"2024-09-14T03:43:51.242136+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:46:53.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238324+0000","last_change":"2024-09-14T03:43:11.294806+0000","last_active":"2024-09-14T03:43:51.238324+0000","last_peered":"2024-09-14T03:43:51.238324+0000","last_clean":"2024-09-14T03:43:51.238324+0000","last_became_active":"2024-09-14T03:43:11.294124+0000","last_became_peered":"2024-09-14T03:43:11.294124+0000","last_unstale":"2024-09-14T03:43:51.238324+0000","last_undegraded":"2024-09-14T03:43:51.238324+0000","last_fullsized":"2024-09-14T03:43:51.238324+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:38:49.218800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241692+0000","last_change":"2024-09-14T03:43:11.278982+0000","last_active":"2024-09-14T03:43:51.241692+0000","last_peered":"2024-09-14T03:43:51.241692+0000","last_clean":"2024-09-14T03:43:51.241692+0000","last_became_active":"2024-09-14T03:43:11.277577+0000","last_became_peered":"2024-09-14T03:43:11.277577+0000","last_unstale":"2024-09-14T03:43:51.241692+0000","last_undegraded":"2024-09-14T03:43:51.241692+0000","last_fullsized":"2024-09-14T03:43:51.241692+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:51.445273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238951+0000","last_change":"2024-09-14T03:43:11.289542+0000","last_active":"2024-09-14T03:43:51.238951+0000","last_peered":"2024-09-14T03:43:51.238951+0000","last_clean":"2024-09-14T03:43:51.238951+0000","last_became_active":"2024-09-14T03:43:11.288788+0000","last_became_peered":"2024-09-14T03:43:11.288788+0000","last_unstale":"2024-09-14T03:43:51.238951+0000","last_undegraded":"2024-09-14T03:43:51.238951+0000","last_fullsized":"2024-09-14T03:43:51.238951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:53:55.370873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239590+0000","last_change":"2024-09-14T03:43:11.270532+0000","last_active":"2024-09-14T03:43:51.239590+0000","last_peered":"2024-09-14T03:43:51.239590+0000","last_clean":"2024-09-14T03:43:51.239590+0000","last_became_active":"2024-09-14T03:43:11.270329+0000","last_became_peered":"2024-09-14T03:43:11.270329+0000","last_unstale":"2024-09-14T03:43:51.239590+0000","last_undegraded":"2024-09-14T03:43:51.239590+0000","last_fullsized":"2024-09-14T03:43:51.239590+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:22:08.985484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238218+0000","last_change":"2024-09-14T03:43:11.286621+0000","last_active":"2024-09-14T03:43:51.238218+0000","last_peered":"2024-09-14T03:43:51.238218+0000","last_clean":"2024-09-14T03:43:51.238218+0000","last_became_active":"2024-09-14T03:43:11.286348+0000","last_became_peered":"2024-09-14T03:43:11.286348+0000","last_unstale":"2024-09-14T03:43:51.238218+0000","last_undegraded":"2024-09-14T03:43:51.238218+0000","last_fullsized":"2024-09-14T03:43:51.238218+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:29:36.688172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":17,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313865+0000","last_change":"2024-09-14T03:43:11.269859+0000","last_active":"2024-09-14T03:43:51.313865+0000","last_peered":"2024-09-14T03:43:51.313865+0000","last_clean":"2024-09-14T03:43:51.313865+0000","last_became_active":"2024-09-14T03:43:11.269563+0000","last_became_peered":"2024-09-14T03:43:11.269563+0000","last_unstale":"2024-09-14T03:43:51.313865+0000","last_undegraded":"2024-09-14T03:43:51.313865+0000","last_fullsized":"2024-09-14T03:43:51.313865+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:15:39.583941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430244,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298600,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446552,"statfs":{"total":95995035648,"available":95689269248,"internally_reserved":0,"allocated":1273856,"data_stored":878920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34300000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61099999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.36099999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66100000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64700000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59199999999999997,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57599999999999996}]},{"osd":2,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.5,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.61799999999999999,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.50700000000000001,"15min":0.50600000000000001},"min":{"1min":0.38200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.629,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.61099999999999999}]},{"osd":3,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.39200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.58099999999999996,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.49099999999999999,"15min":0.48699999999999999},"min":{"1min":0.40200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63700000000000001}]},{"osd":4,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66500000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.39700000000000002}]},{"osd":5,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42599999999999999,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.33200000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.45700000000000002}]},{"osd":6,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.42399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.38},{"interface":"front","average":{"1min":0.497,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69199999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47899999999999998}]}]},{"osd":6,"up_from":42,"seq":180388626473,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298588,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446564,"statfs":{"total":95995035648,"available":95689281536,"internally_reserved":0,"allocated":1261568,"data_stored":878478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51000000000000001,"15min":0.51800000000000002},"min":{"1min":0.32800000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67100000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.755,"15min":0.755},"last":0.503}]},{"osd":1,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69499999999999995,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68300000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52800000000000002,"15min":0.54400000000000004},"min":{"1min":0.32700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63100000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.378,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66200000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.52200000000000002}]},{"osd":3,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52900000000000003,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71699999999999997,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54300000000000004,"15min":0.55200000000000005},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72799999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.438}]},{"osd":4,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70399999999999996,"5min":0.751,"15min":0.751},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.432,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.377}]},{"osd":5,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.44400000000000001,"15min":0.44},"min":{"1min":0.30399999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67500000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45600000000000002,"15min":0.44600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67200000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58599999999999997}]},{"osd":7,"last update":"Sat Sep 14 03:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66200000000000003,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53000000000000003,"15min":0.53300000000000003},"min":{"1min":0.29499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73099999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.61199999999999999}]}]},{"osd":5,"up_from":36,"seq":154618822701,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297996,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447156,"statfs":{"total":95995035648,"available":95689887744,"internally_reserved":0,"allocated":655360,"data_stored":286813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70199999999999996,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53000000000000003,"15min":0.53300000000000003},"min":{"1min":0.38600000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.51800000000000002}]},{"osd":1,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39700000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67000000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.51400000000000001},"min":{"1min":0.42099999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.47299999999999998}]},{"osd":3,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67500000000000004,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.436,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.50700000000000001}]},{"osd":4,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.41599999999999998,"15min":0.40799999999999997},"min":{"1min":0.31,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42499999999999999,"15min":0.40200000000000002},"min":{"1min":0.27800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.624,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.34000000000000002}]},{"osd":6,"last update":"Sat Sep 14 03:44:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.45800000000000002,"15min":0.438},"min":{"1min":0.32500000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64300000000000002,"5min":0.748,"15min":0.748},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.45200000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.75,"5min":0.755,"15min":0.755},"last":0.39600000000000002}]},{"osd":7,"last update":"Sat Sep 14 03:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.495,"15min":0.48899999999999999},"min":{"1min":0.33800000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79500000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.49199999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018929,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298016,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447136,"statfs":{"total":95995035648,"available":95689867264,"internally_reserved":0,"allocated":671744,"data_stored":288167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66500000000000004,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.63},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.505,"15min":0.51600000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.67300000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.68799999999999994}]},{"osd":1,"last update":"Sat Sep 14 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52900000000000003,"15min":0.53600000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69699999999999995,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.52700000000000002,"15min":0.54100000000000004},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65000000000000002,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Sep 14 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52100000000000002,"15min":0.53500000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.60499999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53700000000000003,"15min":0.55200000000000005},"min":{"1min":0.33500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67500000000000004,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.65700000000000003}]},{"osd":3,"last update":"Sat Sep 14 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56899999999999995,"15min":0.58499999999999996},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70699999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73299999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.72399999999999998}]},{"osd":5,"last update":"Sat Sep 14 03:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.34300000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.35599999999999998,"15min":0.33600000000000002},"min":{"1min":0.25800000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.55100000000000005}]},{"osd":6,"last update":"Sat Sep 14 03:44:01 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42099999999999999,"15min":0.40899999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.61899999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.502},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.39800000000000002,"15min":0.38100000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Sep 14 03:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.42599999999999999,"15min":0.42099999999999999},"min":{"1min":0.32700000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74299999999999999,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.57699999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297996,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447156,"statfs":{"total":95995035648,"available":95689887744,"internally_reserved":0,"allocated":655360,"data_stored":286874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.35899999999999999,"15min":0.36899999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47399999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.38600000000000001,"15min":0.39700000000000002},"min":{"1min":0.221,"5min":0.219,"15min":0.219},"max":{"1min":0.49299999999999999,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.39400000000000002}]},{"osd":1,"last update":"Sat Sep 14 03:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.38100000000000001,"15min":0.39700000000000002},"min":{"1min":0.23599999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.40400000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.437},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39800000000000002,"15min":0.42199999999999999},"min":{"1min":0.252,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Sep 14 03:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.41499999999999998,"15min":0.437},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.379,"15min":0.39000000000000001},"min":{"1min":0.24399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.46300000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.41499999999999998}]},{"osd":4,"last update":"Sat Sep 14 03:43:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.36199999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73899999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.38600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.75600000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.65900000000000003}]},{"osd":5,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.41499999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.69399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70599999999999996,"5min":0.876,"15min":0.876},"last":0.79000000000000004}]},{"osd":6,"last update":"Sat Sep 14 03:44:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.41699999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.754,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.754}]},{"osd":7,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75800000000000001,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78400000000000003,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.63800000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297996,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447156,"statfs":{"total":95995035648,"available":95689887744,"internally_reserved":0,"allocated":655360,"data_stored":287073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.35799999999999998,"15min":0.36399999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55000000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.371,"15min":0.36399999999999999},"min":{"1min":0.224,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51100000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.32800000000000001}]},{"osd":1,"last update":"Sat Sep 14 03:43:39 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.38900000000000001,"15min":0.39000000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.50800000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.27900000000000003}]},{"osd":3,"last update":"Sat Sep 14 03:44:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.39900000000000002,"15min":0.40899999999999997},"min":{"1min":0.26400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.47799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.41299999999999998,"15min":0.42899999999999999},"min":{"1min":0.30499999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.49199999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Sep 14 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55600000000000005,"15min":0.55000000000000004},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.58799999999999997,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81999999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.58599999999999997}]},{"osd":5,"last update":"Sat Sep 14 03:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73199999999999998,"5min":1,"15min":1},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55100000000000005,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79100000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.61099999999999999}]},{"osd":6,"last update":"Sat Sep 14 03:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.68899999999999995,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.55700000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51900000000000002}]},{"osd":7,"last update":"Sat Sep 14 03:43:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.60899999999999999,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89300000000000002,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.624}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297996,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447156,"statfs":{"total":95995035648,"available":95689887744,"internally_reserved":0,"allocated":655360,"data_stored":286803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34000000000000002,"15min":0.33900000000000002},"min":{"1min":0.23300000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.51600000000000001,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35699999999999998,"15min":0.34300000000000003},"min":{"1min":0.247,"5min":0.224,"15min":0.224},"max":{"1min":0.56699999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Sat Sep 14 03:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35499999999999998,"15min":0.35099999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.45800000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.35899999999999999,"15min":0.35499999999999998},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.41199999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.39200000000000002}]},{"osd":3,"last update":"Sat Sep 14 03:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.28100000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.47599999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.38700000000000001,"15min":0.40400000000000003},"min":{"1min":0.27000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.505,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.374}]},{"osd":4,"last update":"Sat Sep 14 03:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69299999999999995,"5min":0.872,"15min":0.872},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Sep 14 03:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.64300000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.71899999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Sep 14 03:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.33100000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.443},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72699999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Sep 14 03:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.58099999999999996,"15min":0.59299999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71599999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.5},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57699999999999996,"15min":0.58899999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298576,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446576,"statfs":{"total":95995035648,"available":95689293824,"internally_reserved":0,"allocated":1249280,"data_stored":877429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 14 03:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34200000000000003,"15min":0.32800000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.59599999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36099999999999999,"15min":0.32500000000000001},"min":{"1min":0.224,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Sep 14 03:43:37 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.53000000000000003,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.501,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Sep 14 03:43:56 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39100000000000001,"15min":0.40300000000000002},"min":{"1min":0.27100000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.48399999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41799999999999998,"15min":0.42499999999999999},"min":{"1min":0.26800000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51300000000000001,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Sep 14 03:43:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56599999999999995,"15min":0.56499999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75600000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.40100000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60799999999999998}]},{"osd":5,"last update":"Sat Sep 14 03:43:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41399999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69699999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.40699999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76400000000000001,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.61799999999999999}]},{"osd":6,"last update":"Sat Sep 14 03:44:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.41899999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.69499999999999995}]},{"osd":7,"last update":"Sat Sep 14 03:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.74099999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T03:44:10.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T03:44:10.967 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:11.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:11 smithi114 ceph-mon[29589]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:44:11.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:11 smithi114 ceph-mon[40961]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:44:11.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:44:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:44:11.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:11 smithi160 ceph-mon[32702]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:44:12.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:12 smithi114 ceph-mon[29589]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:12.504 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:12 smithi114 ceph-mon[40961]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:12 smithi160 ceph-mon[32702]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:13.009 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:44:13.010 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T03:44:13.642 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-14T03:44:11.198967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385764,"kb_used_data":6912,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575452,"statfs":{"total":767960285184,"available":765517262848,"internally_reserved":0,"allocated":7077888,"data_stored":4070557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002676"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239626+0000","last_change":"2024-09-14T03:43:11.909883+0000","last_active":"2024-09-14T03:43:51.239626+0000","last_peered":"2024-09-14T03:43:51.239626+0000","last_clean":"2024-09-14T03:43:51.239626+0000","last_became_active":"2024-09-14T03:43:11.909587+0000","last_became_peered":"2024-09-14T03:43:11.909587+0000","last_unstale":"2024-09-14T03:43:51.239626+0000","last_undegraded":"2024-09-14T03:43:51.239626+0000","last_fullsized":"2024-09-14T03:43:51.239626+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:30:18.120038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244211+0000","last_change":"2024-09-14T03:43:11.270513+0000","last_active":"2024-09-14T03:43:51.244211+0000","last_peered":"2024-09-14T03:43:51.244211+0000","last_clean":"2024-09-14T03:43:51.244211+0000","last_became_active":"2024-09-14T03:43:11.269672+0000","last_became_peered":"2024-09-14T03:43:11.269672+0000","last_unstale":"2024-09-14T03:43:51.244211+0000","last_undegraded":"2024-09-14T03:43:51.244211+0000","last_fullsized":"2024-09-14T03:43:51.244211+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:56.227349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241595+0000","last_change":"2024-09-14T03:43:11.910241+0000","last_active":"2024-09-14T03:43:51.241595+0000","last_peered":"2024-09-14T03:43:51.241595+0000","last_clean":"2024-09-14T03:43:51.241595+0000","last_became_active":"2024-09-14T03:43:11.909971+0000","last_became_peered":"2024-09-14T03:43:11.909971+0000","last_unstale":"2024-09-14T03:43:51.241595+0000","last_undegraded":"2024-09-14T03:43:51.241595+0000","last_fullsized":"2024-09-14T03:43:51.241595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:01:58.042085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313658+0000","last_change":"2024-09-14T03:43:11.272567+0000","last_active":"2024-09-14T03:43:51.313658+0000","last_peered":"2024-09-14T03:43:51.313658+0000","last_clean":"2024-09-14T03:43:51.313658+0000","last_became_active":"2024-09-14T03:43:11.272293+0000","last_became_peered":"2024-09-14T03:43:11.272293+0000","last_unstale":"2024-09-14T03:43:51.313658+0000","last_undegraded":"2024-09-14T03:43:51.313658+0000","last_fullsized":"2024-09-14T03:43:51.313658+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:43:39.141576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.314389+0000","last_change":"2024-09-14T03:43:11.910637+0000","last_active":"2024-09-14T03:43:51.314389+0000","last_peered":"2024-09-14T03:43:51.314389+0000","last_clean":"2024-09-14T03:43:51.314389+0000","last_became_active":"2024-09-14T03:43:11.910299+0000","last_became_peered":"2024-09-14T03:43:11.910299+0000","last_unstale":"2024-09-14T03:43:51.314389+0000","last_undegraded":"2024-09-14T03:43:51.314389+0000","last_fullsized":"2024-09-14T03:43:51.314389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:23:48.019457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244186+0000","last_change":"2024-09-14T03:43:11.269452+0000","last_active":"2024-09-14T03:43:51.244186+0000","last_peered":"2024-09-14T03:43:51.244186+0000","last_clean":"2024-09-14T03:43:51.244186+0000","last_became_active":"2024-09-14T03:43:11.269185+0000","last_became_peered":"2024-09-14T03:43:11.269185+0000","last_unstale":"2024-09-14T03:43:51.244186+0000","last_undegraded":"2024-09-14T03:43:51.244186+0000","last_fullsized":"2024-09-14T03:43:51.244186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:53:12.602272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239159+0000","last_change":"2024-09-14T03:43:11.909747+0000","last_active":"2024-09-14T03:43:51.239159+0000","last_peered":"2024-09-14T03:43:51.239159+0000","last_clean":"2024-09-14T03:43:51.239159+0000","last_became_active":"2024-09-14T03:43:11.909446+0000","last_became_peered":"2024-09-14T03:43:11.909446+0000","last_unstale":"2024-09-14T03:43:51.239159+0000","last_undegraded":"2024-09-14T03:43:51.239159+0000","last_fullsized":"2024-09-14T03:43:51.239159+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:18:01.366571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":21,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241072+0000","last_change":"2024-09-14T03:43:11.269616+0000","last_active":"2024-09-14T03:43:51.241072+0000","last_peered":"2024-09-14T03:43:51.241072+0000","last_clean":"2024-09-14T03:43:51.241072+0000","last_became_active":"2024-09-14T03:43:11.269359+0000","last_became_peered":"2024-09-14T03:43:11.269359+0000","last_unstale":"2024-09-14T03:43:51.241072+0000","last_undegraded":"2024-09-14T03:43:51.241072+0000","last_fullsized":"2024-09-14T03:43:51.241072+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:33:15.404348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244945+0000","last_change":"2024-09-14T03:43:11.273602+0000","last_active":"2024-09-14T03:43:51.244945+0000","last_peered":"2024-09-14T03:43:51.244945+0000","last_clean":"2024-09-14T03:43:51.244945+0000","last_became_active":"2024-09-14T03:43:11.273307+0000","last_became_peered":"2024-09-14T03:43:11.273307+0000","last_unstale":"2024-09-14T03:43:51.244945+0000","last_undegraded":"2024-09-14T03:43:51.244945+0000","last_fullsized":"2024-09-14T03:43:51.244945+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:54:07.949045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241058+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T03:43:51.241058+0000","last_peered":"2024-09-14T03:43:51.241058+0000","last_clean":"2024-09-14T03:43:51.241058+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T03:43:51.241058+0000","last_undegraded":"2024-09-14T03:43:51.241058+0000","last_fullsized":"2024-09-14T03:43:51.241058+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244552+0000","last_change":"2024-09-14T03:43:11.273656+0000","last_active":"2024-09-14T03:43:51.244552+0000","last_peered":"2024-09-14T03:43:51.244552+0000","last_clean":"2024-09-14T03:43:51.244552+0000","last_became_active":"2024-09-14T03:43:11.273397+0000","last_became_peered":"2024-09-14T03:43:11.273397+0000","last_unstale":"2024-09-14T03:43:51.244552+0000","last_undegraded":"2024-09-14T03:43:51.244552+0000","last_fullsized":"2024-09-14T03:43:51.244552+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:54.630120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238313+0000","last_change":"2024-09-14T03:43:11.295859+0000","last_active":"2024-09-14T03:43:51.238313+0000","last_peered":"2024-09-14T03:43:51.238313+0000","last_clean":"2024-09-14T03:43:51.238313+0000","last_became_active":"2024-09-14T03:43:11.295431+0000","last_became_peered":"2024-09-14T03:43:11.295431+0000","last_unstale":"2024-09-14T03:43:51.238313+0000","last_undegraded":"2024-09-14T03:43:51.238313+0000","last_fullsized":"2024-09-14T03:43:51.238313+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:44:22.965505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244502+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T03:43:51.244502+0000","last_peered":"2024-09-14T03:43:51.244502+0000","last_clean":"2024-09-14T03:43:51.244502+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T03:43:51.244502+0000","last_undegraded":"2024-09-14T03:43:51.244502+0000","last_fullsized":"2024-09-14T03:43:51.244502+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-14T03:43:52.652451+0000","last_change":"2024-09-14T03:43:52.652451+0000","last_active":"2024-09-14T03:43:52.652451+0000","last_peered":"2024-09-14T03:43:52.652451+0000","last_clean":"2024-09-14T03:43:52.652451+0000","last_became_active":"2024-09-14T03:43:52.652081+0000","last_became_peered":"2024-09-14T03:43:52.652081+0000","last_unstale":"2024-09-14T03:43:52.652451+0000","last_undegraded":"2024-09-14T03:43:52.652451+0000","last_fullsized":"2024-09-14T03:43:52.652451+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:24:49.511684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241862+0000","last_change":"2024-09-14T03:43:11.268385+0000","last_active":"2024-09-14T03:43:51.241862+0000","last_peered":"2024-09-14T03:43:51.241862+0000","last_clean":"2024-09-14T03:43:51.241862+0000","last_became_active":"2024-09-14T03:43:11.268099+0000","last_became_peered":"2024-09-14T03:43:11.268099+0000","last_unstale":"2024-09-14T03:43:51.241862+0000","last_undegraded":"2024-09-14T03:43:51.241862+0000","last_fullsized":"2024-09-14T03:43:51.241862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:02:15.398718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313912+0000","last_change":"2024-09-14T03:43:11.290115+0000","last_active":"2024-09-14T03:43:51.313912+0000","last_peered":"2024-09-14T03:43:51.313912+0000","last_clean":"2024-09-14T03:43:51.313912+0000","last_became_active":"2024-09-14T03:43:11.289914+0000","last_became_peered":"2024-09-14T03:43:11.289914+0000","last_unstale":"2024-09-14T03:43:51.313912+0000","last_undegraded":"2024-09-14T03:43:51.313912+0000","last_fullsized":"2024-09-14T03:43:51.313912+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:55:50.945018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239241+0000","last_change":"2024-09-14T03:43:11.268760+0000","last_active":"2024-09-14T03:43:51.239241+0000","last_peered":"2024-09-14T03:43:51.239241+0000","last_clean":"2024-09-14T03:43:51.239241+0000","last_became_active":"2024-09-14T03:43:11.268339+0000","last_became_peered":"2024-09-14T03:43:11.268339+0000","last_unstale":"2024-09-14T03:43:51.239241+0000","last_undegraded":"2024-09-14T03:43:51.239241+0000","last_fullsized":"2024-09-14T03:43:51.239241+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:13.840894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238588+0000","last_change":"2024-09-14T03:43:11.295695+0000","last_active":"2024-09-14T03:43:51.238588+0000","last_peered":"2024-09-14T03:43:51.238588+0000","last_clean":"2024-09-14T03:43:51.238588+0000","last_became_active":"2024-09-14T03:43:11.295441+0000","last_became_peered":"2024-09-14T03:43:11.295441+0000","last_unstale":"2024-09-14T03:43:51.238588+0000","last_undegraded":"2024-09-14T03:43:51.238588+0000","last_fullsized":"2024-09-14T03:43:51.238588+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:12.038600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.242274+0000","last_change":"2024-09-14T03:43:11.274148+0000","last_active":"2024-09-14T03:43:51.242274+0000","last_peered":"2024-09-14T03:43:51.242274+0000","last_clean":"2024-09-14T03:43:51.242274+0000","last_became_active":"2024-09-14T03:43:11.273754+0000","last_became_peered":"2024-09-14T03:43:11.273754+0000","last_unstale":"2024-09-14T03:43:51.242274+0000","last_undegraded":"2024-09-14T03:43:51.242274+0000","last_fullsized":"2024-09-14T03:43:51.242274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:16:08.201934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238749+0000","last_change":"2024-09-14T03:43:11.292860+0000","last_active":"2024-09-14T03:43:51.238749+0000","last_peered":"2024-09-14T03:43:51.238749+0000","last_clean":"2024-09-14T03:43:51.238749+0000","last_became_active":"2024-09-14T03:43:11.292629+0000","last_became_peered":"2024-09-14T03:43:11.292629+0000","last_unstale":"2024-09-14T03:43:51.238749+0000","last_undegraded":"2024-09-14T03:43:51.238749+0000","last_fullsized":"2024-09-14T03:43:51.238749+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:24:35.112097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238730+0000","last_change":"2024-09-14T03:43:11.268184+0000","last_active":"2024-09-14T03:43:51.238730+0000","last_peered":"2024-09-14T03:43:51.238730+0000","last_clean":"2024-09-14T03:43:51.238730+0000","last_became_active":"2024-09-14T03:43:11.267957+0000","last_became_peered":"2024-09-14T03:43:11.267957+0000","last_unstale":"2024-09-14T03:43:51.238730+0000","last_undegraded":"2024-09-14T03:43:51.238730+0000","last_fullsized":"2024-09-14T03:43:51.238730+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:23:59.632674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238787+0000","last_change":"2024-09-14T03:43:11.286604+0000","last_active":"2024-09-14T03:43:51.238787+0000","last_peered":"2024-09-14T03:43:51.238787+0000","last_clean":"2024-09-14T03:43:51.238787+0000","last_became_active":"2024-09-14T03:43:11.286329+0000","last_became_peered":"2024-09-14T03:43:11.286329+0000","last_unstale":"2024-09-14T03:43:51.238787+0000","last_undegraded":"2024-09-14T03:43:51.238787+0000","last_fullsized":"2024-09-14T03:43:51.238787+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:54:15.941054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238708+0000","last_change":"2024-09-14T03:43:11.275058+0000","last_active":"2024-09-14T03:43:51.238708+0000","last_peered":"2024-09-14T03:43:51.238708+0000","last_clean":"2024-09-14T03:43:51.238708+0000","last_became_active":"2024-09-14T03:43:11.273704+0000","last_became_peered":"2024-09-14T03:43:11.273704+0000","last_unstale":"2024-09-14T03:43:51.238708+0000","last_undegraded":"2024-09-14T03:43:51.238708+0000","last_fullsized":"2024-09-14T03:43:51.238708+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:58:59.923574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240503+0000","last_change":"2024-09-14T03:43:11.293157+0000","last_active":"2024-09-14T03:43:51.240503+0000","last_peered":"2024-09-14T03:43:51.240503+0000","last_clean":"2024-09-14T03:43:51.240503+0000","last_became_active":"2024-09-14T03:43:11.291638+0000","last_became_peered":"2024-09-14T03:43:11.291638+0000","last_unstale":"2024-09-14T03:43:51.240503+0000","last_undegraded":"2024-09-14T03:43:51.240503+0000","last_fullsized":"2024-09-14T03:43:51.240503+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:28:52.185952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"51'79","reported_seq":55,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244102+0000","last_change":"2024-09-14T03:41:25.453993+0000","last_active":"2024-09-14T03:43:51.244102+0000","last_peered":"2024-09-14T03:43:51.244102+0000","last_clean":"2024-09-14T03:43:51.244102+0000","last_became_active":"2024-09-14T03:41:25.446786+0000","last_became_peered":"2024-09-14T03:41:25.446786+0000","last_unstale":"2024-09-14T03:43:51.244102+0000","last_undegraded":"2024-09-14T03:43:51.244102+0000","last_fullsized":"2024-09-14T03:43:51.244102+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:50:17.999993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":26,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241721+0000","last_change":"2024-09-14T03:42:01.465792+0000","last_active":"2024-09-14T03:43:51.241721+0000","last_peered":"2024-09-14T03:43:51.241721+0000","last_clean":"2024-09-14T03:43:51.241721+0000","last_became_active":"2024-09-14T03:41:56.433826+0000","last_became_peered":"2024-09-14T03:41:56.433826+0000","last_unstale":"2024-09-14T03:43:51.241721+0000","last_undegraded":"2024-09-14T03:43:51.241721+0000","last_fullsized":"2024-09-14T03:43:51.241721+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:45:32.233869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239148+0000","last_change":"2024-09-14T03:43:11.269492+0000","last_active":"2024-09-14T03:43:51.239148+0000","last_peered":"2024-09-14T03:43:51.239148+0000","last_clean":"2024-09-14T03:43:51.239148+0000","last_became_active":"2024-09-14T03:43:11.269228+0000","last_became_peered":"2024-09-14T03:43:11.269228+0000","last_unstale":"2024-09-14T03:43:51.239148+0000","last_undegraded":"2024-09-14T03:43:51.239148+0000","last_fullsized":"2024-09-14T03:43:51.239148+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:23:01.054451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241355+0000","last_change":"2024-09-14T03:43:11.909945+0000","last_active":"2024-09-14T03:43:51.241355+0000","last_peered":"2024-09-14T03:43:51.241355+0000","last_clean":"2024-09-14T03:43:51.241355+0000","last_became_active":"2024-09-14T03:43:11.909666+0000","last_became_peered":"2024-09-14T03:43:11.909666+0000","last_unstale":"2024-09-14T03:43:51.241355+0000","last_undegraded":"2024-09-14T03:43:51.241355+0000","last_fullsized":"2024-09-14T03:43:51.241355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:13:19.629641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.243976+0000","last_change":"2024-09-14T03:42:02.106608+0000","last_active":"2024-09-14T03:43:51.243976+0000","last_peered":"2024-09-14T03:43:51.243976+0000","last_clean":"2024-09-14T03:43:51.243976+0000","last_became_active":"2024-09-14T03:41:56.814765+0000","last_became_peered":"2024-09-14T03:41:56.814765+0000","last_unstale":"2024-09-14T03:43:51.243976+0000","last_undegraded":"2024-09-14T03:43:51.243976+0000","last_fullsized":"2024-09-14T03:43:51.243976+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:15:23.921629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313693+0000","last_change":"2024-09-14T03:43:11.272354+0000","last_active":"2024-09-14T03:43:51.313693+0000","last_peered":"2024-09-14T03:43:51.313693+0000","last_clean":"2024-09-14T03:43:51.313693+0000","last_became_active":"2024-09-14T03:43:11.272092+0000","last_became_peered":"2024-09-14T03:43:11.272092+0000","last_unstale":"2024-09-14T03:43:51.313693+0000","last_undegraded":"2024-09-14T03:43:51.313693+0000","last_fullsized":"2024-09-14T03:43:51.313693+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:08.709427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313949+0000","last_change":"2024-09-14T03:43:11.289253+0000","last_active":"2024-09-14T03:43:51.313949+0000","last_peered":"2024-09-14T03:43:51.313949+0000","last_clean":"2024-09-14T03:43:51.313949+0000","last_became_active":"2024-09-14T03:43:11.288840+0000","last_became_peered":"2024-09-14T03:43:11.288840+0000","last_unstale":"2024-09-14T03:43:51.313949+0000","last_undegraded":"2024-09-14T03:43:51.313949+0000","last_fullsized":"2024-09-14T03:43:51.313949+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:48:27.761229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.243540+0000","last_change":"2024-09-14T03:42:01.461806+0000","last_active":"2024-09-14T03:43:51.243540+0000","last_peered":"2024-09-14T03:43:51.243540+0000","last_clean":"2024-09-14T03:43:51.243540+0000","last_became_active":"2024-09-14T03:41:56.814735+0000","last_became_peered":"2024-09-14T03:41:56.814735+0000","last_unstale":"2024-09-14T03:43:51.243540+0000","last_undegraded":"2024-09-14T03:43:51.243540+0000","last_fullsized":"2024-09-14T03:43:51.243540+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:09:02.739141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238925+0000","last_change":"2024-09-14T03:43:11.275584+0000","last_active":"2024-09-14T03:43:51.238925+0000","last_peered":"2024-09-14T03:43:51.238925+0000","last_clean":"2024-09-14T03:43:51.238925+0000","last_became_active":"2024-09-14T03:43:11.275351+0000","last_became_peered":"2024-09-14T03:43:11.275351+0000","last_unstale":"2024-09-14T03:43:51.238925+0000","last_undegraded":"2024-09-14T03:43:51.238925+0000","last_fullsized":"2024-09-14T03:43:51.238925+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:37:30.284825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240573+0000","last_change":"2024-09-14T03:43:11.291340+0000","last_active":"2024-09-14T03:43:51.240573+0000","last_peered":"2024-09-14T03:43:51.240573+0000","last_clean":"2024-09-14T03:43:51.240573+0000","last_became_active":"2024-09-14T03:43:11.291106+0000","last_became_peered":"2024-09-14T03:43:11.291106+0000","last_unstale":"2024-09-14T03:43:51.240573+0000","last_undegraded":"2024-09-14T03:43:51.240573+0000","last_fullsized":"2024-09-14T03:43:51.240573+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:29:13.611685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":27,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241173+0000","last_change":"2024-09-14T03:42:01.466148+0000","last_active":"2024-09-14T03:43:51.241173+0000","last_peered":"2024-09-14T03:43:51.241173+0000","last_clean":"2024-09-14T03:43:51.241173+0000","last_became_active":"2024-09-14T03:41:56.432336+0000","last_became_peered":"2024-09-14T03:41:56.432336+0000","last_unstale":"2024-09-14T03:43:51.241173+0000","last_undegraded":"2024-09-14T03:43:51.241173+0000","last_fullsized":"2024-09-14T03:43:51.241173+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:33:46.232955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238686+0000","last_change":"2024-09-14T03:43:11.273888+0000","last_active":"2024-09-14T03:43:51.238686+0000","last_peered":"2024-09-14T03:43:51.238686+0000","last_clean":"2024-09-14T03:43:51.238686+0000","last_became_active":"2024-09-14T03:43:11.271711+0000","last_became_peered":"2024-09-14T03:43:11.271711+0000","last_unstale":"2024-09-14T03:43:51.238686+0000","last_undegraded":"2024-09-14T03:43:51.238686+0000","last_fullsized":"2024-09-14T03:43:51.238686+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:23.090131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240014+0000","last_change":"2024-09-14T03:43:11.289990+0000","last_active":"2024-09-14T03:43:51.240014+0000","last_peered":"2024-09-14T03:43:51.240014+0000","last_clean":"2024-09-14T03:43:51.240014+0000","last_became_active":"2024-09-14T03:43:11.289738+0000","last_became_peered":"2024-09-14T03:43:11.289738+0000","last_unstale":"2024-09-14T03:43:51.240014+0000","last_undegraded":"2024-09-14T03:43:51.240014+0000","last_fullsized":"2024-09-14T03:43:51.240014+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:06:15.302539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241777+0000","last_change":"2024-09-14T03:43:11.268363+0000","last_active":"2024-09-14T03:43:51.241777+0000","last_peered":"2024-09-14T03:43:51.241777+0000","last_clean":"2024-09-14T03:43:51.241777+0000","last_became_active":"2024-09-14T03:43:11.268096+0000","last_became_peered":"2024-09-14T03:43:11.268096+0000","last_unstale":"2024-09-14T03:43:51.241777+0000","last_undegraded":"2024-09-14T03:43:51.241777+0000","last_fullsized":"2024-09-14T03:43:51.241777+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:52.934085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240178+0000","last_change":"2024-09-14T03:42:01.467194+0000","last_active":"2024-09-14T03:43:51.240178+0000","last_peered":"2024-09-14T03:43:51.240178+0000","last_clean":"2024-09-14T03:43:51.240178+0000","last_became_active":"2024-09-14T03:41:56.814429+0000","last_became_peered":"2024-09-14T03:41:56.814429+0000","last_unstale":"2024-09-14T03:43:51.240178+0000","last_undegraded":"2024-09-14T03:43:51.240178+0000","last_fullsized":"2024-09-14T03:43:51.240178+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:00:09.573444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239335+0000","last_change":"2024-09-14T03:43:11.293055+0000","last_active":"2024-09-14T03:43:51.239335+0000","last_peered":"2024-09-14T03:43:51.239335+0000","last_clean":"2024-09-14T03:43:51.239335+0000","last_became_active":"2024-09-14T03:43:11.292127+0000","last_became_peered":"2024-09-14T03:43:11.292127+0000","last_unstale":"2024-09-14T03:43:51.239335+0000","last_undegraded":"2024-09-14T03:43:51.239335+0000","last_fullsized":"2024-09-14T03:43:51.239335+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:17.310622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240152+0000","last_change":"2024-09-14T03:43:11.270418+0000","last_active":"2024-09-14T03:43:51.240152+0000","last_peered":"2024-09-14T03:43:51.240152+0000","last_clean":"2024-09-14T03:43:51.240152+0000","last_became_active":"2024-09-14T03:43:11.270169+0000","last_became_peered":"2024-09-14T03:43:11.270169+0000","last_unstale":"2024-09-14T03:43:51.240152+0000","last_undegraded":"2024-09-14T03:43:51.240152+0000","last_fullsized":"2024-09-14T03:43:51.240152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:40:00.047417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244617+0000","last_change":"2024-09-14T03:42:02.106658+0000","last_active":"2024-09-14T03:43:51.244617+0000","last_peered":"2024-09-14T03:43:51.244617+0000","last_clean":"2024-09-14T03:43:51.244617+0000","last_became_active":"2024-09-14T03:41:56.814735+0000","last_became_peered":"2024-09-14T03:41:56.814735+0000","last_unstale":"2024-09-14T03:43:51.244617+0000","last_undegraded":"2024-09-14T03:43:51.244617+0000","last_fullsized":"2024-09-14T03:43:51.244617+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:02:06.281098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.314283+0000","last_change":"2024-09-14T03:43:11.909914+0000","last_active":"2024-09-14T03:43:51.314283+0000","last_peered":"2024-09-14T03:43:51.314283+0000","last_clean":"2024-09-14T03:43:51.314283+0000","last_became_active":"2024-09-14T03:43:11.909624+0000","last_became_peered":"2024-09-14T03:43:11.909624+0000","last_unstale":"2024-09-14T03:43:51.314283+0000","last_undegraded":"2024-09-14T03:43:51.314283+0000","last_fullsized":"2024-09-14T03:43:51.314283+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:04.902577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239232+0000","last_change":"2024-09-14T03:43:11.268277+0000","last_active":"2024-09-14T03:43:51.239232+0000","last_peered":"2024-09-14T03:43:51.239232+0000","last_clean":"2024-09-14T03:43:51.239232+0000","last_became_active":"2024-09-14T03:43:11.267472+0000","last_became_peered":"2024-09-14T03:43:11.267472+0000","last_unstale":"2024-09-14T03:43:51.239232+0000","last_undegraded":"2024-09-14T03:43:51.239232+0000","last_fullsized":"2024-09-14T03:43:51.239232+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:00.495699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.240900+0000","last_change":"2024-09-14T03:42:01.467292+0000","last_active":"2024-09-14T03:43:51.240900+0000","last_peered":"2024-09-14T03:43:51.240900+0000","last_clean":"2024-09-14T03:43:51.240900+0000","last_became_active":"2024-09-14T03:41:56.770678+0000","last_became_peered":"2024-09-14T03:41:56.770678+0000","last_unstale":"2024-09-14T03:43:51.240900+0000","last_undegraded":"2024-09-14T03:43:51.240900+0000","last_fullsized":"2024-09-14T03:43:51.240900+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:31:37.700007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040100900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238624+0000","last_change":"2024-09-14T03:43:11.910251+0000","last_active":"2024-09-14T03:43:51.238624+0000","last_peered":"2024-09-14T03:43:51.238624+0000","last_clean":"2024-09-14T03:43:51.238624+0000","last_became_active":"2024-09-14T03:43:11.909784+0000","last_became_peered":"2024-09-14T03:43:11.909784+0000","last_unstale":"2024-09-14T03:43:51.238624+0000","last_undegraded":"2024-09-14T03:43:51.238624+0000","last_fullsized":"2024-09-14T03:43:51.238624+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:47.358499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313758+0000","last_change":"2024-09-14T03:43:11.273116+0000","last_active":"2024-09-14T03:43:51.313758+0000","last_peered":"2024-09-14T03:43:51.313758+0000","last_clean":"2024-09-14T03:43:51.313758+0000","last_became_active":"2024-09-14T03:43:11.272740+0000","last_became_peered":"2024-09-14T03:43:11.272740+0000","last_unstale":"2024-09-14T03:43:51.313758+0000","last_undegraded":"2024-09-14T03:43:51.313758+0000","last_fullsized":"2024-09-14T03:43:51.313758+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:10:22.278926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241931+0000","last_change":"2024-09-14T03:42:01.466959+0000","last_active":"2024-09-14T03:43:51.241931+0000","last_peered":"2024-09-14T03:43:51.241931+0000","last_clean":"2024-09-14T03:43:51.241931+0000","last_became_active":"2024-09-14T03:41:56.432058+0000","last_became_peered":"2024-09-14T03:41:56.432058+0000","last_unstale":"2024-09-14T03:43:51.241931+0000","last_undegraded":"2024-09-14T03:43:51.241931+0000","last_fullsized":"2024-09-14T03:43:51.241931+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:58:30.284511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238511+0000","last_change":"2024-09-14T03:43:11.285521+0000","last_active":"2024-09-14T03:43:51.238511+0000","last_peered":"2024-09-14T03:43:51.238511+0000","last_clean":"2024-09-14T03:43:51.238511+0000","last_became_active":"2024-09-14T03:43:11.284778+0000","last_became_peered":"2024-09-14T03:43:11.284778+0000","last_unstale":"2024-09-14T03:43:51.238511+0000","last_undegraded":"2024-09-14T03:43:51.238511+0000","last_fullsized":"2024-09-14T03:43:51.238511+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:51:32.744643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-14T03:43:52.490179+0000","last_change":"2024-09-14T03:43:52.490179+0000","last_active":"2024-09-14T03:43:52.490179+0000","last_peered":"2024-09-14T03:43:52.490179+0000","last_clean":"2024-09-14T03:43:52.490179+0000","last_became_active":"2024-09-14T03:43:52.489786+0000","last_became_peered":"2024-09-14T03:43:52.489786+0000","last_unstale":"2024-09-14T03:43:52.490179+0000","last_undegraded":"2024-09-14T03:43:52.490179+0000","last_fullsized":"2024-09-14T03:43:52.490179+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:55.590247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"62'6","reported_seq":19,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238641+0000","last_change":"2024-09-14T03:43:11.274139+0000","last_active":"2024-09-14T03:43:51.238641+0000","last_peered":"2024-09-14T03:43:51.238641+0000","last_clean":"2024-09-14T03:43:51.238641+0000","last_became_active":"2024-09-14T03:43:11.271423+0000","last_became_peered":"2024-09-14T03:43:11.271423+0000","last_unstale":"2024-09-14T03:43:51.238641+0000","last_undegraded":"2024-09-14T03:43:51.238641+0000","last_fullsized":"2024-09-14T03:43:51.238641+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:52.045590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239478+0000","last_change":"2024-09-14T03:43:11.292973+0000","last_active":"2024-09-14T03:43:51.239478+0000","last_peered":"2024-09-14T03:43:51.239478+0000","last_clean":"2024-09-14T03:43:51.239478+0000","last_became_active":"2024-09-14T03:43:11.292708+0000","last_became_peered":"2024-09-14T03:43:11.292708+0000","last_unstale":"2024-09-14T03:43:51.239478+0000","last_undegraded":"2024-09-14T03:43:51.239478+0000","last_fullsized":"2024-09-14T03:43:51.239478+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:23:08.353783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":16,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.244556+0000","last_change":"2024-09-14T03:43:11.271915+0000","last_active":"2024-09-14T03:43:51.244556+0000","last_peered":"2024-09-14T03:43:51.244556+0000","last_clean":"2024-09-14T03:43:51.244556+0000","last_became_active":"2024-09-14T03:43:11.271664+0000","last_became_peered":"2024-09-14T03:43:11.271664+0000","last_unstale":"2024-09-14T03:43:51.244556+0000","last_undegraded":"2024-09-14T03:43:51.244556+0000","last_fullsized":"2024-09-14T03:43:51.244556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:04:02.324685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241877+0000","last_change":"2024-09-14T03:43:11.272791+0000","last_active":"2024-09-14T03:43:51.241877+0000","last_peered":"2024-09-14T03:43:51.241877+0000","last_clean":"2024-09-14T03:43:51.241877+0000","last_became_active":"2024-09-14T03:43:11.272486+0000","last_became_peered":"2024-09-14T03:43:11.272486+0000","last_unstale":"2024-09-14T03:43:51.241877+0000","last_undegraded":"2024-09-14T03:43:51.241877+0000","last_fullsized":"2024-09-14T03:43:51.241877+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:24:35.175164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238686+0000","last_change":"2024-09-14T03:43:11.910804+0000","last_active":"2024-09-14T03:43:51.238686+0000","last_peered":"2024-09-14T03:43:51.238686+0000","last_clean":"2024-09-14T03:43:51.238686+0000","last_became_active":"2024-09-14T03:43:11.909452+0000","last_became_peered":"2024-09-14T03:43:11.909452+0000","last_unstale":"2024-09-14T03:43:51.238686+0000","last_undegraded":"2024-09-14T03:43:51.238686+0000","last_fullsized":"2024-09-14T03:43:51.238686+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:45:48.632425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313735+0000","last_change":"2024-09-14T03:43:11.273190+0000","last_active":"2024-09-14T03:43:51.313735+0000","last_peered":"2024-09-14T03:43:51.313735+0000","last_clean":"2024-09-14T03:43:51.313735+0000","last_became_active":"2024-09-14T03:43:11.272919+0000","last_became_peered":"2024-09-14T03:43:11.272919+0000","last_unstale":"2024-09-14T03:43:51.313735+0000","last_undegraded":"2024-09-14T03:43:51.313735+0000","last_fullsized":"2024-09-14T03:43:51.313735+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:55:53.166596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238580+0000","last_change":"2024-09-14T03:43:11.291611+0000","last_active":"2024-09-14T03:43:51.238580+0000","last_peered":"2024-09-14T03:43:51.238580+0000","last_clean":"2024-09-14T03:43:51.238580+0000","last_became_active":"2024-09-14T03:43:11.291356+0000","last_became_peered":"2024-09-14T03:43:11.291356+0000","last_unstale":"2024-09-14T03:43:51.238580+0000","last_undegraded":"2024-09-14T03:43:51.238580+0000","last_fullsized":"2024-09-14T03:43:51.238580+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:45:43.188573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-14T03:43:52.490236+0000","last_change":"2024-09-14T03:43:52.490236+0000","last_active":"2024-09-14T03:43:52.490236+0000","last_peered":"2024-09-14T03:43:52.490236+0000","last_clean":"2024-09-14T03:43:52.490236+0000","last_became_active":"2024-09-14T03:43:52.489801+0000","last_became_peered":"2024-09-14T03:43:52.489801+0000","last_unstale":"2024-09-14T03:43:52.490236+0000","last_undegraded":"2024-09-14T03:43:52.490236+0000","last_fullsized":"2024-09-14T03:43:52.490236+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:59.660861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238670+0000","last_change":"2024-09-14T03:43:11.275156+0000","last_active":"2024-09-14T03:43:51.238670+0000","last_peered":"2024-09-14T03:43:51.238670+0000","last_clean":"2024-09-14T03:43:51.238670+0000","last_became_active":"2024-09-14T03:43:11.274675+0000","last_became_peered":"2024-09-14T03:43:11.274675+0000","last_unstale":"2024-09-14T03:43:51.238670+0000","last_undegraded":"2024-09-14T03:43:51.238670+0000","last_fullsized":"2024-09-14T03:43:51.238670+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:33:13.155014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.314265+0000","last_change":"2024-09-14T03:43:11.909693+0000","last_active":"2024-09-14T03:43:51.314265+0000","last_peered":"2024-09-14T03:43:51.314265+0000","last_clean":"2024-09-14T03:43:51.314265+0000","last_became_active":"2024-09-14T03:43:11.909390+0000","last_became_peered":"2024-09-14T03:43:11.909390+0000","last_unstale":"2024-09-14T03:43:51.314265+0000","last_undegraded":"2024-09-14T03:43:51.314265+0000","last_fullsized":"2024-09-14T03:43:51.314265+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:40:10.020049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239096+0000","last_change":"2024-09-14T03:43:11.278401+0000","last_active":"2024-09-14T03:43:51.239096+0000","last_peered":"2024-09-14T03:43:51.239096+0000","last_clean":"2024-09-14T03:43:51.239096+0000","last_became_active":"2024-09-14T03:43:11.278079+0000","last_became_peered":"2024-09-14T03:43:11.278079+0000","last_unstale":"2024-09-14T03:43:51.239096+0000","last_undegraded":"2024-09-14T03:43:51.239096+0000","last_fullsized":"2024-09-14T03:43:51.239096+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:38:20.204502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239350+0000","last_change":"2024-09-14T03:43:11.292407+0000","last_active":"2024-09-14T03:43:51.239350+0000","last_peered":"2024-09-14T03:43:51.239350+0000","last_clean":"2024-09-14T03:43:51.239350+0000","last_became_active":"2024-09-14T03:43:11.292155+0000","last_became_peered":"2024-09-14T03:43:11.292155+0000","last_unstale":"2024-09-14T03:43:51.239350+0000","last_undegraded":"2024-09-14T03:43:51.239350+0000","last_fullsized":"2024-09-14T03:43:51.239350+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:49:11.418541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238408+0000","last_change":"2024-09-14T03:43:11.274328+0000","last_active":"2024-09-14T03:43:51.238408+0000","last_peered":"2024-09-14T03:43:51.238408+0000","last_clean":"2024-09-14T03:43:51.238408+0000","last_became_active":"2024-09-14T03:43:11.272047+0000","last_became_peered":"2024-09-14T03:43:11.272047+0000","last_unstale":"2024-09-14T03:43:51.238408+0000","last_undegraded":"2024-09-14T03:43:51.238408+0000","last_fullsized":"2024-09-14T03:43:51.238408+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:51:17.083156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.243621+0000","last_change":"2024-09-14T03:43:11.293188+0000","last_active":"2024-09-14T03:43:51.243621+0000","last_peered":"2024-09-14T03:43:51.243621+0000","last_clean":"2024-09-14T03:43:51.243621+0000","last_became_active":"2024-09-14T03:43:11.292319+0000","last_became_peered":"2024-09-14T03:43:11.292319+0000","last_unstale":"2024-09-14T03:43:51.243621+0000","last_undegraded":"2024-09-14T03:43:51.243621+0000","last_fullsized":"2024-09-14T03:43:51.243621+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:11:13.110464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313692+0000","last_change":"2024-09-14T03:43:11.280101+0000","last_active":"2024-09-14T03:43:51.313692+0000","last_peered":"2024-09-14T03:43:51.313692+0000","last_clean":"2024-09-14T03:43:51.313692+0000","last_became_active":"2024-09-14T03:43:11.276865+0000","last_became_peered":"2024-09-14T03:43:11.276865+0000","last_unstale":"2024-09-14T03:43:51.313692+0000","last_undegraded":"2024-09-14T03:43:51.313692+0000","last_fullsized":"2024-09-14T03:43:51.313692+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:56.508197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238560+0000","last_change":"2024-09-14T03:43:11.293828+0000","last_active":"2024-09-14T03:43:51.238560+0000","last_peered":"2024-09-14T03:43:51.238560+0000","last_clean":"2024-09-14T03:43:51.238560+0000","last_became_active":"2024-09-14T03:43:11.293514+0000","last_became_peered":"2024-09-14T03:43:11.293514+0000","last_unstale":"2024-09-14T03:43:51.238560+0000","last_undegraded":"2024-09-14T03:43:51.238560+0000","last_fullsized":"2024-09-14T03:43:51.238560+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:13:01.990809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.242136+0000","last_change":"2024-09-14T03:43:11.271555+0000","last_active":"2024-09-14T03:43:51.242136+0000","last_peered":"2024-09-14T03:43:51.242136+0000","last_clean":"2024-09-14T03:43:51.242136+0000","last_became_active":"2024-09-14T03:43:11.271294+0000","last_became_peered":"2024-09-14T03:43:11.271294+0000","last_unstale":"2024-09-14T03:43:51.242136+0000","last_undegraded":"2024-09-14T03:43:51.242136+0000","last_fullsized":"2024-09-14T03:43:51.242136+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:46:53.110669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238324+0000","last_change":"2024-09-14T03:43:11.294806+0000","last_active":"2024-09-14T03:43:51.238324+0000","last_peered":"2024-09-14T03:43:51.238324+0000","last_clean":"2024-09-14T03:43:51.238324+0000","last_became_active":"2024-09-14T03:43:11.294124+0000","last_became_peered":"2024-09-14T03:43:11.294124+0000","last_unstale":"2024-09-14T03:43:51.238324+0000","last_undegraded":"2024-09-14T03:43:51.238324+0000","last_fullsized":"2024-09-14T03:43:51.238324+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:38:49.218800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.241692+0000","last_change":"2024-09-14T03:43:11.278982+0000","last_active":"2024-09-14T03:43:51.241692+0000","last_peered":"2024-09-14T03:43:51.241692+0000","last_clean":"2024-09-14T03:43:51.241692+0000","last_became_active":"2024-09-14T03:43:11.277577+0000","last_became_peered":"2024-09-14T03:43:11.277577+0000","last_unstale":"2024-09-14T03:43:51.241692+0000","last_undegraded":"2024-09-14T03:43:51.241692+0000","last_fullsized":"2024-09-14T03:43:51.241692+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:51.445273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238951+0000","last_change":"2024-09-14T03:43:11.289542+0000","last_active":"2024-09-14T03:43:51.238951+0000","last_peered":"2024-09-14T03:43:51.238951+0000","last_clean":"2024-09-14T03:43:51.238951+0000","last_became_active":"2024-09-14T03:43:11.288788+0000","last_became_peered":"2024-09-14T03:43:11.288788+0000","last_unstale":"2024-09-14T03:43:51.238951+0000","last_undegraded":"2024-09-14T03:43:51.238951+0000","last_fullsized":"2024-09-14T03:43:51.238951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:53:55.370873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.239590+0000","last_change":"2024-09-14T03:43:11.270532+0000","last_active":"2024-09-14T03:43:51.239590+0000","last_peered":"2024-09-14T03:43:51.239590+0000","last_clean":"2024-09-14T03:43:51.239590+0000","last_became_active":"2024-09-14T03:43:11.270329+0000","last_became_peered":"2024-09-14T03:43:11.270329+0000","last_unstale":"2024-09-14T03:43:51.239590+0000","last_undegraded":"2024-09-14T03:43:51.239590+0000","last_fullsized":"2024-09-14T03:43:51.239590+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:22:08.985484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.238218+0000","last_change":"2024-09-14T03:43:11.286621+0000","last_active":"2024-09-14T03:43:51.238218+0000","last_peered":"2024-09-14T03:43:51.238218+0000","last_clean":"2024-09-14T03:43:51.238218+0000","last_became_active":"2024-09-14T03:43:11.286348+0000","last_became_peered":"2024-09-14T03:43:11.286348+0000","last_unstale":"2024-09-14T03:43:51.238218+0000","last_undegraded":"2024-09-14T03:43:51.238218+0000","last_fullsized":"2024-09-14T03:43:51.238218+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:29:36.688172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":17,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-14T03:43:51.313865+0000","last_change":"2024-09-14T03:43:11.269859+0000","last_active":"2024-09-14T03:43:51.313865+0000","last_peered":"2024-09-14T03:43:51.313865+0000","last_clean":"2024-09-14T03:43:51.313865+0000","last_became_active":"2024-09-14T03:43:11.269563+0000","last_became_peered":"2024-09-14T03:43:11.269563+0000","last_unstale":"2024-09-14T03:43:51.313865+0000","last_undegraded":"2024-09-14T03:43:51.313865+0000","last_fullsized":"2024-09-14T03:43:51.313865+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:15:39.583941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430244,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298600,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446552,"statfs":{"total":95995035648,"available":95689269248,"internally_reserved":0,"allocated":1273856,"data_stored":878920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34300000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61099999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.36099999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66100000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64700000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59199999999999997,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57599999999999996}]},{"osd":2,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.5,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.61799999999999999,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.50700000000000001,"15min":0.50600000000000001},"min":{"1min":0.38200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.629,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.61099999999999999}]},{"osd":3,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.39200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.58099999999999996,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.49099999999999999,"15min":0.48699999999999999},"min":{"1min":0.40200000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63700000000000001}]},{"osd":4,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66500000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.42999999999999999,"15min":0.42599999999999999},"min":{"1min":0.312,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.39700000000000002}]},{"osd":5,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42599999999999999,"15min":0.42199999999999999},"min":{"1min":0.33500000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.33200000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.45700000000000002}]},{"osd":6,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.48799999999999999,"15min":0.48199999999999998},"min":{"1min":0.42399999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.38},{"interface":"front","average":{"1min":0.497,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69199999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47899999999999998}]}]},{"osd":6,"up_from":42,"seq":180388626473,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298588,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446564,"statfs":{"total":95995035648,"available":95689281536,"internally_reserved":0,"allocated":1261568,"data_stored":878478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51000000000000001,"15min":0.51800000000000002},"min":{"1min":0.32800000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67100000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.52900000000000003,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70499999999999996,"5min":0.755,"15min":0.755},"last":0.503}]},{"osd":1,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54400000000000004,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69499999999999995,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55400000000000005,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68300000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.52800000000000002,"15min":0.54400000000000004},"min":{"1min":0.32700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63100000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.378,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66200000000000003,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.52200000000000002}]},{"osd":3,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52900000000000003,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71699999999999997,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54300000000000004,"15min":0.55200000000000005},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72799999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.438}]},{"osd":4,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42699999999999999,"15min":0.42799999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70399999999999996,"5min":0.751,"15min":0.751},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.432,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71499999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.377}]},{"osd":5,"last update":"Sat Sep 14 03:44:06 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.44400000000000001,"15min":0.44},"min":{"1min":0.30399999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67500000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45600000000000002,"15min":0.44600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67200000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58599999999999997}]},{"osd":7,"last update":"Sat Sep 14 03:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66200000000000003,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53000000000000003,"15min":0.53300000000000003},"min":{"1min":0.29499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.73099999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.61199999999999999}]}]},{"osd":5,"up_from":36,"seq":154618822701,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297996,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447156,"statfs":{"total":95995035648,"available":95689887744,"internally_reserved":0,"allocated":655360,"data_stored":286813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.38700000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70199999999999996,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53000000000000003,"15min":0.53300000000000003},"min":{"1min":0.38600000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.51800000000000002}]},{"osd":1,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39700000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67000000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.51400000000000001},"min":{"1min":0.42099999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.47299999999999998}]},{"osd":3,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.67500000000000004,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.436,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.50700000000000001}]},{"osd":4,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.41599999999999998,"15min":0.40799999999999997},"min":{"1min":0.31,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60599999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42499999999999999,"15min":0.40200000000000002},"min":{"1min":0.27800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.624,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.34000000000000002}]},{"osd":6,"last update":"Sat Sep 14 03:44:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.45800000000000002,"15min":0.438},"min":{"1min":0.32500000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64300000000000002,"5min":0.748,"15min":0.748},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46800000000000003,"15min":0.45200000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.75,"5min":0.755,"15min":0.755},"last":0.39600000000000002}]},{"osd":7,"last update":"Sat Sep 14 03:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.495,"15min":0.48899999999999999},"min":{"1min":0.33800000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79500000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.49199999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018929,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298016,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447136,"statfs":{"total":95995035648,"available":95689867264,"internally_reserved":0,"allocated":671744,"data_stored":288167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66500000000000004,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.63},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.505,"15min":0.51600000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.67300000000000004,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.68799999999999994}]},{"osd":1,"last update":"Sat Sep 14 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52900000000000003,"15min":0.53600000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69699999999999995,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.52700000000000002,"15min":0.54100000000000004},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65000000000000002,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Sep 14 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52100000000000002,"15min":0.53500000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.60499999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53700000000000003,"15min":0.55200000000000005},"min":{"1min":0.33500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67500000000000004,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.65700000000000003}]},{"osd":3,"last update":"Sat Sep 14 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56899999999999995,"15min":0.58499999999999996},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70699999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.73299999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.72399999999999998}]},{"osd":5,"last update":"Sat Sep 14 03:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.34300000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.35599999999999998,"15min":0.33600000000000002},"min":{"1min":0.25800000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.55100000000000005}]},{"osd":6,"last update":"Sat Sep 14 03:44:01 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42099999999999999,"15min":0.40899999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.61899999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.502},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.39800000000000002,"15min":0.38100000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Sep 14 03:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.42599999999999999,"15min":0.42099999999999999},"min":{"1min":0.32700000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74299999999999999,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.57699999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297996,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447156,"statfs":{"total":95995035648,"available":95689887744,"internally_reserved":0,"allocated":655360,"data_stored":286874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.35899999999999999,"15min":0.36899999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47399999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.38600000000000001,"15min":0.39700000000000002},"min":{"1min":0.221,"5min":0.219,"15min":0.219},"max":{"1min":0.49299999999999999,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.39400000000000002}]},{"osd":1,"last update":"Sat Sep 14 03:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.38100000000000001,"15min":0.39700000000000002},"min":{"1min":0.23599999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.40400000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.437},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39800000000000002,"15min":0.42199999999999999},"min":{"1min":0.252,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Sep 14 03:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.41499999999999998,"15min":0.437},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.379,"15min":0.39000000000000001},"min":{"1min":0.24399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.46300000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.41499999999999998}]},{"osd":4,"last update":"Sat Sep 14 03:43:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52200000000000002,"15min":0.51600000000000001},"min":{"1min":0.36199999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73899999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.38600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.75600000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.65900000000000003}]},{"osd":5,"last update":"Sat Sep 14 03:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.41499999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.69399999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.59599999999999997,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.70599999999999996,"5min":0.876,"15min":0.876},"last":0.79000000000000004}]},{"osd":6,"last update":"Sat Sep 14 03:44:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.41699999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.754,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.754}]},{"osd":7,"last update":"Sat Sep 14 03:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75800000000000001,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78400000000000003,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.63800000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297996,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447156,"statfs":{"total":95995035648,"available":95689887744,"internally_reserved":0,"allocated":655360,"data_stored":287073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.35799999999999998,"15min":0.36399999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.55000000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.371,"15min":0.36399999999999999},"min":{"1min":0.224,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.51100000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.32800000000000001}]},{"osd":1,"last update":"Sat Sep 14 03:43:39 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.376,"15min":0.38200000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.38900000000000001,"15min":0.39000000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.50800000000000001,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.27900000000000003}]},{"osd":3,"last update":"Sat Sep 14 03:44:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.39900000000000002,"15min":0.40899999999999997},"min":{"1min":0.26400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.47799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.41299999999999998,"15min":0.42899999999999999},"min":{"1min":0.30499999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.49199999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.34999999999999998}]},{"osd":4,"last update":"Sat Sep 14 03:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.55600000000000005,"15min":0.55000000000000004},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.58799999999999997,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81999999999999995,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.58599999999999997}]},{"osd":5,"last update":"Sat Sep 14 03:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73199999999999998,"5min":1,"15min":1},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55100000000000005,"15min":0.53700000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79100000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.61099999999999999}]},{"osd":6,"last update":"Sat Sep 14 03:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.42699999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.68899999999999995,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.55700000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51900000000000002}]},{"osd":7,"last update":"Sat Sep 14 03:43:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.60899999999999999,"15min":0.60599999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89300000000000002,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.624}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297996,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447156,"statfs":{"total":95995035648,"available":95689887744,"internally_reserved":0,"allocated":655360,"data_stored":286803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 14 03:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34000000000000002,"15min":0.33900000000000002},"min":{"1min":0.23300000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.51600000000000001,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.35699999999999998,"15min":0.34300000000000003},"min":{"1min":0.247,"5min":0.224,"15min":0.224},"max":{"1min":0.56699999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Sat Sep 14 03:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.35499999999999998,"15min":0.35099999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.45800000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.35899999999999999,"15min":0.35499999999999998},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.41199999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.39200000000000002}]},{"osd":3,"last update":"Sat Sep 14 03:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38300000000000001,"15min":0.39600000000000002},"min":{"1min":0.28100000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.47599999999999998,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.38700000000000001,"15min":0.40400000000000003},"min":{"1min":0.27000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.505,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.374}]},{"osd":4,"last update":"Sat Sep 14 03:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69099999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54800000000000004,"15min":0.55400000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69299999999999995,"5min":0.872,"15min":0.872},"last":0.59699999999999998}]},{"osd":5,"last update":"Sat Sep 14 03:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.53000000000000003,"15min":0.53400000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.64300000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.71899999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Sep 14 03:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53900000000000003,"15min":0.54200000000000004},"min":{"1min":0.33100000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.443},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.39000000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72699999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Sep 14 03:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.58099999999999996,"15min":0.59299999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71599999999999997,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.5},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57699999999999996,"15min":0.58899999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71499999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298576,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446576,"statfs":{"total":95995035648,"available":95689293824,"internally_reserved":0,"allocated":1249280,"data_stored":877429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 14 03:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34200000000000003,"15min":0.32800000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.59599999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36099999999999999,"15min":0.32500000000000001},"min":{"1min":0.224,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Sep 14 03:43:37 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.36599999999999999,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.53000000000000003,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.501,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Sep 14 03:43:56 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39100000000000001,"15min":0.40300000000000002},"min":{"1min":0.27100000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.48399999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41799999999999998,"15min":0.42499999999999999},"min":{"1min":0.26800000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51300000000000001,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Sep 14 03:43:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56599999999999995,"15min":0.56499999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.75600000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.40100000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60799999999999998}]},{"osd":5,"last update":"Sat Sep 14 03:43:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41399999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69699999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.40699999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76400000000000001,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.61799999999999999}]},{"osd":6,"last update":"Sat Sep 14 03:44:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.41899999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.69499999999999995}]},{"osd":7,"last update":"Sat Sep 14 03:43:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.74099999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T03:44:13.646 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-14T03:44:13.646 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-14T03:44:13.646 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-14T03:44:13.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-14T03:44:13.886 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:14.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:14 smithi114 ceph-mon[40961]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:44:14.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:14 smithi114 ceph-mon[40961]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:14 smithi114 ceph-mon[29589]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:44:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:14 smithi114 ceph-mon[29589]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:14 smithi160 ceph-mon[32702]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T03:44:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:14 smithi160 ceph-mon[32702]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:15.758 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T03:44:15.758 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-14T03:44:16.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:16 smithi114 ceph-mon[29589]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:16 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3939832653' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-14T03:44:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:16 smithi114 ceph-mon[40961]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:16 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3939832653' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-14T03:44:16.519 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-14T03:44:16.520 INFO:teuthology.run_tasks:Running task print... 2024-09-14T03:44:16.535 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-09-14T03:44:16.535 INFO:teuthology.run_tasks:Running task parallel... 2024-09-14T03:44:16.566 INFO:teuthology.task.parallel:starting parallel... 2024-09-14T03:44:16.567 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-14T03:44:16.567 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-14T03:44:16.568 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-14T03:44:16.570 INFO:tasks.radosbench:Beginning radosbench... 2024-09-14T03:44:16.570 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-14T03:44:16.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-14T03:44:16.571 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-14T03:44:16.571 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-14T03:44:16.572 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-14T03:44:16.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-09-14T03:44:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:16 smithi160 ceph-mon[32702]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:16 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3939832653' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-14T03:44:16.836 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:16.842 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:18.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:18 smithi114 ceph-mon[40961]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:18.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:18 smithi114 ceph-mon[29589]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:18 smithi160 ceph-mon[32702]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:19.315 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_0' created 2024-09-14T03:44:19.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:19 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/881115556' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-14T03:44:19.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:19 smithi114 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-14T03:44:19.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:19 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/881115556' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-14T03:44:19.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:19 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-14T03:44:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:19 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/881115556' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-14T03:44:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:19 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-14T03:44:19.750 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:19] "GET /metrics HTTP/1.1" 200 239652 "" "Prometheus/2.43.0" 2024-09-14T03:44:20.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:44:20] ENGINE Bus STOPPING 2024-09-14T03:44:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:20 smithi114 ceph-mon[29589]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:20 smithi114 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-14T03:44:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:20 smithi114 ceph-mon[29589]: osdmap e65: 8 total, 8 up, 8 in 2024-09-14T03:44:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:20 smithi114 ceph-mon[40961]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:20 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-14T03:44:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:20 smithi114 ceph-mon[40961]: osdmap e65: 8 total, 8 up, 8 in 2024-09-14T03:44:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:20 smithi160 ceph-mon[32702]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:20 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-14T03:44:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:20 smithi160 ceph-mon[32702]: osdmap e65: 8 total, 8 up, 8 in 2024-09-14T03:44:20.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-14T03:44:21.029 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:44:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-14T03:44:21.029 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:44:20] ENGINE Bus STOPPED 2024-09-14T03:44:21.030 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:44:20] ENGINE Bus STARTING 2024-09-14T03:44:21.322 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:44:21] ENGINE Serving on http://:::9283 2024-09-14T03:44:21.322 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:44:21] ENGINE Bus STARTED 2024-09-14T03:44:21.502 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:21.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-14T03:44:21.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[29589]: osdmap e66: 8 total, 8 up, 8 in 2024-09-14T03:44:21.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2054010134' entity='client.admin' 2024-09-14T03:44:21.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:44:21.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:44:21.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:44:21.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:44:21.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:21.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[40961]: osdmap e66: 8 total, 8 up, 8 in 2024-09-14T03:44:21.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2054010134' entity='client.admin' 2024-09-14T03:44:21.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:44:21.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:44:21.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:44:21.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:44:21.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:21 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:21.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:44:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:44:21.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:21 smithi160 ceph-mon[32702]: osdmap e66: 8 total, 8 up, 8 in 2024-09-14T03:44:21.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:21 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2054010134' entity='client.admin' 2024-09-14T03:44:21.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:21 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:44:21.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:21 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:44:21.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:21 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:44:21.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:21 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:44:21.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:21 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:22.244 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:22.595 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:22 smithi114 ceph-mon[40961]: pgmap v93: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:22.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:22 smithi114 ceph-mon[40961]: osdmap e67: 8 total, 8 up, 8 in 2024-09-14T03:44:22.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:22 smithi114 ceph-mon[40961]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:44:22.596 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:22 smithi114 ceph-mon[29589]: pgmap v93: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:22.596 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:22 smithi114 ceph-mon[29589]: osdmap e67: 8 total, 8 up, 8 in 2024-09-14T03:44:22.596 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:22 smithi114 ceph-mon[29589]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:44:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:22 smithi160 ceph-mon[32702]: pgmap v93: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:22 smithi160 ceph-mon[32702]: osdmap e67: 8 total, 8 up, 8 in 2024-09-14T03:44:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:22 smithi160 ceph-mon[32702]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:44:24.334 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-14T03:44:24.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:24 smithi114 ceph-mon[29589]: pgmap v95: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:24.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:24 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3895002372' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:44:24.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:24 smithi114 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:44:24.449 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:24 smithi114 ceph-mon[40961]: pgmap v95: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:24.449 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:24 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3895002372' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:44:24.449 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:24 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:44:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:24 smithi160 ceph-mon[32702]: pgmap v95: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:24 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3895002372' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:44:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:24 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:44:25.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:25 smithi114 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:44:25.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:25 smithi114 ceph-mon[29589]: osdmap e68: 8 total, 8 up, 8 in 2024-09-14T03:44:25.447 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:25 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:44:25.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:25 smithi114 ceph-mon[40961]: osdmap e68: 8 total, 8 up, 8 in 2024-09-14T03:44:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:25 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:44:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:25 smithi160 ceph-mon[32702]: osdmap e68: 8 total, 8 up, 8 in 2024-09-14T03:44:26.225 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-14T03:44:26.228 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-14T03:44:26.228 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-14T03:44:26.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-14T03:44:26.300 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-14T03:44:26.300 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-14T03:44:26.300 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_85940 2024-09-14T03:44:26.300 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:44:26.300 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-14T03:44:26.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:26 smithi114 ceph-mon[29589]: pgmap v97: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:26.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:26 smithi114 ceph-mon[29589]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:44:26.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:26 smithi114 ceph-mon[29589]: Cluster is now healthy 2024-09-14T03:44:26.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:26 smithi114 ceph-mon[40961]: pgmap v97: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:26.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:26 smithi114 ceph-mon[40961]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:44:26.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:26 smithi114 ceph-mon[40961]: Cluster is now healthy 2024-09-14T03:44:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:26 smithi160 ceph-mon[32702]: pgmap v97: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:44:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:26 smithi160 ceph-mon[32702]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:44:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:26 smithi160 ceph-mon[32702]: Cluster is now healthy 2024-09-14T03:44:26.741 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:27.300 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 15 4252 4237 264.798 264.812 0.00220542 0.00376781 2024-09-14T03:44:28.300 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 8467 8451 264.075 263.375 0.00553694 0.00378022 2024-09-14T03:44:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:28 smithi114 ceph-mon[40961]: pgmap v98: 89 pgs: 89 active+clean; 236 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s wr, 547 op/s 2024-09-14T03:44:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:28 smithi114 ceph-mon[29589]: pgmap v98: 89 pgs: 89 active+clean; 236 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s wr, 547 op/s 2024-09-14T03:44:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:28 smithi160 ceph-mon[32702]: pgmap v98: 89 pgs: 89 active+clean; 236 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s wr, 547 op/s 2024-09-14T03:44:29.300 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 12118 12102 252.106 228.188 0.00507701 0.00396022 2024-09-14T03:44:29.332 INFO:teuthology.orchestra.run.smithi114.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:44:29.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:29] "GET /metrics HTTP/1.1" 200 241925 "" "Prometheus/2.43.0" 2024-09-14T03:44:30.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch ps' 2024-09-14T03:44:30.301 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 15435 15419 240.851 207.312 0.00460529 0.00414605 2024-09-14T03:44:30.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[29589]: pgmap v99: 89 pgs: 89 active+clean; 236 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s wr, 471 op/s 2024-09-14T03:44:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[29589]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a2782315255a3e7d5f6cdc9eb35785187ff6ddf8", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[29589]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:44:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:44:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:44:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:44:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:44:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:44:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[29589]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:44:30.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[40961]: pgmap v99: 89 pgs: 89 active+clean; 236 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s wr, 471 op/s 2024-09-14T03:44:30.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[40961]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a2782315255a3e7d5f6cdc9eb35785187ff6ddf8", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:30.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[40961]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:44:30.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:44:30.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:44:30.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:44:30.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:44:30.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:44:30.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:30 smithi114 ceph-mon[40961]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:44:30.623 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:30 smithi160 ceph-mon[32702]: pgmap v99: 89 pgs: 89 active+clean; 236 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s wr, 471 op/s 2024-09-14T03:44:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:30 smithi160 ceph-mon[32702]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a2782315255a3e7d5f6cdc9eb35785187ff6ddf8", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:30 smithi160 ceph-mon[32702]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:44:30.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:30 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:44:30.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:30 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:44:30.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:30 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:44:30.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:30 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:44:30.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:30 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:44:30.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:30 smithi160 ceph-mon[32702]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 2024-09-14T03:44:31.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 18413 18397 229.878 186.125 0.010064 0.00434461 2024-09-14T03:44:31.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:44:31 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:44:32.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 21201 21185 220.606 174.25 0.0064571 0.00452829 2024-09-14T03:44:32.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:32 smithi114 ceph-mon[29589]: pgmap v100: 89 pgs: 89 active+clean; 236 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s wr, 381 op/s 2024-09-14T03:44:32.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:32 smithi114 ceph-mon[40961]: pgmap v100: 89 pgs: 89 active+clean; 236 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s wr, 381 op/s 2024-09-14T03:44:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:32 smithi160 ceph-mon[32702]: pgmap v100: 89 pgs: 89 active+clean; 236 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s wr, 381 op/s 2024-09-14T03:44:33.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 23913 23897 213.304 169.5 0.00224726 0.00468271 2024-09-14T03:44:34.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 26279 26263 205.125 147.875 0.00417846 0.00486901 2024-09-14T03:44:34.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:34 smithi114 ceph-mon[40961]: pgmap v101: 89 pgs: 89 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 130 MiB/s wr, 2.09k op/s 2024-09-14T03:44:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:34 smithi114 ceph-mon[29589]: pgmap v101: 89 pgs: 89 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 130 MiB/s wr, 2.09k op/s 2024-09-14T03:44:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:34 smithi160 ceph-mon[32702]: pgmap v101: 89 pgs: 89 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 130 MiB/s wr, 2.09k op/s 2024-09-14T03:44:34.832 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.a smithi114 *:9093,9094 running (90s) 45s ago 2m 16.1M - 0.25.0 c8568f914cd2 80f53fb9f598 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.a smithi160 *:3000 running (2m) 62s ago 2m 85.2M - 9.4.7 954c08fa6188 cb3fc2419525 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi114.ebmrat smithi114 running (75s) 45s ago 75s 15.4M - 17.2.7-1629-geb16b97b ab51d1bebd12 00f4a6ea534e 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi160.ywyjae smithi160 running (73s) 62s ago 72s 12.3M - 17.2.7-1629-geb16b97b ab51d1bebd12 a87b5ff7e907 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x smithi160 *:8443 running (6m) 62s ago 6m 399M - 17.2.7-1629-geb16b97b ab51d1bebd12 ad77a9b0ace6 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y smithi114 *:9283,8765 running (7m) 45s ago 7m 443M - 17.2.7-1629-geb16b97b ab51d1bebd12 57c15055b7ea 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a smithi114 running (7m) 45s ago 8m 46.6M 2048M 17.2.7-1629-geb16b97b ab51d1bebd12 92df6a472c26 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b smithi160 running (6m) 62s ago 6m 36.4M 2048M 17.2.7-1629-geb16b97b ab51d1bebd12 a30277b5b152 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c smithi114 running (6m) 45s ago 6m 37.1M 2048M 17.2.7-1629-geb16b97b ab51d1bebd12 f8446e48bf23 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.a smithi114 *:9100 running (2m) 45s ago 119s 16.2M - 1.5.0 0da6a335fe13 40522b00d68c 2024-09-14T03:44:34.833 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.b smithi160 *:9100 running (116s) 62s ago 115s 8346k - 1.5.0 0da6a335fe13 a97cccb21ef2 2024-09-14T03:44:34.834 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (6m) 45s ago 6m 44.8M 4016M 17.2.7-1629-geb16b97b ab51d1bebd12 ddb654f06efc 2024-09-14T03:44:34.834 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (5m) 45s ago 5m 42.5M 4016M 17.2.7-1629-geb16b97b ab51d1bebd12 a709e75520db 2024-09-14T03:44:34.834 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (5m) 45s ago 5m 40.2M 4016M 17.2.7-1629-geb16b97b ab51d1bebd12 a76e6ee3bc54 2024-09-14T03:44:34.834 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi114 running (4m) 45s ago 4m 41.7M 4016M 17.2.7-1629-geb16b97b ab51d1bebd12 b43e985ecf6c 2024-09-14T03:44:34.834 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi160 running (4m) 62s ago 4m 40.3M 4016M 17.2.7-1629-geb16b97b ab51d1bebd12 e89693bb3b76 2024-09-14T03:44:34.834 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi160 running (4m) 62s ago 4m 40.3M 4016M 17.2.7-1629-geb16b97b ab51d1bebd12 fb664f7f461a 2024-09-14T03:44:34.834 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 smithi160 running (3m) 62s ago 3m 40.5M 4016M 17.2.7-1629-geb16b97b ab51d1bebd12 0403446dedd1 2024-09-14T03:44:34.834 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 smithi160 running (3m) 62s ago 3m 41.6M 4016M 17.2.7-1629-geb16b97b ab51d1bebd12 39f0a617d104 2024-09-14T03:44:34.834 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.a smithi160 *:9095 running (85s) 62s ago 2m 33.2M - 2.43.0 a07b618ecd1d d05492e18223 2024-09-14T03:44:35.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 28811 28795 199.916 158.25 0.00633066 0.00499727 2024-09-14T03:44:35.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:35 smithi114 ceph-mon[40961]: from='client.24740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:35 smithi114 ceph-mon[29589]: from='client.24740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:35 smithi160 ceph-mon[32702]: from='client.24740 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:35.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'echo wait for minority of mons to upgrade' 2024-09-14T03:44:36.132 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:36.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 31264 31248 195.238 153.312 0.00676747 0.00511693 2024-09-14T03:44:36.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:36 smithi114 ceph-mon[40961]: pgmap v102: 89 pgs: 89 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-14T03:44:36.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:36 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:36.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:36 smithi114 ceph-mon[29589]: pgmap v102: 89 pgs: 89 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-14T03:44:36.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:36 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:36 smithi160 ceph-mon[32702]: pgmap v102: 89 pgs: 89 active+clean; 1.3 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-14T03:44:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:36 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:37.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 33401 33385 189.631 133.562 0.00898816 0.00526872 2024-09-14T03:44:38.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 35388 35372 184.178 124.188 0.00546036 0.0054247 2024-09-14T03:44:38.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:38 smithi114 ceph-mon[40961]: pgmap v103: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 173 MiB/s wr, 2.76k op/s 2024-09-14T03:44:38.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:38 smithi114 ceph-mon[29589]: pgmap v103: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 173 MiB/s wr, 2.76k op/s 2024-09-14T03:44:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:38 smithi160 ceph-mon[32702]: pgmap v103: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 173 MiB/s wr, 2.76k op/s 2024-09-14T03:44:39.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 37504 37488 180.183 132.25 0.00199117 0.00553808 2024-09-14T03:44:39.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:39] "GET /metrics HTTP/1.1" 200 242828 "" "Prometheus/2.43.0" 2024-09-14T03:44:40.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 39453 39437 176.013 121.812 0.00187208 0.00565574 2024-09-14T03:44:40.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:40 smithi114 ceph-mon[29589]: pgmap v104: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-14T03:44:40.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:40 smithi114 ceph-mon[40961]: pgmap v104: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-14T03:44:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:40 smithi160 ceph-mon[32702]: pgmap v104: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-14T03:44:41.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 41551 41535 173.02 131.125 0.00590464 0.00577455 2024-09-14T03:44:41.665 INFO:teuthology.orchestra.run.smithi114.stdout:wait for minority of mons to upgrade 2024-09-14T03:44:41.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:44:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:44:42.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 43692 43676 170.569 133.812 0.00454181 0.00585809 2024-09-14T03:44:42.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-14T03:44:42.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:42 smithi114 ceph-mon[40961]: pgmap v105: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-14T03:44:42.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:42 smithi114 ceph-mon[29589]: pgmap v105: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-14T03:44:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:42 smithi160 ceph-mon[32702]: pgmap v105: 89 pgs: 89 active+clean; 2.0 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-14T03:44:42.948 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:44:43.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 45663 45647 167.777 123.188 0.0133392 0.00595527 2024-09-14T03:44:44.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 47640 47624 165.321 123.562 0.00506111 0.00604382 2024-09-14T03:44:44.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:44 smithi114 ceph-mon[40961]: pgmap v106: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 207 MiB/s wr, 3.31k op/s 2024-09-14T03:44:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:44 smithi114 ceph-mon[29589]: pgmap v106: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 207 MiB/s wr, 3.31k op/s 2024-09-14T03:44:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:44 smithi160 ceph-mon[32702]: pgmap v106: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 207 MiB/s wr, 3.31k op/s 2024-09-14T03:44:45.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 49545 49529 162.886 119.062 0.00359502 0.00613454 2024-09-14T03:44:45.999 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:44:46.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:44:46.304692+0000 min lat: 0.0012048 max lat: 0.0982719 avg lat: 0.006238 2024-09-14T03:44:46.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:44:46.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 51284 51268 160.176 108.688 0.00718867 0.006238 2024-09-14T03:44:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:46 smithi160 ceph-mon[32702]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-14T03:44:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:46 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2029957481' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:46.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:46 smithi114 ceph-mon[40961]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-14T03:44:46.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:46 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2029957481' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:46.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:46 smithi114 ceph-mon[29589]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-14T03:44:46.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:46 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2029957481' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:47.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 53062 53046 157.84 111.125 0.00302165 0.00633018 2024-09-14T03:44:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:47 smithi160 ceph-mon[32702]: from='client.24752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:47 smithi160 ceph-mon[32702]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:47.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:47 smithi114 ceph-mon[40961]: from='client.24752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:47.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:47 smithi114 ceph-mon[40961]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:47 smithi114 ceph-mon[29589]: from='client.24752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:47 smithi114 ceph-mon[29589]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:48.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 54911 54895 155.918 115.562 0.00939199 0.00640898 2024-09-14T03:44:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:48 smithi160 ceph-mon[32702]: pgmap v108: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T03:44:48.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:48 smithi114 ceph-mon[40961]: pgmap v108: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T03:44:48.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:48 smithi114 ceph-mon[29589]: pgmap v108: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T03:44:49.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 56571 56555 153.65 103.75 0.00804675 0.00650284 2024-09-14T03:44:49.609 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:44:49.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:49] "GET /metrics HTTP/1.1" 200 242828 "" "Prometheus/2.43.0" 2024-09-14T03:44:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:49 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2164889016' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:49.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:49 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2164889016' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:49 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2164889016' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:50.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 15 58168 58153 151.409 99.875 0.00830662 0.00660005 2024-09-14T03:44:50.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:50 smithi114 ceph-mon[29589]: pgmap v109: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-14T03:44:50.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:50 smithi114 ceph-mon[29589]: from='client.24767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:50.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:50 smithi114 ceph-mon[40961]: pgmap v109: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-14T03:44:50.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:50 smithi114 ceph-mon[40961]: from='client.24767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:50.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:50 smithi160 ceph-mon[32702]: pgmap v109: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-14T03:44:50.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:50 smithi160 ceph-mon[32702]: from='client.24767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:51.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 59911 59895 149.707 108.875 0.00731233 0.00667472 2024-09-14T03:44:51.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:51 smithi114 ceph-mon[40961]: from='client.24773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:51.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:51.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:51 smithi114 ceph-mon[29589]: from='client.24773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:51.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:51.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:44:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:44:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:51 smithi160 ceph-mon[32702]: from='client.24773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:44:52.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 15 61588 61573 147.982 104.875 0.0146144 0.0067518 2024-09-14T03:44:52.771 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:52 smithi114 ceph-mon[40961]: pgmap v110: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-14T03:44:52.771 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:52 smithi114 ceph-mon[29589]: pgmap v110: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-14T03:44:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:52 smithi160 ceph-mon[32702]: pgmap v110: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-14T03:44:53.206 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:44:53.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 63051 63035 145.886 91.375 0.0155348 0.00684949 2024-09-14T03:44:53.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:53 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1573237830' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:53 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1573237830' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:53 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1573237830' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:54.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 15 64503 64488 143.919 90.8125 0.00848436 0.00694361 2024-09-14T03:44:54.535 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:54 smithi114 ceph-mon[40961]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:54.535 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:54 smithi114 ceph-mon[40961]: pgmap v111: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 147 MiB/s wr, 2.36k op/s 2024-09-14T03:44:54.535 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:54 smithi114 ceph-mon[40961]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:54.536 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:54 smithi114 ceph-mon[29589]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:54.536 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:54 smithi114 ceph-mon[29589]: pgmap v111: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 147 MiB/s wr, 2.36k op/s 2024-09-14T03:44:54.536 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:54 smithi114 ceph-mon[29589]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:54 smithi160 ceph-mon[32702]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:54 smithi160 ceph-mon[32702]: pgmap v111: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 147 MiB/s wr, 2.36k op/s 2024-09-14T03:44:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:54 smithi160 ceph-mon[32702]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:55.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 65904 65888 141.973 87.5 0.00695791 0.00703834 2024-09-14T03:44:56.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 15 67276 67261 140.101 85.8125 0.00686148 0.00713256 2024-09-14T03:44:56.829 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:56 smithi114 ceph-mon[29589]: pgmap v112: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-14T03:44:56.829 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:56 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/935852189' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:56.830 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:44:56.830 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:56 smithi114 ceph-mon[40961]: pgmap v112: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-14T03:44:56.830 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:56 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/935852189' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:56 smithi160 ceph-mon[32702]: pgmap v112: 89 pgs: 89 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-14T03:44:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:56 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/935852189' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:44:57.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 68640 68624 138.33 85.1875 0.0241554 0.00722453 2024-09-14T03:44:57.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:57 smithi114 ceph-mon[40961]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:57.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:57 smithi114 ceph-mon[29589]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:57 smithi160 ceph-mon[32702]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:58.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 70151 70135 136.958 94.4375 0.0115857 0.00729731 2024-09-14T03:44:58.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:58 smithi114 ceph-mon[29589]: pgmap v113: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-14T03:44:58.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:44:58 smithi114 ceph-mon[29589]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:58.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:58 smithi114 ceph-mon[40961]: pgmap v113: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-14T03:44:58.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:44:58 smithi114 ceph-mon[40961]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:58.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:58 smithi160 ceph-mon[32702]: pgmap v113: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-14T03:44:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:44:58 smithi160 ceph-mon[32702]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:44:59.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 71659 71643 135.66 94.25 0.00361065 0.0073662 2024-09-14T03:44:59.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:44:59 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:44:59] "GET /metrics HTTP/1.1" 200 243055 "" "Prometheus/2.43.0" 2024-09-14T03:45:00.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 72995 72979 134.126 83.5 0.0119883 0.00745081 2024-09-14T03:45:00.425 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:00.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:00 smithi114 ceph-mon[40961]: pgmap v114: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-14T03:45:00.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:00 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/243464619' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:00.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:00 smithi114 ceph-mon[29589]: pgmap v114: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-14T03:45:00.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:00 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/243464619' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:00.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:00 smithi160 ceph-mon[32702]: pgmap v114: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-14T03:45:00.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:00 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/243464619' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:01.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 74328 74312 132.671 83.3125 0.00715 0.007532 2024-09-14T03:45:01.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:01 smithi114 ceph-mon[29589]: from='client.24631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:01.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:01 smithi114 ceph-mon[40961]: from='client.24631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:01.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:01 smithi160 ceph-mon[32702]: from='client.24631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:01.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:45:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:45:02.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 75749 75733 131.452 88.8125 0.0114658 0.00760282 2024-09-14T03:45:02.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:02 smithi114 ceph-mon[29589]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:02.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:02 smithi114 ceph-mon[29589]: pgmap v115: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-14T03:45:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:02 smithi114 ceph-mon[40961]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:02 smithi114 ceph-mon[40961]: pgmap v115: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-14T03:45:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:02 smithi160 ceph-mon[32702]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:02 smithi160 ceph-mon[32702]: pgmap v115: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-14T03:45:03.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 77114 77098 130.205 85.3125 0.00314238 0.00766766 2024-09-14T03:45:03.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:03 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/4067375184' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:03.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:03 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/4067375184' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:03.907 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:03.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:03 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/4067375184' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:04.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 78405 78389 128.901 80.6875 0.00911565 0.00775242 2024-09-14T03:45:04.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:04 smithi114 ceph-mon[29589]: pgmap v116: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-14T03:45:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:04 smithi114 ceph-mon[40961]: pgmap v116: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-14T03:45:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:04 smithi160 ceph-mon[32702]: pgmap v116: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-14T03:45:05.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 79707 79691 127.682 81.375 0.0180103 0.00782711 2024-09-14T03:45:05.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:05 smithi114 ceph-mon[29589]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:05.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:05 smithi114 ceph-mon[29589]: from='client.24640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:05 smithi114 ceph-mon[40961]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:05 smithi114 ceph-mon[40961]: from='client.24640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:05 smithi160 ceph-mon[32702]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:05 smithi160 ceph-mon[32702]: from='client.24640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:06.310 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:45:06.308591+0000 min lat: 0.0012048 max lat: 0.0982719 avg lat: 0.00789985 2024-09-14T03:45:06.310 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:45:06.310 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 81000 80984 126.511 80.8125 0.0175311 0.00789985 2024-09-14T03:45:06.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:06 smithi114 ceph-mon[40961]: pgmap v117: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-14T03:45:06.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:06 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:06.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:06 smithi114 ceph-mon[29589]: pgmap v117: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-14T03:45:06.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:06 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:06 smithi160 ceph-mon[32702]: pgmap v117: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-14T03:45:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:06 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:07.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 82162 82146 125.196 72.625 0.0242893 0.00798202 2024-09-14T03:45:07.517 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:07.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:07 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3202956213' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:07.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:07 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3202956213' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:07 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3202956213' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:08.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 83348 83332 123.98 74.125 0.0124905 0.00806113 2024-09-14T03:45:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:08 smithi114 ceph-mon[29589]: pgmap v118: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-14T03:45:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:08 smithi114 ceph-mon[29589]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:08 smithi114 ceph-mon[40961]: pgmap v118: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-14T03:45:08.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:08 smithi114 ceph-mon[40961]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:08 smithi160 ceph-mon[32702]: pgmap v118: 89 pgs: 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-14T03:45:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:08 smithi160 ceph-mon[32702]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:09.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 84634 84618 122.966 80.375 0.0109223 0.00812776 2024-09-14T03:45:09.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:09 smithi114 ceph-mon[40961]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:09.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:09 smithi114 ceph-mon[29589]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:09.873 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:45:09 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:09] "GET /metrics HTTP/1.1" 200 243125 "" "Prometheus/2.43.0" 2024-09-14T03:45:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:09 smithi160 ceph-mon[32702]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:10.311 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 85857 85841 121.904 76.4375 0.0186216 0.00819868 2024-09-14T03:45:10.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:10 smithi114 ceph-mon[40961]: pgmap v119: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T03:45:10.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:10 smithi114 ceph-mon[29589]: pgmap v119: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T03:45:10.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:10 smithi160 ceph-mon[32702]: pgmap v119: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T03:45:11.138 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:11.311 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 86960 86944 120.725 68.9375 0.00926072 0.00827865 2024-09-14T03:45:11.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:11 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/342049880' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:11.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:11 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/342049880' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:11.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:45:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:45:11.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:11 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/342049880' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:12.314 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 88115 88099 119.664 72.1875 0.0140812 0.00835209 2024-09-14T03:45:12.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:12 smithi114 ceph-mon[40961]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:12.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:12 smithi114 ceph-mon[40961]: pgmap v120: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-14T03:45:12.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:12 smithi114 ceph-mon[40961]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:12.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:12 smithi114 ceph-mon[29589]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:12.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:12 smithi114 ceph-mon[29589]: pgmap v120: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-14T03:45:12.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:12 smithi114 ceph-mon[29589]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:12.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:12 smithi160 ceph-mon[32702]: from='client.24667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:12.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:12 smithi160 ceph-mon[32702]: pgmap v120: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-14T03:45:12.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:12 smithi160 ceph-mon[32702]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:13.315 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 89348 89332 118.755 77.0625 0.0141429 0.00841538 2024-09-14T03:45:14.315 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 90492 90476 117.771 71.5 0.0275391 0.008486 2024-09-14T03:45:14.761 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:14.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:14 smithi114 ceph-mon[40961]: pgmap v121: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-14T03:45:14.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:14 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3958002785' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:14.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:14 smithi114 ceph-mon[29589]: pgmap v121: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-14T03:45:14.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:14 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3958002785' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:14 smithi160 ceph-mon[32702]: pgmap v121: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-14T03:45:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:14 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3958002785' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:15.315 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 91532 91516 116.694 65 0.0106484 0.00856463 2024-09-14T03:45:15.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:15 smithi114 ceph-mon[29589]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:15 smithi114 ceph-mon[40961]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:15 smithi160 ceph-mon[32702]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:16.315 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 92803 92787 115.949 79.4375 0.0167998 0.00861975 2024-09-14T03:45:16.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:16 smithi114 ceph-mon[40961]: pgmap v122: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-14T03:45:16.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:16 smithi114 ceph-mon[40961]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:16 smithi114 ceph-mon[29589]: pgmap v122: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-14T03:45:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:16 smithi114 ceph-mon[29589]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:16 smithi160 ceph-mon[32702]: pgmap v122: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-14T03:45:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:16 smithi160 ceph-mon[32702]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:17.315 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 94073 94057 115.232 79.375 0.0152235 0.00867338 2024-09-14T03:45:18.315 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 95290 95274 114.478 76.0625 0.0104733 0.00873013 2024-09-14T03:45:18.345 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:18 smithi114 ceph-mon[40961]: pgmap v123: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:18 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1132912557' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:18 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:18 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:18 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:45:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:18 smithi114 ceph-mon[29589]: pgmap v123: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T03:45:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:18 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1132912557' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:18 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:45:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:18 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:45:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:18 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:45:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:18 smithi160 ceph-mon[32702]: pgmap v123: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T03:45:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:18 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1132912557' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:18 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:45:18.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:18 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:45:18.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:18 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:45:19.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 96384 96368 113.609 68.375 0.00556659 0.00879685 2024-09-14T03:45:19.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:19] "GET /metrics HTTP/1.1" 200 243125 "" "Prometheus/2.43.0" 2024-09-14T03:45:19.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[40961]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:19.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[40961]: Upgrade: Target is version 19.3.0-4907-ga2782315 (unknown) 2024-09-14T03:45:19.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[40961]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, digests ['quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802'] 2024-09-14T03:45:19.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[40961]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-14T03:45:19.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[40961]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:19.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[40961]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 on smithi160 2024-09-14T03:45:19.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[29589]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:19.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[29589]: Upgrade: Target is version 19.3.0-4907-ga2782315 (unknown) 2024-09-14T03:45:19.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[29589]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, digests ['quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802'] 2024-09-14T03:45:19.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[29589]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-14T03:45:19.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[29589]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:19.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:19 smithi114 ceph-mon[29589]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 on smithi160 2024-09-14T03:45:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:19 smithi160 ceph-mon[32702]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:19 smithi160 ceph-mon[32702]: Upgrade: Target is version 19.3.0-4907-ga2782315 (unknown) 2024-09-14T03:45:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:19 smithi160 ceph-mon[32702]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, digests ['quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802'] 2024-09-14T03:45:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:19 smithi160 ceph-mon[32702]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-14T03:45:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:19 smithi160 ceph-mon[32702]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:19 smithi160 ceph-mon[32702]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 on smithi160 2024-09-14T03:45:20.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 97519 97503 112.818 70.9375 0.00972634 0.00885846 2024-09-14T03:45:20.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:20 smithi160 ceph-mon[32702]: pgmap v124: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T03:45:21.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:20 smithi114 ceph-mon[40961]: pgmap v124: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T03:45:21.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:20 smithi114 ceph-mon[29589]: pgmap v124: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T03:45:21.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 98592 98576 111.987 67.0625 0.00673117 0.00892478 2024-09-14T03:45:21.722 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:21.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:45:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:45:21.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:21 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:21.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:21 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1908362826' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:22.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:21 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:22.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:21 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1908362826' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:22.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:21 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:22.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:21 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1908362826' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:22.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 99653 99637 111.171 66.3125 0.00881319 0.0089897 2024-09-14T03:45:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:22 smithi160 ceph-mon[32702]: pgmap v125: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 62 MiB/s wr, 992 op/s 2024-09-14T03:45:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:22 smithi160 ceph-mon[32702]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:23.032 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:22 smithi114 ceph-mon[40961]: pgmap v125: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 62 MiB/s wr, 992 op/s 2024-09-14T03:45:23.032 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:22 smithi114 ceph-mon[40961]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:23.033 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:22 smithi114 ceph-mon[29589]: pgmap v125: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 62 MiB/s wr, 992 op/s 2024-09-14T03:45:23.033 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:22 smithi114 ceph-mon[29589]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:23.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 100703 100687 110.372 65.625 0.0140911 0.00905507 2024-09-14T03:45:23.712 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:23 smithi114 ceph-mon[40961]: from='client.24890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:23.713 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:23 smithi114 ceph-mon[29589]: from='client.24890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:23 smithi160 ceph-mon[32702]: from='client.24890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:24.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 101739 101723 109.585 64.75 0.0185945 0.00912101 2024-09-14T03:45:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:24 smithi160 ceph-mon[32702]: pgmap v126: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T03:45:25.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:24 smithi114 ceph-mon[40961]: pgmap v126: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T03:45:25.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:24 smithi114 ceph-mon[29589]: pgmap v126: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T03:45:25.148 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:25.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 102674 102658 108.719 58.4375 0.0162801 0.00919308 2024-09-14T03:45:26.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:25 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1459592234' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:26.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:25 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1459592234' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:26.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:25 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1459592234' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:26.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:45:26.316079+0000 min lat: 0.0012048 max lat: 0.122 avg lat: 0.00925358 2024-09-14T03:45:26.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:45:26.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 103723 103707 107.999 65.5625 0.0494045 0.00925358 2024-09-14T03:45:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:26 smithi114 ceph-mon[40961]: from='client.24902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:26 smithi114 ceph-mon[40961]: pgmap v127: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-14T03:45:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:26 smithi114 ceph-mon[40961]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:26 smithi114 ceph-mon[29589]: from='client.24902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:26 smithi114 ceph-mon[29589]: pgmap v127: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-14T03:45:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:26 smithi114 ceph-mon[29589]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:26 smithi160 ceph-mon[32702]: from='client.24902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:26 smithi160 ceph-mon[32702]: pgmap v127: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 963 op/s 2024-09-14T03:45:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:26 smithi160 ceph-mon[32702]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:27.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 104767 104751 107.299 65.25 0.0114662 0.00931467 2024-09-14T03:45:28.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 105769 105753 106.578 62.625 0.0146084 0.00937771 2024-09-14T03:45:28.538 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:28.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:28 smithi114 ceph-mon[40961]: pgmap v128: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-14T03:45:28.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:28 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1763503316' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:28.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:28 smithi114 ceph-mon[29589]: pgmap v128: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-14T03:45:28.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:28 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1763503316' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:28 smithi160 ceph-mon[32702]: pgmap v128: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-14T03:45:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:28 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1763503316' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:29.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 106746 106730 105.856 61.0625 0.0165868 0.00944142 2024-09-14T03:45:29.679 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:45:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:29] "GET /metrics HTTP/1.1" 200 243182 "" "Prometheus/2.43.0" 2024-09-14T03:45:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:29 smithi160 ceph-mon[32702]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:30.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:29 smithi114 ceph-mon[40961]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:30.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:29 smithi114 ceph-mon[29589]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:30.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 107677 107661 105.111 58.1875 0.0197679 0.00950892 2024-09-14T03:45:31.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:30 smithi114 ceph-mon[40961]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:31.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:30 smithi114 ceph-mon[40961]: pgmap v129: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-14T03:45:31.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:30 smithi114 ceph-mon[29589]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:31.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:30 smithi114 ceph-mon[29589]: pgmap v129: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-14T03:45:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:30 smithi160 ceph-mon[32702]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:30 smithi160 ceph-mon[32702]: pgmap v129: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-14T03:45:31.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 108654 108638 104.433 61.0625 0.0144244 0.00957063 2024-09-14T03:45:31.896 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:31.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:31 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3965618030' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:31.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:45:31 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:45:32.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:31 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3965618030' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:32.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:31 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3965618030' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:32.316 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 109636 109620 103.781 61.375 0.0131616 0.00963057 2024-09-14T03:45:33.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:32 smithi114 ceph-mon[40961]: pgmap v130: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 892 op/s 2024-09-14T03:45:33.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:32 smithi114 ceph-mon[29589]: pgmap v130: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 892 op/s 2024-09-14T03:45:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:32 smithi160 ceph-mon[32702]: pgmap v130: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 892 op/s 2024-09-14T03:45:33.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 110614 110598 103.144 61.125 0.00450624 0.00968952 2024-09-14T03:45:34.104 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:33 smithi114 ceph-mon[40961]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:34.104 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:33 smithi114 ceph-mon[40961]: from='client.24748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:34.105 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:33 smithi114 ceph-mon[29589]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:34.105 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:33 smithi114 ceph-mon[29589]: from='client.24748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:33 smithi160 ceph-mon[32702]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:33 smithi160 ceph-mon[32702]: from='client.24748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:34.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 111605 111589 102.538 61.9375 0.0144056 0.00974691 2024-09-14T03:45:34.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:34 smithi114 ceph-mon[29589]: pgmap v131: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-14T03:45:34.963 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:34 smithi114 ceph-mon[40961]: pgmap v131: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-14T03:45:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:34 smithi160 ceph-mon[32702]: pgmap v131: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-14T03:45:35.303 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:35.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 112582 112566 101.937 61.0625 0.0205965 0.00980528 2024-09-14T03:45:36.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:35 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/939569115' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:36.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:35 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/939569115' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:35 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/939569115' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:36.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 113527 113511 101.325 59.0625 0.00731529 0.0098639 2024-09-14T03:45:37.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:36 smithi114 ceph-mon[40961]: pgmap v132: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 52 MiB/s wr, 833 op/s 2024-09-14T03:45:37.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:36 smithi114 ceph-mon[40961]: from='client.24953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:37.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:36 smithi114 ceph-mon[40961]: from='client.24959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:37.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:36 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:37.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:36 smithi114 ceph-mon[29589]: pgmap v132: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 52 MiB/s wr, 833 op/s 2024-09-14T03:45:37.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:36 smithi114 ceph-mon[29589]: from='client.24953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:37.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:36 smithi114 ceph-mon[29589]: from='client.24959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:37.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:36 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:36 smithi160 ceph-mon[32702]: pgmap v132: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 52 MiB/s wr, 833 op/s 2024-09-14T03:45:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:36 smithi160 ceph-mon[32702]: from='client.24953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:36 smithi160 ceph-mon[32702]: from='client.24959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:36 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:37.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 114534 114518 100.784 62.9375 0.0112585 0.0099176 2024-09-14T03:45:38.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 15 115482 115467 100.208 59.3125 0.0125895 0.00997424 2024-09-14T03:45:38.691 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:39.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:38 smithi114 ceph-mon[40961]: pgmap v133: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s 2024-09-14T03:45:39.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:38 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3132127097' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:39.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:38 smithi114 ceph-mon[29589]: pgmap v133: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s 2024-09-14T03:45:39.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:38 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3132127097' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:38 smithi160 ceph-mon[32702]: pgmap v133: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s 2024-09-14T03:45:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:38 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3132127097' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:39.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 116429 116413 99.6453 59.125 0.0163612 0.0100305 2024-09-14T03:45:39.736 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:45:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:39] "GET /metrics HTTP/1.1" 200 243222 "" "Prometheus/2.43.0" 2024-09-14T03:45:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:39 smithi114 ceph-mon[29589]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:39 smithi114 ceph-mon[40961]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:39 smithi160 ceph-mon[32702]: from='client.24971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:40.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 117383 117367 99.1045 59.625 0.0069677 0.0100854 2024-09-14T03:45:41.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:40 smithi114 ceph-mon[29589]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:41.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:40 smithi114 ceph-mon[29589]: pgmap v134: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T03:45:41.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:40 smithi114 ceph-mon[40961]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:41.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:40 smithi114 ceph-mon[40961]: pgmap v134: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T03:45:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:40 smithi160 ceph-mon[32702]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:40 smithi160 ceph-mon[32702]: pgmap v134: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T03:45:41.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 118353 118337 98.5914 60.625 0.0167691 0.0101382 2024-09-14T03:45:41.741 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:45:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:45:41.742 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:41 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/322554464' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:42.091 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:42.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:41 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/322554464' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:42.091 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:41 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/322554464' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:42.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 119244 119228 98.0269 55.6875 0.0119199 0.0101961 2024-09-14T03:45:43.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:42 smithi114 ceph-mon[40961]: pgmap v135: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 51 MiB/s wr, 814 op/s 2024-09-14T03:45:43.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:42 smithi114 ceph-mon[29589]: pgmap v135: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 51 MiB/s wr, 814 op/s 2024-09-14T03:45:43.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:42 smithi160 ceph-mon[32702]: pgmap v135: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 51 MiB/s wr, 814 op/s 2024-09-14T03:45:43.317 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 120139 120123 97.4803 55.9375 0.0342829 0.0102532 2024-09-14T03:45:44.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:43 smithi114 ceph-mon[40961]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:44.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:43 smithi114 ceph-mon[40961]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:44.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:43 smithi114 ceph-mon[29589]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:44.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:43 smithi114 ceph-mon[29589]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:44.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:43 smithi160 ceph-mon[32702]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:44.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:43 smithi160 ceph-mon[32702]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:44.321 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 121029 121013 96.9437 55.625 0.00560161 0.0103106 2024-09-14T03:45:45.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:44 smithi114 ceph-mon[40961]: pgmap v136: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-14T03:45:45.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:44 smithi114 ceph-mon[29589]: pgmap v136: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-14T03:45:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:44 smithi160 ceph-mon[32702]: pgmap v136: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-14T03:45:45.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 121925 121909 96.4255 56 0.0316385 0.0103656 2024-09-14T03:45:45.428 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:46.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:45 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3822129962' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:46.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:45 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3822129962' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:45 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3822129962' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:46.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:45:46.317820+0000 min lat: 0.0012048 max lat: 0.122 avg lat: 0.0104222 2024-09-14T03:45:46.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:45:46.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 122800 122784 95.9038 54.6875 0.00494674 0.0104222 2024-09-14T03:45:47.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:46 smithi114 ceph-mon[40961]: pgmap v137: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 801 op/s 2024-09-14T03:45:47.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:46 smithi114 ceph-mon[40961]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:46 smithi114 ceph-mon[40961]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:46 smithi114 ceph-mon[29589]: pgmap v137: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 801 op/s 2024-09-14T03:45:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:46 smithi114 ceph-mon[29589]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:46 smithi114 ceph-mon[29589]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:46 smithi160 ceph-mon[32702]: pgmap v137: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 801 op/s 2024-09-14T03:45:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:46 smithi160 ceph-mon[32702]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:46 smithi160 ceph-mon[32702]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:47.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 123650 123634 95.3756 53.125 0.0224318 0.0104794 2024-09-14T03:45:48.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 124523 124507 94.8779 54.5625 0.0278951 0.0105351 2024-09-14T03:45:48.733 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:49.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:48 smithi114 ceph-mon[40961]: pgmap v138: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s 2024-09-14T03:45:49.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:48 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2212167444' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:49.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:48 smithi114 ceph-mon[29589]: pgmap v138: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s 2024-09-14T03:45:49.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:48 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2212167444' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:48 smithi160 ceph-mon[32702]: pgmap v138: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s 2024-09-14T03:45:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:48 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2212167444' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:49.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 125357 125341 94.3629 52.125 0.0137014 0.0105917 2024-09-14T03:45:49.789 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:45:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:49] "GET /metrics HTTP/1.1" 200 243222 "" "Prometheus/2.43.0" 2024-09-14T03:45:50.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:49 smithi114 ceph-mon[29589]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:50.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:49 smithi114 ceph-mon[40961]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:49 smithi160 ceph-mon[32702]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:50.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 126221 126205 93.8824 54 0.012275 0.010646 2024-09-14T03:45:51.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:50 smithi114 ceph-mon[40961]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:51.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:50 smithi114 ceph-mon[40961]: pgmap v139: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-14T03:45:51.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:50 smithi114 ceph-mon[29589]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:51.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:50 smithi114 ceph-mon[29589]: pgmap v139: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-14T03:45:51.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:50 smithi160 ceph-mon[32702]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:51.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:50 smithi160 ceph-mon[32702]: pgmap v139: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-14T03:45:51.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 127054 127038 93.3904 52.0625 0.00795407 0.010703 2024-09-14T03:45:51.795 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:45:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:45:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:51 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:51 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3845505334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:52.072 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:52.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:51 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:52.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:51 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3845505334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:52.073 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:51 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:45:52.073 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:51 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3845505334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:52.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 127880 127864 92.9047 51.625 0.0101705 0.0107582 2024-09-14T03:45:53.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:52 smithi114 ceph-mon[40961]: pgmap v140: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 761 op/s 2024-09-14T03:45:53.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:52 smithi114 ceph-mon[29589]: pgmap v140: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 761 op/s 2024-09-14T03:45:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:52 smithi160 ceph-mon[32702]: pgmap v140: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 761 op/s 2024-09-14T03:45:53.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 128752 128736 92.4633 54.5 0.00930433 0.01081 2024-09-14T03:45:54.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:53 smithi114 ceph-mon[29589]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:54.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:53 smithi114 ceph-mon[29589]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:53 smithi114 ceph-mon[40961]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:53 smithi114 ceph-mon[40961]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:53 smithi160 ceph-mon[32702]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:53 smithi160 ceph-mon[32702]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:54.318 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 129759 129743 92.1278 62.9375 0.0132782 0.0108496 2024-09-14T03:45:55.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:54 smithi114 ceph-mon[40961]: pgmap v141: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-14T03:45:55.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:54 smithi114 ceph-mon[29589]: pgmap v141: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-14T03:45:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:54 smithi160 ceph-mon[32702]: pgmap v141: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-14T03:45:55.319 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 130719 130703 91.7668 60 0.0243121 0.0108925 2024-09-14T03:45:55.491 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:56.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:55 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/664427401' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:56.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:55 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/664427401' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:55 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/664427401' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:56.319 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0141 2024-09-14T03:45:56.319 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 131678 2024-09-14T03:45:56.319 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-14T03:45:56.319 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-14T03:45:56.319 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 91.4287 2024-09-14T03:45:56.319 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 45.3306 2024-09-14T03:45:56.319 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 264.812 2024-09-14T03:45:56.319 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 51.625 2024-09-14T03:45:56.319 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1462 2024-09-14T03:45:56.320 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 725.289 2024-09-14T03:45:56.320 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 4237 2024-09-14T03:45:56.320 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 826 2024-09-14T03:45:56.320 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0109329 2024-09-14T03:45:56.320 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.00767984 2024-09-14T03:45:56.320 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 0.122 2024-09-14T03:45:56.320 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.0012048 2024-09-14T03:45:56.324 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-14T03:45:57.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:56 smithi114 ceph-mon[40961]: pgmap v142: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 45 MiB/s wr, 720 op/s 2024-09-14T03:45:57.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:56 smithi114 ceph-mon[40961]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:57.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:56 smithi114 ceph-mon[29589]: pgmap v142: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 45 MiB/s wr, 720 op/s 2024-09-14T03:45:57.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:56 smithi114 ceph-mon[29589]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:56 smithi160 ceph-mon[32702]: pgmap v142: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 45 MiB/s wr, 720 op/s 2024-09-14T03:45:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:56 smithi160 ceph-mon[32702]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:58.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:57 smithi114 ceph-mon[40961]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:58.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:57 smithi114 ceph-mon[29589]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:57 smithi160 ceph-mon[32702]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:58.844 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:45:59.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:58 smithi114 ceph-mon[29589]: pgmap v143: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 46 MiB/s wr, 748 op/s 2024-09-14T03:45:59.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:58 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3949790141' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:59.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:58 smithi114 ceph-mon[29589]: from='client.25076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:59.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:58 smithi114 ceph-mon[40961]: pgmap v143: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 46 MiB/s wr, 748 op/s 2024-09-14T03:45:59.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:58 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3949790141' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:59.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:58 smithi114 ceph-mon[40961]: from='client.25076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:58 smithi160 ceph-mon[32702]: pgmap v143: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 46 MiB/s wr, 748 op/s 2024-09-14T03:45:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:58 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3949790141' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:45:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:58 smithi160 ceph-mon[32702]: from='client.25076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:45:59.862 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:45:59 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:45:59] "GET /metrics HTTP/1.1" 200 243224 "" "Prometheus/2.43.0" 2024-09-14T03:46:00.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:59 smithi114 ceph-mon[40961]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:00.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:45:59 smithi114 ceph-mon[40961]: pgmap v144: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 62 MiB/s wr, 1.08k op/s 2024-09-14T03:46:00.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:59 smithi114 ceph-mon[29589]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:00.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:45:59 smithi114 ceph-mon[29589]: pgmap v144: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 62 MiB/s wr, 1.08k op/s 2024-09-14T03:46:00.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:59 smithi160 ceph-mon[32702]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:00.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:45:59 smithi160 ceph-mon[32702]: pgmap v144: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 62 MiB/s wr, 1.08k op/s 2024-09-14T03:46:01.708 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[34744]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:46:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:46:02.183 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:02 smithi160 ceph-mon[32702]: Upgrade: Updating mgr.x 2024-09-14T03:46:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:02 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:02 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:46:02.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:02 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:46:02.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:02 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:46:02.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:02 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:46:02.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:02 smithi160 ceph-mon[32702]: Deploying daemon mgr.x on smithi160 2024-09-14T03:46:02.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:02 smithi160 ceph-mon[32702]: pgmap v145: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 42 MiB/s wr, 757 op/s 2024-09-14T03:46:02.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:02 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2588577450' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[29589]: Upgrade: Updating mgr.x 2024-09-14T03:46:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:46:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:46:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:46:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:46:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[29589]: Deploying daemon mgr.x on smithi160 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[29589]: pgmap v145: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 42 MiB/s wr, 757 op/s 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2588577450' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[40961]: Upgrade: Updating mgr.x 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[40961]: Deploying daemon mgr.x on smithi160 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[40961]: pgmap v145: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 42 MiB/s wr, 757 op/s 2024-09-14T03:46:02.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:02 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2588577450' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:03.396 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:03 smithi160 ceph-mon[32702]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:03.396 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:03 smithi160 ceph-mon[32702]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:03.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:03 smithi114 ceph-mon[40961]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:03.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:03 smithi114 ceph-mon[40961]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:03.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:03 smithi114 ceph-mon[29589]: from='client.25094 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:03.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:03 smithi114 ceph-mon[29589]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:04.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:04 smithi160 systemd[1]: Stopping Ceph mgr.x for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:46:04.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:04 smithi160 ceph-mon[32702]: pgmap v146: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 42 MiB/s wr, 1.14k op/s 2024-09-14T03:46:04.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:04 smithi114 ceph-mon[40961]: pgmap v146: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 42 MiB/s wr, 1.14k op/s 2024-09-14T03:46:04.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:04 smithi114 ceph-mon[29589]: pgmap v146: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 42 MiB/s wr, 1.14k op/s 2024-09-14T03:46:04.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:04 smithi160 podman[68502]: 2024-09-14 03:46:04.63301814 +0000 UTC m=+0.182556080 container died ad77a9b0ace61c11658bc59acc32b7f91486f78fd19aac360fc6ddf9b3b83cbe (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, RELEASE=quincy-eb16b97, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-14T03:46:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:05 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3810853517' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:05.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:05 smithi160 podman[68502]: 2024-09-14 03:46:05.367254476 +0000 UTC m=+0.916792420 container cleanup ad77a9b0ace61c11658bc59acc32b7f91486f78fd19aac360fc6ddf9b3b83cbe (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, RELEASE=quincy-eb16b97, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-14T03:46:05.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:05 smithi160 bash[68502]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x 2024-09-14T03:46:05.611 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:05 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3810853517' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:05.612 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:05 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3810853517' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:05.612 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:05.906 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:05 smithi160 podman[68520]: 2024-09-14 03:46:05.636142491 +0000 UTC m=+0.995580913 container remove ad77a9b0ace61c11658bc59acc32b7f91486f78fd19aac360fc6ddf9b3b83cbe (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, RELEASE=quincy-eb16b97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-14T03:46:05.906 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:05 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-14T03:46:05.907 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:05 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-14T03:46:05.907 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:05 smithi160 systemd[1]: Stopped Ceph mgr.x for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:46:05.907 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:05 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.x.service: Consumed 10.471s CPU time. 2024-09-14T03:46:06.513 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:06 smithi160 systemd[1]: Starting Ceph mgr.x for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:46:06.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:06 smithi160 ceph-mon[32702]: pgmap v147: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 785 op/s 2024-09-14T03:46:06.514 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:06 smithi160 ceph-mon[32702]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:06.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:06 smithi114 ceph-mon[29589]: pgmap v147: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 785 op/s 2024-09-14T03:46:06.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:06 smithi114 ceph-mon[29589]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:06.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:06 smithi114 ceph-mon[40961]: pgmap v147: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 785 op/s 2024-09-14T03:46:06.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:06 smithi114 ceph-mon[40961]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:06.875 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:06 smithi160 podman[68635]: 2024-09-14 03:46:06.417226273 +0000 UTC m=+0.029225488 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:46:06.875 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:06 smithi160 podman[68635]: 2024-09-14 03:46:06.570581074 +0000 UTC m=+0.182580285 container create e4ad5ee26d6afba939042057254af181ac8495816a9a680868fe0f31f166d10c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a278231, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-09-14T03:46:07.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:06 smithi160 podman[68635]: 2024-09-14 03:46:06.874018185 +0000 UTC m=+0.486017403 container init e4ad5ee26d6afba939042057254af181ac8495816a9a680868fe0f31f166d10c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=main-a278231, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-14T03:46:07.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:06 smithi160 podman[68635]: 2024-09-14 03:46:06.881686767 +0000 UTC m=+0.493685974 container start e4ad5ee26d6afba939042057254af181ac8495816a9a680868fe0f31f166d10c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-a278231) 2024-09-14T03:46:07.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:06 smithi160 ceph-mgr[68671]: -- 172.21.15.160:0/759892159 <== mon.2 v2:172.21.15.114:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557660e8eea0 con 0x557660e63400 2024-09-14T03:46:07.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:06 smithi160 ceph-mgr[68671]: -- 172.21.15.160:0/759892159 <== mon.2 v2:172.21.15.114:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557660e850e0 con 0x557660e63400 2024-09-14T03:46:07.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:06 smithi160 bash[68635]: e4ad5ee26d6afba939042057254af181ac8495816a9a680868fe0f31f166d10c 2024-09-14T03:46:07.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:06 smithi160 systemd[1]: Started Ceph mgr.x for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:46:07.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:07 smithi114 ceph-mon[29589]: from='client.25115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:07 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:07 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:07 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:07 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:46:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:07 smithi114 ceph-mon[40961]: from='client.25115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:07 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:07 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:07 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:07 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:46:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:07 smithi160 ceph-mon[32702]: from='client.25115 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:07 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:07 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:07 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:07 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:46:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:08 smithi160 ceph-mon[32702]: pgmap v148: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 807 op/s 2024-09-14T03:46:08.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:08 smithi114 ceph-mon[29589]: pgmap v148: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 807 op/s 2024-09-14T03:46:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:08 smithi114 ceph-mon[40961]: pgmap v148: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 807 op/s 2024-09-14T03:46:08.946 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:09.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:09 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/973613699' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:09.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:09 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/973613699' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:09 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/973613699' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:09.734 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:09 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:46:09] "GET /metrics HTTP/1.1" 200 243325 "" "Prometheus/2.43.0" 2024-09-14T03:46:10.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:10 smithi114 ceph-mon[29589]: from='client.25130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:10.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:10 smithi114 ceph-mon[29589]: pgmap v149: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 19 MiB/s wr, 1.13k op/s 2024-09-14T03:46:10.452 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:10 smithi114 ceph-mon[29589]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:10.452 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:10 smithi114 ceph-mon[40961]: from='client.25130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:10.452 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:10 smithi114 ceph-mon[40961]: pgmap v149: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 19 MiB/s wr, 1.13k op/s 2024-09-14T03:46:10.452 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:10 smithi114 ceph-mon[40961]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:10.468 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:10 smithi160 ceph-mon[32702]: from='client.25130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:10.468 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:10 smithi160 ceph-mon[32702]: pgmap v149: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 19 MiB/s wr, 1.13k op/s 2024-09-14T03:46:10.469 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:10 smithi160 ceph-mon[32702]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:12.305 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:11 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:12.305 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:11 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:12.305 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:11 smithi160 ceph-mon[32702]: pgmap v150: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T03:46:12.305 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:11 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/624147397' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:12.360 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:12.361 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:11 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:12.361 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:11 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:12.361 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:11 smithi114 ceph-mon[40961]: pgmap v150: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T03:46:12.361 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:11 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/624147397' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:12.361 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:11 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:12.361 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:11 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:12.361 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:11 smithi114 ceph-mon[29589]: pgmap v150: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T03:46:12.361 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:11 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/624147397' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:13.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:12 smithi114 ceph-mon[29589]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:13.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:12 smithi114 ceph-mon[29589]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:13.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:12 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:13.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:12 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:13.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:12 smithi114 ceph-mon[40961]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:13.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:12 smithi114 ceph-mon[40961]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:13.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:12 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:13.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:12 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:13.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:12 smithi160 ceph-mon[32702]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:13.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:12 smithi160 ceph-mon[32702]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:13.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:12 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:13.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:12 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:14.297 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:13 smithi160 ceph-mon[32702]: pgmap v151: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T03:46:14.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:13 smithi114 ceph-mon[40961]: pgmap v151: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T03:46:14.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:13 smithi114 ceph-mon[29589]: pgmap v151: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T03:46:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:15 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:15 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:15.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:15 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:15.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:15 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:15.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:15 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:15.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:15 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:15.720 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:16.132 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:46:15] ENGINE Bus STARTING 2024-09-14T03:46:16.133 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: CherryPy Checker: 2024-09-14T03:46:16.133 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: The Application mounted at '' has an empty config. 2024-09-14T03:46:16.133 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:16.133 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:46:15] ENGINE Serving on http://:::9283 2024-09-14T03:46:16.133 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:46:15] ENGINE Bus STARTED 2024-09-14T03:46:16.388 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:16 smithi160 ceph-mon[32702]: pgmap v152: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-14T03:46:16.388 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:16 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2763641212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:16.388 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:16 smithi160 ceph-mon[32702]: Standby manager daemon x restarted 2024-09-14T03:46:16.389 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:16 smithi160 ceph-mon[32702]: Standby manager daemon x started 2024-09-14T03:46:16.389 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:16 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:46:16.389 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:16 smithi160 ceph-mon[32702]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:16.389 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:16 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:46:16.389 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:16 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:46:16.389 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:16 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:46:16.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[40961]: pgmap v152: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-14T03:46:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2763641212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[40961]: Standby manager daemon x restarted 2024-09-14T03:46:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[40961]: Standby manager daemon x started 2024-09-14T03:46:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:46:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[40961]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:46:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:46:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:46:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[29589]: pgmap v152: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-14T03:46:16.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2763641212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:16.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[29589]: Standby manager daemon x restarted 2024-09-14T03:46:16.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[29589]: Standby manager daemon x started 2024-09-14T03:46:16.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:46:16.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[29589]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:16.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:46:16.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:46:16.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:16 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:46:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:17 smithi160 ceph-mon[32702]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:17.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:17 smithi160 ceph-mon[32702]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-14T03:46:17.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:17 smithi114 ceph-mon[29589]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:17.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:17 smithi114 ceph-mon[29589]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-14T03:46:17.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:17 smithi114 ceph-mon[40961]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:17.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:17 smithi114 ceph-mon[40961]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-14T03:46:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:18 smithi160 ceph-mon[32702]: pgmap v153: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-14T03:46:18.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:18 smithi114 ceph-mon[40961]: pgmap v153: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-14T03:46:18.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:18 smithi114 ceph-mon[29589]: pgmap v153: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-14T03:46:19.145 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:19.389 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:19 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3623571891' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:19.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:19 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3623571891' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:19.421 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:19.370+0000 7eff44311640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-14T03:46:19.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:19 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3623571891' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:19.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:46:19] ENGINE Bus STOPPING 2024-09-14T03:46:19.701 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ignoring --setuser ceph since I am not root 2024-09-14T03:46:19.701 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ignoring --setgroup ceph since I am not root 2024-09-14T03:46:19.701 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:19.607+0000 7fe552f36ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-14T03:46:19.701 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:19.699+0000 7fe552f36ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-14T03:46:20.013 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:19 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:19.800+0000 7fe552f36ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-14T03:46:20.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:46:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-14T03:46:20.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:19 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:46:19] ENGINE Bus STOPPED 2024-09-14T03:46:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:46:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:46:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: Activating manager daemon x 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: osdmap e69: 8 total, 8 up, 8 in 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: mgrmap e22: x(active, starting, since 0.0118585s) 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"}]: dispatch 2024-09-14T03:46:20.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:46:20.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:46:20.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[40961]: Manager daemon x is now available 2024-09-14T03:46:20.374 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:20.011+0000 7fe552f36ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:46:20.374 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:20.225+0000 7fe552f36ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-14T03:46:20.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:20.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:20.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:46:20.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:46:20.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: Activating manager daemon x 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: osdmap e69: 8 total, 8 up, 8 in 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: mgrmap e22: x(active, starting, since 0.0118585s) 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"}]: dispatch 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"}]: dispatch 2024-09-14T03:46:20.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:46:20.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-mon[29589]: Manager daemon x is now available 2024-09-14T03:46:20.378 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date 2024-09-14T03:46:20.379 INFO:teuthology.orchestra.run.smithi114.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-09-14T03:46:20.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:20.376+0000 7f753d902640 -1 mgr.server reply reply (95) Operation not supported Warning: due to ceph-mgr restart, some PG states may not be up to date 2024-09-14T03:46:20.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-09-14T03:46:20.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:20.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:20.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:46:20.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:46:20.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.14448 172.21.15.114:0/261085501' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: Activating manager daemon x 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: osdmap e69: 8 total, 8 up, 8 in 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.14448 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: mgrmap e22: x(active, starting, since 0.0118585s) 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"}]: dispatch 2024-09-14T03:46:20.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:46:20.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:20 smithi160 ceph-mon[32702]: Manager daemon x is now available 2024-09-14T03:46:20.707 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:20.614+0000 7fe552f36ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-14T03:46:20.972 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:20.706+0000 7fe552f36ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-14T03:46:21.368 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:20.971+0000 7fe552f36ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-14T03:46:21.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:21 smithi114 ceph-mon[40961]: mgrmap e23: x(active, since 1.01605s) 2024-09-14T03:46:21.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:21 smithi114 ceph-mon[40961]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:21.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:21 smithi114 ceph-mon[40961]: pgmap v3: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail 2024-09-14T03:46:21.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:21 smithi114 ceph-mon[29589]: mgrmap e23: x(active, since 1.01605s) 2024-09-14T03:46:21.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:21 smithi114 ceph-mon[29589]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:21.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:21 smithi114 ceph-mon[29589]: pgmap v3: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail 2024-09-14T03:46:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-mon[32702]: mgrmap e23: x(active, since 1.01605s) 2024-09-14T03:46:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-mon[32702]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-mon[32702]: pgmap v3: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail 2024-09-14T03:46:21.969 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:46:21] ENGINE Bus STARTING 2024-09-14T03:46:21.969 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:21.879+0000 7f75130ad640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:46:21.969 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:21.879+0000 7f75130ad640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:46:21.969 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:21.879+0000 7f75130ad640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:46:21.969 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:21.879+0000 7f75130ad640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:46:21.970 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:21.879+0000 7f75130ad640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:46:21.970 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:21.887+0000 7f751b0bd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:46:21.970 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:21.887+0000 7f751b0bd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:46:21.970 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:21.887+0000 7f751b0bd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:46:21.970 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:21.887+0000 7f751b0bd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:46:21.970 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:21.887+0000 7f751b0bd640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:46:21.970 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:46:21] ENGINE Serving on http://:::9283 2024-09-14T03:46:21.970 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:21 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:46:21] ENGINE Bus STARTED 2024-09-14T03:46:22.579 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:22.260+0000 7fe552f36ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-14T03:46:22.579 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:22.355+0000 7fe552f36ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-14T03:46:22.579 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:46:22.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-09-14T03:46:22.580 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:46:22.322Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi160.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.160:8443: connect: connection refused" 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: Checking for cert/key for grafana.a 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: Migrating grafana.a cert to cert store 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: Migrating grafana.a key to cert store 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:46:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:46:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:46:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:46:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-14T03:46:22.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-14T03:46:22.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: Checking for cert/key for grafana.a 2024-09-14T03:46:22.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: Migrating grafana.a cert to cert store 2024-09-14T03:46:22.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: Migrating grafana.a key to cert store 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-14T03:46:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:22 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-14T03:46:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: Checking for cert/key for grafana.a 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: Migrating grafana.a cert to cert store 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: Migrating grafana.a key to cert store 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:46:22.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:46:22.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-14T03:46:22.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:22 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-14T03:46:23.370 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:46:23.207Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=2 2024-09-14T03:46:23.587 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:23.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-mon[40961]: Deploying cephadm binary to smithi160 2024-09-14T03:46:23.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-mon[40961]: Deploying cephadm binary to smithi114 2024-09-14T03:46:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-mon[40961]: mgrmap e24: x(active, since 3s) 2024-09-14T03:46:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1053334857' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-mon[29589]: Deploying cephadm binary to smithi160 2024-09-14T03:46:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-mon[29589]: Deploying cephadm binary to smithi114 2024-09-14T03:46:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-mon[29589]: mgrmap e24: x(active, since 3s) 2024-09-14T03:46:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1053334857' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:23.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:23.621+0000 7fe552f36ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-14T03:46:23.872 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:23 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:23.799+0000 7fe552f36ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-14T03:46:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:23 smithi160 ceph-mon[32702]: Deploying cephadm binary to smithi160 2024-09-14T03:46:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:23 smithi160 ceph-mon[32702]: Deploying cephadm binary to smithi114 2024-09-14T03:46:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:23 smithi160 ceph-mon[32702]: mgrmap e24: x(active, since 3s) 2024-09-14T03:46:23.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:23 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1053334857' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:24.263 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:24.008+0000 7fe552f36ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-14T03:46:24.264 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:24.110+0000 7fe552f36ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-14T03:46:24.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:24.082+0000 7f74f937d640 -1 log_channel(cephadm) log [ERR] : [14/Sep/2024:03:46:24] ENGINE Error in HTTPServer.serve 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: Traceback (most recent call last): 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: self._connections.run(self.expiration_interval) 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: self._run(expiration_interval) 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: s = self.context.wrap_socket( 2024-09-14T03:46:24.409 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:46:24.410 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: return self.sslsocket_class._create( 2024-09-14T03:46:24.410 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:46:24.410 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: self.do_handshake() 2024-09-14T03:46:24.410 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:46:24.410 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: self._sslobj.do_handshake() 2024-09-14T03:46:24.410 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:46:24.410 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:46:24.523 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:24.319+0000 7fe552f36ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-14T03:46:24.523 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:24.428+0000 7fe552f36ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-14T03:46:24.847 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-mon[29589]: pgmap v5: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail 2024-09-14T03:46:24.847 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-mon[29589]: from='client.25202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:24.847 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-mon[29589]: [14/Sep/2024:03:46:23] ENGINE Bus STARTING 2024-09-14T03:46:24.847 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-mon[40961]: pgmap v5: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail 2024-09-14T03:46:24.847 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-mon[40961]: from='client.25202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:24.847 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-mon[40961]: [14/Sep/2024:03:46:23] ENGINE Bus STARTING 2024-09-14T03:46:24.847 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:24.522+0000 7fe552f36ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-14T03:46:24.848 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:24.741+0000 7fe552f36ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-14T03:46:24.848 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:24.845+0000 7fe552f36ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-14T03:46:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-mon[32702]: pgmap v5: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail 2024-09-14T03:46:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-mon[32702]: from='client.25202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:24 smithi160 ceph-mon[32702]: [14/Sep/2024:03:46:23] ENGINE Bus STARTING 2024-09-14T03:46:25.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:24.946+0000 7fe552f36ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-14T03:46:25.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:25.058+0000 7fe552f36ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-14T03:46:25.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:25.149+0000 7fe552f36ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-14T03:46:25.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:25.239+0000 7fe552f36ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-14T03:46:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:25 smithi160 ceph-mon[32702]: [14/Sep/2024:03:46:24] ENGINE Serving on https://172.21.15.160:7150 2024-09-14T03:46:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:25 smithi160 ceph-mon[32702]: [14/Sep/2024:03:46:24] ENGINE Error in HTTPServer.serve 2024-09-14T03:46:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout: Traceback (most recent call last): 2024-09-14T03:46:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:46:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:46:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._run(expiration_interval) 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: s = self.context.wrap_socket( 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: return self.sslsocket_class._create( 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: self.do_handshake() 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:46:25.909 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._sslobj.do_handshake() 2024-09-14T03:46:25.910 INFO:journalctl@ceph.mon.b.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:46:25.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:25 smithi160 ceph-mon[32702]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:25.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:25 smithi160 ceph-mon[32702]: [14/Sep/2024:03:46:24] ENGINE Serving on http://172.21.15.160:8765 2024-09-14T03:46:25.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:25 smithi160 ceph-mon[32702]: [14/Sep/2024:03:46:24] ENGINE Bus STARTED 2024-09-14T03:46:25.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:25 smithi160 ceph-mon[32702]: mgrmap e25: x(active, since 5s) 2024-09-14T03:46:26.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[29589]: [14/Sep/2024:03:46:24] ENGINE Serving on https://172.21.15.160:7150 2024-09-14T03:46:26.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[29589]: [14/Sep/2024:03:46:24] ENGINE Error in HTTPServer.serve 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:46:26.122 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[29589]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[29589]: [14/Sep/2024:03:46:24] ENGINE Serving on http://172.21.15.160:8765 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[29589]: [14/Sep/2024:03:46:24] ENGINE Bus STARTED 2024-09-14T03:46:26.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[29589]: mgrmap e25: x(active, since 5s) 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[40961]: [14/Sep/2024:03:46:24] ENGINE Serving on https://172.21.15.160:7150 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[40961]: [14/Sep/2024:03:46:24] ENGINE Error in HTTPServer.serve 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout: Traceback (most recent call last): 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._run(expiration_interval) 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:46:26.124 INFO:journalctl@ceph.mon.c.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout: s = self.context.wrap_socket( 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout: return self.sslsocket_class._create( 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout: self.do_handshake() 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._sslobj.do_handshake() 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[40961]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[40961]: [14/Sep/2024:03:46:24] ENGINE Serving on http://172.21.15.160:8765 2024-09-14T03:46:26.125 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[40961]: [14/Sep/2024:03:46:24] ENGINE Bus STARTED 2024-09-14T03:46:26.126 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:25 smithi114 ceph-mon[40961]: mgrmap e25: x(active, since 5s) 2024-09-14T03:46:26.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:26.508+0000 7fe552f36ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-14T03:46:26.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:46:26] ENGINE Bus STARTING 2024-09-14T03:46:26.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: CherryPy Checker: 2024-09-14T03:46:26.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: The Application mounted at '' has an empty config. 2024-09-14T03:46:26.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: 2024-09-14T03:46:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[29589]: pgmap v6: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail 2024-09-14T03:46:27.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[29589]: Standby manager daemon y started 2024-09-14T03:46:27.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-14T03:46:27.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:46:27.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-14T03:46:27.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:46:27.122 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:46:26] ENGINE Serving on http://:::9283 2024-09-14T03:46:27.122 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: [14/Sep/2024:03:46:26] ENGINE Bus STARTED 2024-09-14T03:46:27.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[40961]: pgmap v6: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail 2024-09-14T03:46:27.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[40961]: Standby manager daemon y started 2024-09-14T03:46:27.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-14T03:46:27.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:46:27.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-14T03:46:27.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:26 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:46:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:26 smithi160 ceph-mon[32702]: pgmap v6: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail 2024-09-14T03:46:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:26 smithi160 ceph-mon[32702]: Standby manager daemon y started 2024-09-14T03:46:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:26 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-14T03:46:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:26 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:46:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:26 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-14T03:46:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:26 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.114:0/1938419536' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:46:27.274 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:28.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:27 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:46:28.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:27 smithi114 ceph-mon[29589]: mgrmap e26: x(active, since 7s), standbys: y 2024-09-14T03:46:28.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:27 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3253181776' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:28.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:27 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:46:28.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:27 smithi114 ceph-mon[40961]: mgrmap e26: x(active, since 7s), standbys: y 2024-09-14T03:46:28.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:27 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3253181776' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:27 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:46:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:27 smithi160 ceph-mon[32702]: mgrmap e26: x(active, since 7s), standbys: y 2024-09-14T03:46:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:27 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3253181776' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:28 smithi160 ceph-mon[32702]: from='client.25223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:28 smithi160 ceph-mon[32702]: pgmap v7: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 25 KiB/s rd, 440 B/s wr, 545 op/s 2024-09-14T03:46:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:28 smithi160 ceph-mon[32702]: from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:29.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:28 smithi114 ceph-mon[40961]: from='client.25223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:29.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:28 smithi114 ceph-mon[40961]: pgmap v7: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 25 KiB/s rd, 440 B/s wr, 545 op/s 2024-09-14T03:46:29.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:28 smithi114 ceph-mon[40961]: from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:29.099 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:28 smithi114 ceph-mon[29589]: from='client.25223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:29.099 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:28 smithi114 ceph-mon[29589]: pgmap v7: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 25 KiB/s rd, 440 B/s wr, 545 op/s 2024-09-14T03:46:29.099 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:28 smithi114 ceph-mon[29589]: from='client.24952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:29.370 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:46:29.098Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=6 2024-09-14T03:46:29.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:46:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:46:30.819 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:30.983 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:30 smithi160 ceph-mon[32702]: pgmap v8: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 20 KiB/s rd, 342 B/s wr, 423 op/s 2024-09-14T03:46:30.983 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:30 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1170957850' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:31.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:30 smithi114 ceph-mon[29589]: pgmap v8: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 20 KiB/s rd, 342 B/s wr, 423 op/s 2024-09-14T03:46:31.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:30 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1170957850' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:31.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:30 smithi114 ceph-mon[40961]: pgmap v8: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 20 KiB/s rd, 342 B/s wr, 423 op/s 2024-09-14T03:46:31.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:30 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1170957850' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:31.804 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:46:31] "GET /metrics HTTP/1.1" 200 35985 "" "Prometheus/2.43.0" 2024-09-14T03:46:32.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:32.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:31 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:32.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:31 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:33.352 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:32 smithi160 ceph-mon[32702]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:33.353 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:32 smithi160 ceph-mon[32702]: pgmap v9: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 16 KiB/s rd, 280 B/s wr, 715 op/s 2024-09-14T03:46:33.353 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:32 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi114 to 2480M 2024-09-14T03:46:33.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:32 smithi114 ceph-mon[40961]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:33.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:32 smithi114 ceph-mon[40961]: pgmap v9: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 16 KiB/s rd, 280 B/s wr, 715 op/s 2024-09-14T03:46:33.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:32 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi114 to 2480M 2024-09-14T03:46:33.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:32 smithi114 ceph-mon[29589]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:33.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:32 smithi114 ceph-mon[29589]: pgmap v9: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 16 KiB/s rd, 280 B/s wr, 715 op/s 2024-09-14T03:46:33.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:32 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi114 to 2480M 2024-09-14T03:46:34.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:33 smithi114 ceph-mon[40961]: pgmap v10: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 653 op/s 2024-09-14T03:46:34.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:33 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1408101706' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:34.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:33 smithi114 ceph-mon[29589]: pgmap v10: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 653 op/s 2024-09-14T03:46:34.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:33 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1408101706' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:33 smithi160 ceph-mon[32702]: pgmap v10: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 653 op/s 2024-09-14T03:46:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:33 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1408101706' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:34.437 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:35.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[40961]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[40961]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[29589]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[29589]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:34 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:34 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:34 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:34 smithi160 ceph-mon[32702]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:34 smithi160 ceph-mon[32702]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:34 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:34 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:35 smithi160 ceph-mon[32702]: pgmap v11: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 653 op/s 2024-09-14T03:46:36.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:35 smithi114 ceph-mon[40961]: pgmap v11: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 653 op/s 2024-09-14T03:46:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:35 smithi114 ceph-mon[29589]: pgmap v11: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 653 op/s 2024-09-14T03:46:37.025 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:37 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:37.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:37 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:37.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:37 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:37.937 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:38.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:38 smithi114 ceph-mon[40961]: pgmap v12: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 986 op/s 2024-09-14T03:46:38.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:38 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1482742189' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:38.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:38 smithi114 ceph-mon[40961]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:38 smithi114 ceph-mon[29589]: pgmap v12: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 986 op/s 2024-09-14T03:46:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:38 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1482742189' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:38 smithi114 ceph-mon[29589]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:38 smithi160 ceph-mon[32702]: pgmap v12: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 986 op/s 2024-09-14T03:46:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:38 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1482742189' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:38 smithi160 ceph-mon[32702]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:39.017 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:39 smithi160 ceph-mon[32702]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:39.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:39 smithi114 ceph-mon[40961]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:39.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:39 smithi114 ceph-mon[29589]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:39.872 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:46:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:46:40.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: pgmap v13: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 670 op/s 2024-09-14T03:46:40.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:40.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:40.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: Adjusting osd_memory_target on smithi160 to 2480M 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:46:40.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:46:40.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:46:40.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:46:40.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:40.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:40 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: pgmap v13: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 670 op/s 2024-09-14T03:46:41.153 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: Adjusting osd_memory_target on smithi160 to 2480M 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:46:41.155 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:46:41.155 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:46:41.155 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:46:41.155 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:46:41.155 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:46:41.155 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.155 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.155 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: pgmap v13: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 670 op/s 2024-09-14T03:46:41.155 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.155 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: Adjusting osd_memory_target on smithi160 to 2480M 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:46:41.156 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:46:41.157 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:46:41.157 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:46:41.157 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:46:41.157 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.157 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:40 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.573 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:41.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-14T03:46:41.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: Deploying daemon alertmanager.a on smithi114 2024-09-14T03:46:41.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3300193723' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:41.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:46:41.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:46:41.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:46:41.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:46:41.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-14T03:46:41.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: Deploying daemon alertmanager.a on smithi114 2024-09-14T03:46:41.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:41 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3300193723' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:41.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:46:41] "GET /metrics HTTP/1.1" 200 36198 "" "Prometheus/2.43.0" 2024-09-14T03:46:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:46:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:46:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:46:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:46:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:41.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-14T03:46:41.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: Deploying daemon alertmanager.a on smithi114 2024-09-14T03:46:41.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:41 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3300193723' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:43.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:42 smithi114 ceph-mon[29589]: pgmap v14: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:46:43.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:42 smithi114 ceph-mon[29589]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:43.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:42 smithi114 ceph-mon[40961]: pgmap v14: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:46:43.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:42 smithi114 ceph-mon[40961]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:42 smithi160 ceph-mon[32702]: pgmap v14: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:46:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:42 smithi160 ceph-mon[32702]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:44.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:43 smithi114 ceph-mon[29589]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:44.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:43 smithi114 ceph-mon[40961]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:44.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:43 smithi160 ceph-mon[32702]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:45.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:44 smithi114 ceph-mon[29589]: pgmap v15: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-14T03:46:45.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:44 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/4289548870' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:45.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:44 smithi114 ceph-mon[40961]: pgmap v15: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-14T03:46:45.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:44 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/4289548870' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:44 smithi160 ceph-mon[32702]: pgmap v15: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-14T03:46:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:44 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/4289548870' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:45.199 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:46.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:46 smithi114 ceph-mon[29589]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:46.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:46 smithi114 ceph-mon[29589]: pgmap v16: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-14T03:46:46.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:46 smithi114 ceph-mon[29589]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:46.332 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:46 smithi114 ceph-mon[40961]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:46.332 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:46 smithi114 ceph-mon[40961]: pgmap v16: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-14T03:46:46.332 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:46 smithi114 ceph-mon[40961]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:46 smithi160 ceph-mon[32702]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:46 smithi160 ceph-mon[32702]: pgmap v16: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 696 op/s 2024-09-14T03:46:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:46 smithi160 ceph-mon[32702]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:48.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:48 smithi114 ceph-mon[40961]: pgmap v17: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-14T03:46:48.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:48 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2337477212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:48.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:48 smithi114 ceph-mon[29589]: pgmap v17: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-14T03:46:48.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:48 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2337477212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:48.799 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:48 smithi160 ceph-mon[32702]: pgmap v17: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-14T03:46:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:48 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2337477212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:49 smithi114 ceph-mon[40961]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:49.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:49 smithi114 ceph-mon[29589]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:49.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:46:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:46:49.872 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:49 smithi114 systemd[1]: Stopping Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:46:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:49 smithi160 ceph-mon[32702]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:50.327 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:49 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[77768]: ts=2024-09-14T03:46:49.944Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-14T03:46:50.327 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:49 smithi114 podman[91571]: 2024-09-14 03:46:49.998132912 +0000 UTC m=+0.195813867 container died 80f53fb9f5983b665626832a2f84e6951abdd2835c6fe723f9a38588db8c1ca3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:46:50.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:50 smithi114 ceph-mon[29589]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:50 smithi114 ceph-mon[29589]: pgmap v18: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-09-14T03:46:50.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:50 smithi114 ceph-mon[40961]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:50.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:50 smithi114 ceph-mon[40961]: pgmap v18: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-09-14T03:46:50.621 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:50 smithi114 podman[91571]: 2024-09-14 03:46:50.424949395 +0000 UTC m=+0.622630355 container cleanup 80f53fb9f5983b665626832a2f84e6951abdd2835c6fe723f9a38588db8c1ca3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:46:50.621 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:50 smithi114 bash[91571]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a 2024-09-14T03:46:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:50 smithi160 ceph-mon[32702]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:50 smithi160 ceph-mon[32702]: pgmap v18: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-09-14T03:46:51.621 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:51 smithi114 podman[91587]: 2024-09-14 03:46:51.311205317 +0000 UTC m=+1.308743237 container remove 80f53fb9f5983b665626832a2f84e6951abdd2835c6fe723f9a38588db8c1ca3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:46:51.621 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:51 smithi114 podman[91587]: 2024-09-14 03:46:51.404103911 +0000 UTC m=+1.401641838 volume remove 2f8f51a23eee7f9117165f53d552234d7d05da556ec12094f5a0a3faaf0e502e 2024-09-14T03:46:51.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:46:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:46:51] "GET /metrics HTTP/1.1" 200 36198 "" "Prometheus/2.43.0" 2024-09-14T03:46:52.120 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:51 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-14T03:46:52.121 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:51 smithi114 systemd[1]: Stopped Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:46:52.121 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:51 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.446s CPU time. 2024-09-14T03:46:52.466 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:52 smithi114 systemd[1]: Starting Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:46:52.509 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:52 smithi160 ceph-mon[32702]: pgmap v19: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:46:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:52 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:52 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2401761932' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:52.659 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:46:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:46:52.315Z caller=notifier.go:532 level=error component=notifier alertmanager=http://smithi114.front.sepia.ceph.com:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://smithi114.front.sepia.ceph.com:9093/api/v2/alerts\": dial tcp 172.21.15.114:9093: connect: connection refused" 2024-09-14T03:46:52.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:52 smithi114 ceph-mon[40961]: pgmap v19: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:46:52.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:52 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:52.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:52 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2401761932' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:52.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:52 smithi114 ceph-mon[29589]: pgmap v19: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:46:52.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:52 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:46:52.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:52 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2401761932' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:52.872 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:52 smithi114 podman[91744]: 2024-09-14 03:46:52.455615517 +0000 UTC m=+0.033641207 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-14T03:46:52.872 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:52 smithi114 podman[91744]: 2024-09-14 03:46:52.555774851 +0000 UTC m=+0.133800533 volume create 824b68cd45c331c5968a641ac24931324bb1fdb5fbc80ff6c8f096f714ac757d 2024-09-14T03:46:52.872 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:52 smithi114 podman[91744]: 2024-09-14 03:46:52.664899831 +0000 UTC m=+0.242925517 container create eaf39a5cb5818b4f343a2214dded03e379a31dc549a2fa902d393e86b914cba3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:46:53.175 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 podman[91744]: 2024-09-14 03:46:53.085625396 +0000 UTC m=+0.663651091 container init eaf39a5cb5818b4f343a2214dded03e379a31dc549a2fa902d393e86b914cba3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:46:53.176 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 podman[91744]: 2024-09-14 03:46:53.09562779 +0000 UTC m=+0.673653478 container start eaf39a5cb5818b4f343a2214dded03e379a31dc549a2fa902d393e86b914cba3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T03:46:53.176 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T03:46:53.121Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-14T03:46:53.176 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T03:46:53.121Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-14T03:46:53.176 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T03:46:53.123Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.114 port=9094 2024-09-14T03:46:53.176 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T03:46:53.126Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-14T03:46:53.469 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T03:46:53.173Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-14T03:46:53.469 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T03:46:53.174Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-14T03:46:53.469 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T03:46:53.177Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-14T03:46:53.469 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T03:46:53.177Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-14T03:46:53.469 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 bash[91744]: eaf39a5cb5818b4f343a2214dded03e379a31dc549a2fa902d393e86b914cba3 2024-09-14T03:46:53.469 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:53 smithi114 systemd[1]: Started Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:46:53.771 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-mon[40961]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:53.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:53.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:53.772 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-mon[29589]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:53.772 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:53.772 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:53 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:53 smithi160 ceph-mon[32702]: from='client.25316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:53 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:53 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:46:54.370 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:54 smithi114 systemd[1]: Stopping Ceph node-exporter.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:46:54.818 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:54 smithi114 podman[91930]: 2024-09-14 03:46:54.506952652 +0000 UTC m=+0.145995732 container died 40522b00d68c4486465a77ac08abd8faa62f2bbd5c27cc09be07663e1a352e83 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T03:46:54.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:54 smithi114 ceph-mon[40961]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:54.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:54 smithi114 ceph-mon[40961]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-14T03:46:54.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:54 smithi114 ceph-mon[40961]: Deploying daemon node-exporter.a on smithi114 2024-09-14T03:46:54.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:54 smithi114 ceph-mon[40961]: pgmap v20: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:46:54.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:54 smithi114 ceph-mon[29589]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:54.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:54 smithi114 ceph-mon[29589]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-14T03:46:54.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:54 smithi114 ceph-mon[29589]: Deploying daemon node-exporter.a on smithi114 2024-09-14T03:46:54.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:54 smithi114 ceph-mon[29589]: pgmap v20: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:46:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:54 smithi160 ceph-mon[32702]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:54 smithi160 ceph-mon[32702]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-14T03:46:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:54 smithi160 ceph-mon[32702]: Deploying daemon node-exporter.a on smithi114 2024-09-14T03:46:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:54 smithi160 ceph-mon[32702]: pgmap v20: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:46:55.120 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:55 smithi114 podman[91930]: 2024-09-14 03:46:55.025890121 +0000 UTC m=+0.664933201 container cleanup 40522b00d68c4486465a77ac08abd8faa62f2bbd5c27cc09be07663e1a352e83 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T03:46:55.120 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:55 smithi114 bash[91930]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a 2024-09-14T03:46:55.605 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:46:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T03:46:55.126Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000477919s 2024-09-14T03:46:55.872 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:55 smithi114 podman[91946]: 2024-09-14 03:46:55.604204304 +0000 UTC m=+1.085816161 container remove 40522b00d68c4486465a77ac08abd8faa62f2bbd5c27cc09be07663e1a352e83 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T03:46:55.872 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:55 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-14T03:46:56.215 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:56.215 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:55 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-14T03:46:56.215 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:55 smithi114 systemd[1]: Stopped Ceph node-exporter.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:46:56.215 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:55 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 2.623s CPU time. 2024-09-14T03:46:56.477 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:56 smithi114 systemd[1]: Starting Ceph node-exporter.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:46:56.870 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:56 smithi114 bash[92134]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-14T03:46:56.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:56 smithi114 ceph-mon[40961]: pgmap v21: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:46:56.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:56 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2423436646' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:56.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:56 smithi114 ceph-mon[29589]: pgmap v21: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:46:56.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:56 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2423436646' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:56 smithi160 ceph-mon[32702]: pgmap v21: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:46:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:56 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2423436646' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:57.370 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:57 smithi114 bash[92134]: Getting image source signatures 2024-09-14T03:46:57.370 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:57 smithi114 bash[92134]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-14T03:46:57.370 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:57 smithi114 bash[92134]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-14T03:46:57.371 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:57 smithi114 bash[92134]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-14T03:46:57.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:57 smithi114 ceph-mon[40961]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:57.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:57 smithi114 ceph-mon[40961]: from='client.25331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:57 smithi114 ceph-mon[29589]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:57 smithi114 ceph-mon[29589]: from='client.25331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:57 smithi160 ceph-mon[32702]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:57 smithi160 ceph-mon[32702]: from='client.25331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:46:58.841 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:58 smithi114 ceph-mon[40961]: pgmap v22: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:46:58.842 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:58 smithi114 ceph-mon[29589]: pgmap v22: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:46:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:58 smithi160 ceph-mon[32702]: pgmap v22: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:46:59.120 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:58 smithi114 bash[92134]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-14T03:46:59.120 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:59 smithi114 bash[92134]: Writing manifest to image destination 2024-09-14T03:46:59.852 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:46:59.853 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:59 smithi114 podman[92134]: 2024-09-14 03:46:59.330695371 +0000 UTC m=+2.901778426 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-14T03:46:59.853 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:46:59 smithi114 podman[92134]: 2024-09-14 03:46:59.444725818 +0000 UTC m=+3.015808857 container create ff1fe8404b338c6750d27388968d983e7e561607d00ec025fcf13c2a69394684 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T03:46:59.853 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:46:59 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3590883418' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:59.854 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:46:59 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:46:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:46:59.854 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:46:59 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3590883418' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:46:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:46:59 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3590883418' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:00.122 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 podman[92134]: 2024-09-14 03:47:00.056282626 +0000 UTC m=+3.627365681 container init ff1fe8404b338c6750d27388968d983e7e561607d00ec025fcf13c2a69394684 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T03:47:00.123 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 podman[92134]: 2024-09-14 03:47:00.067332197 +0000 UTC m=+3.638415234 container start ff1fe8404b338c6750d27388968d983e7e561607d00ec025fcf13c2a69394684 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T03:47:00.123 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.073Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-14T03:47:00.123 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.073Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-14T03:47:00.123 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.075Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-14T03:47:00.123 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.075Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-14T03:47:00.123 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.075Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-14T03:47:00.123 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.075Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-14T03:47:00.123 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-14T03:47:00.123 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=arp 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=edac 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-14T03:47:00.124 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-14T03:47:00.125 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=os 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=stat 2024-09-14T03:47:00.126 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-14T03:47:00.127 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-14T03:47:00.127 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-14T03:47:00.127 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=time 2024-09-14T03:47:00.127 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-14T03:47:00.127 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=uname 2024-09-14T03:47:00.127 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-14T03:47:00.127 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-14T03:47:00.127 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.076Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-14T03:47:00.127 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.077Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-14T03:47:00.127 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a[92252]: ts=2024-09-14T03:47:00.077Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-14T03:47:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:00 smithi160 ceph-mon[32702]: pgmap v23: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 729 op/s 2024-09-14T03:47:00.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:00 smithi160 ceph-mon[32702]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:00.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-mon[40961]: pgmap v23: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 729 op/s 2024-09-14T03:47:00.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-mon[40961]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:00.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-mon[29589]: pgmap v23: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 729 op/s 2024-09-14T03:47:00.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:00 smithi114 ceph-mon[29589]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:01.370 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:01 smithi114 bash[92134]: ff1fe8404b338c6750d27388968d983e7e561607d00ec025fcf13c2a69394684 2024-09-14T03:47:01.370 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 03:47:01 smithi114 systemd[1]: Started Ceph node-exporter.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:47:01.865 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:47:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:47:01] "GET /metrics HTTP/1.1" 200 36200 "" "Prometheus/2.43.0" 2024-09-14T03:47:01.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:01 smithi160 ceph-mon[32702]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:01.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:01 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:01.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:01 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:01.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:01 smithi114 ceph-mon[40961]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:01.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:01 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:01.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:01 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:01.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:01 smithi114 ceph-mon[29589]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:01.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:01 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:01.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:01 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:02.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:02 smithi114 ceph-mon[29589]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-14T03:47:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:02 smithi114 ceph-mon[29589]: pgmap v24: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:47:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:02 smithi114 ceph-mon[29589]: Deploying daemon prometheus.a on smithi160 2024-09-14T03:47:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:02 smithi114 ceph-mon[40961]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-14T03:47:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:02 smithi114 ceph-mon[40961]: pgmap v24: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:47:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:02 smithi114 ceph-mon[40961]: Deploying daemon prometheus.a on smithi160 2024-09-14T03:47:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:02 smithi160 ceph-mon[32702]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-14T03:47:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:02 smithi160 ceph-mon[32702]: pgmap v24: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:47:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:02 smithi160 ceph-mon[32702]: Deploying daemon prometheus.a on smithi160 2024-09-14T03:47:03.296 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:03.516 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 03:47:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T03:47:03.130Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004027351s 2024-09-14T03:47:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:03 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1758766677' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:03.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:03 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1758766677' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:03.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:03 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1758766677' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:04.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:04 smithi114 ceph-mon[29589]: from='client.25361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:04.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:04 smithi114 ceph-mon[29589]: pgmap v25: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-14T03:47:04.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:04 smithi114 ceph-mon[29589]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:04 smithi114 ceph-mon[40961]: from='client.25361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:04 smithi114 ceph-mon[40961]: pgmap v25: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-14T03:47:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:04 smithi114 ceph-mon[40961]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:04 smithi160 ceph-mon[32702]: from='client.25361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:04 smithi160 ceph-mon[32702]: pgmap v25: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-14T03:47:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:04 smithi160 ceph-mon[32702]: from='client.25069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:06.750 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:06.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:06 smithi114 ceph-mon[40961]: pgmap v26: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-14T03:47:06.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:06 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3115548985' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:06.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:06 smithi114 ceph-mon[29589]: pgmap v26: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-14T03:47:06.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:06 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3115548985' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:06.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:06 smithi160 ceph-mon[32702]: pgmap v26: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-14T03:47:06.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:06 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3115548985' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:07.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:07 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:07.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:07 smithi114 ceph-mon[29589]: from='client.25376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:07 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:07 smithi114 ceph-mon[40961]: from='client.25376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:07 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:07 smithi160 ceph-mon[32702]: from='client.25376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:08 smithi114 ceph-mon[40961]: from='client.25382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:08 smithi114 ceph-mon[40961]: pgmap v27: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:47:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:08 smithi114 ceph-mon[29589]: from='client.25382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:08 smithi114 ceph-mon[29589]: pgmap v27: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:47:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:08 smithi160 ceph-mon[32702]: from='client.25382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:08 smithi160 ceph-mon[32702]: pgmap v27: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:47:09.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:09 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[29838]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:47:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-14T03:47:10.286 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:10 smithi160 ceph-mon[32702]: pgmap v28: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 680 op/s 2024-09-14T03:47:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:10 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2969540753' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:10.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:10 smithi114 ceph-mon[40961]: pgmap v28: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 680 op/s 2024-09-14T03:47:10.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:10 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2969540753' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:10.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:10 smithi114 ceph-mon[29589]: pgmap v28: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 680 op/s 2024-09-14T03:47:10.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:10 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2969540753' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:11.658 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:47:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:47:11] "GET /metrics HTTP/1.1" 200 36199 "" "Prometheus/2.43.0" 2024-09-14T03:47:11.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:11 smithi160 ceph-mon[32702]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:11.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:11 smithi160 ceph-mon[32702]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:11.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:11 smithi114 ceph-mon[40961]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:11.914 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:11 smithi114 ceph-mon[40961]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:11 smithi114 ceph-mon[29589]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:11 smithi114 ceph-mon[29589]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:12.751 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:12 smithi160 ceph-mon[32702]: pgmap v29: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:47:12.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:12 smithi114 ceph-mon[29589]: pgmap v29: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:47:12.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:12 smithi114 ceph-mon[40961]: pgmap v29: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:47:13.499 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 systemd[1]: Stopping Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:47:13.844 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:13.845 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:13 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/4266082442' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:13.849 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:13 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/4266082442' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/4266082442' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:13.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.684Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-14T03:47:13.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.685Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-14T03:47:13.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.685Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-14T03:47:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.685Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-14T03:47:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.685Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-14T03:47:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.685Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-14T03:47:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.685Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-14T03:47:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.685Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-14T03:47:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.685Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-14T03:47:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.686Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-14T03:47:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.686Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-14T03:47:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[65642]: ts=2024-09-14T03:47:13.686Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-14T03:47:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:13 smithi160 podman[76511]: 2024-09-14 03:47:13.70234522 +0000 UTC m=+0.168655811 container died d05492e182231bf1ad8d4d51c2c6b70679b74993a5e248d69fbb68bafd49f77e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:47:14.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:14 smithi160 podman[76511]: 2024-09-14 03:47:14.231705629 +0000 UTC m=+0.698016196 container cleanup d05492e182231bf1ad8d4d51c2c6b70679b74993a5e248d69fbb68bafd49f77e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:47:14.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:14 smithi160 bash[76511]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a 2024-09-14T03:47:14.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:14 smithi114 ceph-mon[29589]: pgmap v30: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 675 op/s 2024-09-14T03:47:14.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:14 smithi114 ceph-mon[29589]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:14.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:14 smithi114 ceph-mon[40961]: pgmap v30: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 675 op/s 2024-09-14T03:47:14.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:14 smithi114 ceph-mon[40961]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:14.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:14 smithi160 podman[76527]: 2024-09-14 03:47:14.483955272 +0000 UTC m=+0.780391219 container remove d05492e182231bf1ad8d4d51c2c6b70679b74993a5e248d69fbb68bafd49f77e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:47:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:14 smithi160 ceph-mon[32702]: pgmap v30: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 675 op/s 2024-09-14T03:47:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:14 smithi160 ceph-mon[32702]: from='client.15495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:15.244 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:14 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-14T03:47:15.244 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:14 smithi160 systemd[1]: Stopped Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:47:15.244 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:14 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 2.216s CPU time. 2024-09-14T03:47:15.244 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 systemd[1]: Starting Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:47:15.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:15 smithi114 ceph-mon[40961]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:15.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:15 smithi114 ceph-mon[29589]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:15.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 podman[76647]: 2024-09-14 03:47:15.485583377 +0000 UTC m=+0.034655186 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-14T03:47:15.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 podman[76647]: 2024-09-14 03:47:15.628304476 +0000 UTC m=+0.177376293 container create 11966e02e35850cd153d1646c21a0d7b95035e80945d0ded7e2298c35e2219e9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:47:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-mon[32702]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:16.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 podman[76647]: 2024-09-14 03:47:15.913356969 +0000 UTC m=+0.462428785 container init 11966e02e35850cd153d1646c21a0d7b95035e80945d0ded7e2298c35e2219e9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:47:16.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 podman[76647]: 2024-09-14 03:47:15.920378506 +0000 UTC m=+0.469450331 container start 11966e02e35850cd153d1646c21a0d7b95035e80945d0ded7e2298c35e2219e9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:47:16.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.964Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-14T03:47:16.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.964Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-14T03:47:16.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.964Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi160 (none))" 2024-09-14T03:47:16.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.964Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.964Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.970Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.971Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.972Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.973Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.975Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.975Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=5.291µs 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.975Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:15.976Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 bash[76647]: 11966e02e35850cd153d1646c21a0d7b95035e80945d0ded7e2298c35e2219e9 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:16.025Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-14T03:47:16.410 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:16.026Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-14T03:47:16.411 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:16.026Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=38.774µs wal_replay_duration=50.608565ms wbl_replay_duration=143ns total_replay_duration=50.722741ms 2024-09-14T03:47:16.411 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:16.031Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-14T03:47:16.411 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:16.031Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-14T03:47:16.411 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:16.031Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-14T03:47:16.411 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 systemd[1]: Started Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:47:16.411 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:16.063Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=31.618905ms db_storage=1.111µs remote_storage=1.668µs web_handler=373ns query_engine=1.213µs scrape=658.48µs scrape_sd=202.346µs notify=15.258µs notify_sd=3.990044ms rules=26.057151ms tracing=8.488µs 2024-09-14T03:47:16.411 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:16.063Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-14T03:47:16.411 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:16.063Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-14T03:47:16.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:16 smithi114 ceph-mon[29589]: pgmap v31: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 675 op/s 2024-09-14T03:47:16.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:16 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:16 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:16 smithi114 ceph-mon[40961]: pgmap v31: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 675 op/s 2024-09-14T03:47:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:16 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:16 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-mon[32702]: pgmap v31: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 675 op/s 2024-09-14T03:47:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:16 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:17.334 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:17.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:17 smithi114 ceph-mon[40961]: Reconfiguring grafana.a (dependencies changed)... 2024-09-14T03:47:17.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:17 smithi114 ceph-mon[40961]: Reconfiguring daemon grafana.a on smithi160 2024-09-14T03:47:17.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:17 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1622504229' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:17 smithi114 ceph-mon[29589]: Reconfiguring grafana.a (dependencies changed)... 2024-09-14T03:47:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:17 smithi114 ceph-mon[29589]: Reconfiguring daemon grafana.a on smithi160 2024-09-14T03:47:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:17 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1622504229' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:17.893 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:17 smithi160 ceph-mon[32702]: Reconfiguring grafana.a (dependencies changed)... 2024-09-14T03:47:17.894 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:17 smithi160 ceph-mon[32702]: Reconfiguring daemon grafana.a on smithi160 2024-09-14T03:47:17.894 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:17 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1622504229' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:18.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:18 smithi114 ceph-mon[29589]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:18.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:18 smithi114 ceph-mon[29589]: pgmap v32: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-14T03:47:18.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:18 smithi114 ceph-mon[29589]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:18 smithi114 ceph-mon[40961]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:18 smithi114 ceph-mon[40961]: pgmap v32: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-14T03:47:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:18 smithi114 ceph-mon[40961]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:18 smithi160 ceph-mon[32702]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:18 smithi160 ceph-mon[32702]: pgmap v32: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-14T03:47:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:18 smithi160 ceph-mon[32702]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:20.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:20 smithi114 ceph-mon[40961]: pgmap v33: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T03:47:20.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:20 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3404006250' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:20.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:20 smithi114 ceph-mon[29589]: pgmap v33: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T03:47:20.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:20 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3404006250' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:20.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:20 smithi160 ceph-mon[32702]: pgmap v33: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T03:47:20.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:20 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3404006250' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:20.925 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:21.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:21 smithi114 ceph-mon[29589]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:21.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:21 smithi114 ceph-mon[40961]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:21.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:21 smithi160 ceph-mon[32702]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:22.622 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:22 smithi160 systemd[1]: Stopping Ceph grafana.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:47:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:22 smithi160 ceph-mon[32702]: pgmap v34: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-14T03:47:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:22 smithi160 ceph-mon[32702]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:22 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:22.908 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:22 smithi160 podman[76888]: 2024-09-14 03:47:22.747675815 +0000 UTC m=+0.170688099 container died cb3fc2419525356e2a30c1dc0253b4907e3a9318c688fadbd15a6dd1eaec3001 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, architecture=x86_64, io.openshift.tags=base rhel8, version=8.5, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0) 2024-09-14T03:47:23.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:22 smithi114 ceph-mon[40961]: pgmap v34: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-14T03:47:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:22 smithi114 ceph-mon[40961]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:22 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:23.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:22 smithi114 ceph-mon[29589]: pgmap v34: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-14T03:47:23.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:22 smithi114 ceph-mon[29589]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:23.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:22 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:24.589 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:24.625 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:24 smithi160 podman[76888]: 2024-09-14 03:47:24.213677458 +0000 UTC m=+1.636689739 container cleanup cb3fc2419525356e2a30c1dc0253b4907e3a9318c688fadbd15a6dd1eaec3001 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, name=ubi8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, vcs-type=git) 2024-09-14T03:47:24.625 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:24 smithi160 bash[76888]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a 2024-09-14T03:47:24.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:24 smithi114 ceph-mon[40961]: pgmap v35: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T03:47:24.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:24 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1302953172' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:24.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:24 smithi114 ceph-mon[29589]: pgmap v35: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T03:47:24.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:24 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1302953172' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:24 smithi160 ceph-mon[32702]: pgmap v35: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T03:47:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:24 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1302953172' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:25.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:25 smithi160 podman[76902]: 2024-09-14 03:47:25.032830427 +0000 UTC m=+2.285518199 container remove cb3fc2419525356e2a30c1dc0253b4907e3a9318c688fadbd15a6dd1eaec3001 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, description=Ceph Grafana Container, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, name=ubi8, vcs-type=git, vendor=Red Hat, Inc., io.openshift.expose-services=, architecture=x86_64, build-date=2022-04-29T11:48:11.055897) 2024-09-14T03:47:25.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:25 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-14T03:47:25.822 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:25 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-14T03:47:25.822 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:25 smithi160 systemd[1]: Stopped Ceph grafana.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:47:25.822 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:25 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@grafana.a.service: Consumed 6.458s CPU time. 2024-09-14T03:47:25.822 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:25 smithi160 systemd[1]: Starting Ceph grafana.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:47:25.823 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:25 smithi160 ceph-mon[32702]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:26.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:25 smithi114 ceph-mon[40961]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:26.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:25 smithi114 ceph-mon[29589]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:26.129 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:25 smithi160 podman[76986]: 2024-09-14 03:47:25.72612822 +0000 UTC m=+0.036750153 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-14T03:47:26.129 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:25 smithi160 podman[76986]: 2024-09-14 03:47:25.833940291 +0000 UTC m=+0.144562215 container create aa46fcb7cedcf780313941f07c0f806bb5e42b0077760592a68712a4b5462f62 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, io.openshift.expose-services=, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, vcs-type=git, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, distribution-scope=public, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc.) 2024-09-14T03:47:26.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 podman[76986]: 2024-09-14 03:47:26.128629981 +0000 UTC m=+0.439251915 container init aa46fcb7cedcf780313941f07c0f806bb5e42b0077760592a68712a4b5462f62 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, io.buildah.version=1.32.0, description=Ceph Grafana Container, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8) 2024-09-14T03:47:26.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 podman[76986]: 2024-09-14 03:47:26.136821575 +0000 UTC m=+0.447443499 container start aa46fcb7cedcf780313941f07c0f806bb5e42b0077760592a68712a4b5462f62 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, io.buildah.version=1.32.0, description=Ceph Grafana Container, vendor=Red Hat, Inc., name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, distribution-scope=public, architecture=x86_64, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner ) 2024-09-14T03:47:26.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 bash[76986]: aa46fcb7cedcf780313941f07c0f806bb5e42b0077760592a68712a4b5462f62 2024-09-14T03:47:26.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 systemd[1]: Started Ceph grafana.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:47:26.913 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-mon[32702]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:26.913 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-mon[32702]: pgmap v36: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T03:47:26.913 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:26.913 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:26.913 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.461946875Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-14T03:47:26.914 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462435137Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-14T03:47:26.914 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462498532Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-14T03:47:26.914 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462539846Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-14T03:47:26.914 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462576181Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-14T03:47:26.914 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462610604Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-14T03:47:26.914 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462647747Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-14T03:47:26.914 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462682887Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-14T03:47:26.914 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462720066Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-14T03:47:26.914 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462756814Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-14T03:47:26.914 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462794272Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.46283726Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462875428Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=settings t=2024-09-14T03:47:26.462912123Z level=info msg="App mode production" 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=sqlstore t=2024-09-14T03:47:26.463017391Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=sqlstore t=2024-09-14T03:47:26.463074128Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=migrator t=2024-09-14T03:47:26.487833889Z level=info msg="Starting DB migrations" 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=migrator t=2024-09-14T03:47:26.495848861Z level=info msg="migrations completed" performed=0 skipped=481 duration=705.358µs 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=sqlstore t=2024-09-14T03:47:26.496649211Z level=info msg="Created default organization" 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=plugin.loader t=2024-09-14T03:47:26.594250034Z level=info msg="Plugin registered" pluginID=input 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=plugin.loader t=2024-09-14T03:47:26.638146502Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=plugin.loader t=2024-09-14T03:47:26.638391287Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=secrets t=2024-09-14T03:47:26.638543485Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-14T03:47:26.915 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=query_data t=2024-09-14T03:47:26.64114857Z level=info msg="Query Service initialization" 2024-09-14T03:47:26.916 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=live.push_http t=2024-09-14T03:47:26.658001718Z level=info msg="Live Push Gateway initialization" 2024-09-14T03:47:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:26 smithi114 ceph-mon[40961]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:26 smithi114 ceph-mon[40961]: pgmap v36: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T03:47:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:26 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:26 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:26 smithi114 ceph-mon[29589]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:26 smithi114 ceph-mon[29589]: pgmap v36: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T03:47:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:26 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:26 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:27.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:27.762 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:27 smithi160 ceph-mon[32702]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-14T03:47:27.762 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:27 smithi160 ceph-mon[32702]: Deploying daemon node-exporter.b on smithi160 2024-09-14T03:47:27.762 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:27 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/110679845' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:28.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:27 smithi114 ceph-mon[40961]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-14T03:47:28.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:27 smithi114 ceph-mon[40961]: Deploying daemon node-exporter.b on smithi160 2024-09-14T03:47:28.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:27 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/110679845' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:28.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:27 smithi114 ceph-mon[29589]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-14T03:47:28.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:27 smithi114 ceph-mon[29589]: Deploying daemon node-exporter.b on smithi160 2024-09-14T03:47:28.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:27 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/110679845' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:28.042 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:28.066 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:27 smithi160 systemd[1]: Stopping Ceph node-exporter.b for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:47:28.067 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:27 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=infra.usagestats.collector t=2024-09-14T03:47:27.975830259Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-14T03:47:28.067 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=provisioning.datasources t=2024-09-14T03:47:28.065959657Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-09-14T03:47:28.408 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:28 smithi160 podman[77141]: 2024-09-14 03:47:28.264079738 +0000 UTC m=+0.177398838 container died a97cccb21ef2351615d919194581484a99cd187bfcfac45877fec8f23f17f2b0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T03:47:28.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=provisioning.datasources t=2024-09-14T03:47:28.066002931Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-09-14T03:47:28.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=provisioning.datasources t=2024-09-14T03:47:28.076739939Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-14T03:47:28.408 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=provisioning.alerting t=2024-09-14T03:47:28.17860062Z level=info msg="starting to provision alerting" 2024-09-14T03:47:28.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=provisioning.alerting t=2024-09-14T03:47:28.178858113Z level=info msg="finished to provision alerting" 2024-09-14T03:47:28.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=grafanaStorageLogger t=2024-09-14T03:47:28.179181365Z level=info msg="storage starting" 2024-09-14T03:47:28.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=ngalert.state.manager t=2024-09-14T03:47:28.179940379Z level=info msg="Warming state cache for startup" 2024-09-14T03:47:28.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=http.server t=2024-09-14T03:47:28.181791846Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-14T03:47:28.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=ngalert.state.manager t=2024-09-14T03:47:28.190517062Z level=info msg="State cache has been initialized" states=0 duration=10.575102ms 2024-09-14T03:47:28.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=ngalert.multiorg.alertmanager t=2024-09-14T03:47:28.190791888Z level=info msg="starting MultiOrg Alertmanager" 2024-09-14T03:47:28.409 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=ticker t=2024-09-14T03:47:28.19096453Z level=info msg=starting first_tick=2024-09-14T03:47:30Z 2024-09-14T03:47:28.896 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-mon[32702]: pgmap v37: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-14T03:47:28.897 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:47:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:47:28] "GET /metrics HTTP/1.1" 200 36198 "" "Prometheus/2.51.0" 2024-09-14T03:47:28.897 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:28 smithi160 podman[77141]: 2024-09-14 03:47:28.677014166 +0000 UTC m=+0.590333272 container cleanup a97cccb21ef2351615d919194581484a99cd187bfcfac45877fec8f23f17f2b0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T03:47:28.897 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:28 smithi160 bash[77141]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b 2024-09-14T03:47:29.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:28 smithi114 ceph-mon[40961]: pgmap v37: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-14T03:47:29.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:28 smithi114 ceph-mon[29589]: pgmap v37: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-14T03:47:29.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:28 smithi160 podman[77158]: 2024-09-14 03:47:28.895444746 +0000 UTC m=+0.632391208 container remove a97cccb21ef2351615d919194581484a99cd187bfcfac45877fec8f23f17f2b0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T03:47:29.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:28 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-14T03:47:29.631 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:29 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-14T03:47:29.631 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:29 smithi160 systemd[1]: Stopped Ceph node-exporter.b for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:47:29.631 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:29 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 2.693s CPU time. 2024-09-14T03:47:29.631 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:29 smithi160 systemd[1]: Starting Ceph node-exporter.b for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:47:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:29 smithi160 ceph-mon[32702]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:29 smithi160 ceph-mon[32702]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:30.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:29 smithi114 ceph-mon[40961]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:29 smithi114 ceph-mon[40961]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:30.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:29 smithi114 ceph-mon[29589]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:30.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:29 smithi114 ceph-mon[29589]: from='client.15561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:30.396 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:30 smithi160 bash[77271]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-14T03:47:30.658 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:30 smithi160 bash[77271]: Getting image source signatures 2024-09-14T03:47:30.658 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:30 smithi160 bash[77271]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-14T03:47:30.658 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:30 smithi160 bash[77271]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-14T03:47:30.658 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:30 smithi160 bash[77271]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-14T03:47:31.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:30 smithi114 ceph-mon[40961]: pgmap v38: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 706 op/s 2024-09-14T03:47:31.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:30 smithi114 ceph-mon[29589]: pgmap v38: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 706 op/s 2024-09-14T03:47:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:30 smithi160 ceph-mon[32702]: pgmap v38: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 706 op/s 2024-09-14T03:47:31.549 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:31.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:31 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1962746348' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:31.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:31 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1962746348' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:32.131 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:31 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1962746348' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:32.408 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:32 smithi160 bash[77271]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-14T03:47:32.900 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:32 smithi160 bash[77271]: Writing manifest to image destination 2024-09-14T03:47:32.900 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:32 smithi160 ceph-mon[32702]: pgmap v39: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-14T03:47:32.900 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:32 smithi160 ceph-mon[32702]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:32.901 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:32 smithi160 ceph-mon[32702]: from='client.25150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:33.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:32 smithi114 ceph-mon[29589]: pgmap v39: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-14T03:47:33.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:32 smithi114 ceph-mon[29589]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:33.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:32 smithi114 ceph-mon[29589]: from='client.25150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:33.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:32 smithi114 ceph-mon[40961]: pgmap v39: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-14T03:47:33.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:32 smithi114 ceph-mon[40961]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:33.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:32 smithi114 ceph-mon[40961]: from='client.25150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:33.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:32 smithi160 podman[77271]: 2024-09-14 03:47:32.829645348 +0000 UTC m=+2.980862538 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-14T03:47:33.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:32 smithi160 podman[77271]: 2024-09-14 03:47:32.930955901 +0000 UTC m=+3.082173084 container create 289de68494ecd175eb9b4e216c16393502e5a229ef26b3d6bae03a265e7a3133 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T03:47:33.517 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 podman[77271]: 2024-09-14 03:47:33.238741673 +0000 UTC m=+3.389958853 container init 289de68494ecd175eb9b4e216c16393502e5a229ef26b3d6bae03a265e7a3133 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T03:47:33.517 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 podman[77271]: 2024-09-14 03:47:33.246902808 +0000 UTC m=+3.398119984 container start 289de68494ecd175eb9b4e216c16393502e5a229ef26b3d6bae03a265e7a3133 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T03:47:33.517 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.251Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-14T03:47:33.517 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.251Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-14T03:47:33.517 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.252Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-14T03:47:33.517 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.252Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-14T03:47:33.517 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-14T03:47:33.517 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-14T03:47:33.517 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=arp 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=edac 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-14T03:47:33.518 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.253Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-14T03:47:33.519 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=os 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-14T03:47:33.520 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=stat 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=time 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=uname 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.254Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-14T03:47:33.521 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.255Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-14T03:47:33.522 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b[77349]: ts=2024-09-14T03:47:33.255Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-14T03:47:33.522 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 bash[77271]: 289de68494ecd175eb9b4e216c16393502e5a229ef26b3d6bae03a265e7a3133 2024-09-14T03:47:33.908 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 03:47:33 smithi160 systemd[1]: Started Ceph node-exporter.b for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:47:34.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:34.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:34.781 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: pgmap v40: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 691 op/s 2024-09-14T03:47:34.781 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:34.781 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:34.781 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:47:34.781 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:47:34.781 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:47:34.781 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:47:34.781 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/535817430' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: pgmap v40: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 691 op/s 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:47:34.782 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:47:34.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:47:34.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:47:34.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:47:34.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:34.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-14T03:47:34.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-14T03:47:34.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:34 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/535817430' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: pgmap v40: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 691 op/s 2024-09-14T03:47:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:47:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-14T03:47:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:47:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:47:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:47:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-14T03:47:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-14T03:47:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-14T03:47:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:34 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/535817430' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:35.139 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:36.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:36.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:37.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:37.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: pgmap v41: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 691 op/s 2024-09-14T03:47:37.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:37.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: Upgrade: Updating mgr.y 2024-09-14T03:47:37.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:37.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:47:37.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:47:37.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:47:37.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:47:37.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: Deploying daemon mgr.y on smithi114 2024-09-14T03:47:37.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:37.125 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:37.125 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: pgmap v41: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 691 op/s 2024-09-14T03:47:37.126 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:37.126 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: Upgrade: Updating mgr.y 2024-09-14T03:47:37.126 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:37.126 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:47:37.126 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:47:37.126 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:47:37.126 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:47:37.126 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: Deploying daemon mgr.y on smithi114 2024-09-14T03:47:37.126 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:36 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:37.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:37.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:37.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: pgmap v41: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 691 op/s 2024-09-14T03:47:37.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:37.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: Upgrade: Updating mgr.y 2024-09-14T03:47:37.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:37.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:47:37.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:47:37.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:47:37.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:47:37.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: Deploying daemon mgr.y on smithi114 2024-09-14T03:47:37.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:36 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:38.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:38 smithi114 systemd[1]: Stopping Ceph mgr.y for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:47:38.740 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:38.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:38 smithi114 podman[93348]: 2024-09-14 03:47:38.568959441 +0000 UTC m=+0.182388039 container died 57c15055b7ead15cffc93972c5164187465a5e69c4db9e29f495606c140c40c0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-eb16b97, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2) 2024-09-14T03:47:38.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:47:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:47:38] "GET /metrics HTTP/1.1" 200 36195 "" "Prometheus/2.51.0" 2024-09-14T03:47:39.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:38 smithi114 ceph-mon[40961]: pgmap v42: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:47:39.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:38 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2469379722' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:39.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:38 smithi114 ceph-mon[29589]: pgmap v42: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:47:39.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:38 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2469379722' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:39.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:39 smithi114 podman[93348]: 2024-09-14 03:47:39.072408447 +0000 UTC m=+0.685837043 container cleanup 57c15055b7ead15cffc93972c5164187465a5e69c4db9e29f495606c140c40c0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-14T03:47:39.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:39 smithi114 bash[93348]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y 2024-09-14T03:47:39.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:38 smithi160 ceph-mon[32702]: pgmap v42: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:47:39.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:38 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2469379722' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:40.051 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:39 smithi114 ceph-mon[40961]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:40.051 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:39 smithi114 podman[93363]: 2024-09-14 03:47:39.783778593 +0000 UTC m=+1.204743449 container remove 57c15055b7ead15cffc93972c5164187465a5e69c4db9e29f495606c140c40c0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y, RELEASE=quincy-eb16b97, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-14T03:47:40.051 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:39 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-14T03:47:40.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:39 smithi114 ceph-mon[29589]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:40.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:40 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-14T03:47:40.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:40 smithi114 systemd[1]: Stopped Ceph mgr.y for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:47:40.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:40 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.y.service: Consumed 43.806s CPU time. 2024-09-14T03:47:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:39 smithi160 ceph-mon[32702]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:40.713 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:40 smithi114 systemd[1]: Starting Ceph mgr.y for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:47:40.985 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:40 smithi114 podman[93509]: 2024-09-14 03:47:40.618387637 +0000 UTC m=+0.027733974 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:47:41.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:40 smithi114 ceph-mon[29589]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:41.314 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:40 smithi114 ceph-mon[29589]: pgmap v43: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 680 op/s 2024-09-14T03:47:41.314 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:40 smithi114 ceph-mon[40961]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:41.315 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:40 smithi114 ceph-mon[40961]: pgmap v43: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 680 op/s 2024-09-14T03:47:41.315 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:41 smithi114 podman[93509]: 2024-09-14 03:47:41.043959906 +0000 UTC m=+0.453306219 container create 4f6e65854aef018510965487055eab011e7b65549dde665524b668e4b38955d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-a278231, GIT_CLEAN=True) 2024-09-14T03:47:41.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:40 smithi160 ceph-mon[32702]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:41.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:40 smithi160 ceph-mon[32702]: pgmap v43: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 680 op/s 2024-09-14T03:47:41.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:41 smithi114 podman[93509]: 2024-09-14 03:47:41.313776303 +0000 UTC m=+0.723122622 container init 4f6e65854aef018510965487055eab011e7b65549dde665524b668e4b38955d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=main-a278231, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-14T03:47:41.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:41 smithi114 podman[93509]: 2024-09-14 03:47:41.324292228 +0000 UTC m=+0.733638550 container start 4f6e65854aef018510965487055eab011e7b65549dde665524b668e4b38955d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y, RELEASE=main-a278231, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:47:41.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:41 smithi114 ceph-mgr[93546]: -- 172.21.15.114:0/3737016614 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560c9032f0e0 con 0x560c9030d400 2024-09-14T03:47:41.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:41 smithi114 bash[93509]: 4f6e65854aef018510965487055eab011e7b65549dde665524b668e4b38955d0 2024-09-14T03:47:41.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:41 smithi114 systemd[1]: Started Ceph mgr.y for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:47:42.460 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:42 smithi160 ceph-mon[32702]: pgmap v44: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-14T03:47:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:42 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:42 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:42 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:42.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:42 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2703894141' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:42.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:42 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:42 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[29589]: pgmap v44: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-14T03:47:42.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2703894141' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[40961]: pgmap v44: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-14T03:47:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2703894141' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:42.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:42.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:42 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:43.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:43 smithi114 ceph-mon[29589]: from='client.25186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:43.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:43 smithi114 ceph-mon[40961]: from='client.25186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:43 smithi160 ceph-mon[32702]: from='client.25186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:44.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:44.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:44.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:44 smithi114 ceph-mon[29589]: from='client.25514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:44.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:44 smithi114 ceph-mon[29589]: pgmap v45: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-14T03:47:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:44 smithi114 ceph-mon[40961]: from='client.25514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:44 smithi114 ceph-mon[40961]: pgmap v45: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-14T03:47:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:44 smithi160 ceph-mon[32702]: from='client.25514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:44 smithi160 ceph-mon[32702]: pgmap v45: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-14T03:47:46.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:46.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:46.206 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:46.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:46 smithi114 ceph-mon[40961]: pgmap v46: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-14T03:47:46.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:46 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/274489248' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:46.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:46 smithi114 ceph-mon[29589]: pgmap v46: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-14T03:47:46.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:46 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/274489248' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:46 smithi160 ceph-mon[32702]: pgmap v46: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 701 op/s 2024-09-14T03:47:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:46 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/274489248' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:47.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:46.952Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:48.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[29589]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:48.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[29589]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:48.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[40961]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[40961]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:47 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:47 smithi160 ceph-mon[32702]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:47 smithi160 ceph-mon[32702]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:47 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:47 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:47 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:47 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:47 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:48.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:47:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:47:48] "GET /metrics HTTP/1.1" 200 36195 "" "Prometheus/2.51.0" 2024-09-14T03:47:49.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:48 smithi114 ceph-mon[29589]: pgmap v47: 89 pgs: 89 active+clean; 2.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-14T03:47:49.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:48 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:49.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:48 smithi114 ceph-mon[40961]: pgmap v47: 89 pgs: 89 active+clean; 2.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-14T03:47:49.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:48 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:48 smithi160 ceph-mon[32702]: pgmap v47: 89 pgs: 89 active+clean; 2.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-14T03:47:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:48 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:49.953 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:50.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:49 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2322212303' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:49 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2322212303' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:49 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2322212303' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:50.487 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:47:50] ENGINE Bus STARTING 2024-09-14T03:47:50.487 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: CherryPy Checker: 2024-09-14T03:47:50.487 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: The Application mounted at '' has an empty config. 2024-09-14T03:47:50.487 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:50.487 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:47:50] ENGINE Serving on http://:::9283 2024-09-14T03:47:50.487 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:47:50] ENGINE Bus STARTED 2024-09-14T03:47:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[40961]: pgmap v48: 89 pgs: 89 active+clean; 2.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 683 op/s 2024-09-14T03:47:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[40961]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[40961]: Standby manager daemon y restarted 2024-09-14T03:47:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[40961]: Standby manager daemon y started 2024-09-14T03:47:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-14T03:47:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:47:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-14T03:47:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:47:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[40961]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:51.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[29589]: pgmap v48: 89 pgs: 89 active+clean; 2.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 683 op/s 2024-09-14T03:47:51.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[29589]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:51.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[29589]: Standby manager daemon y restarted 2024-09-14T03:47:51.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[29589]: Standby manager daemon y started 2024-09-14T03:47:51.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-14T03:47:51.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:47:51.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-14T03:47:51.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:47:51.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:51 smithi114 ceph-mon[29589]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:51 smithi160 ceph-mon[32702]: pgmap v48: 89 pgs: 89 active+clean; 2.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 683 op/s 2024-09-14T03:47:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:51 smithi160 ceph-mon[32702]: from='client.15660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:51 smithi160 ceph-mon[32702]: Standby manager daemon y restarted 2024-09-14T03:47:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:51 smithi160 ceph-mon[32702]: Standby manager daemon y started 2024-09-14T03:47:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:51 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-14T03:47:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:51 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:47:51.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:51 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-14T03:47:51.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:51 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:47:51.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:51 smithi160 ceph-mon[32702]: from='client.15663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:52.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:52 smithi114 ceph-mon[40961]: mgrmap e27: x(active, since 91s), standbys: y 2024-09-14T03:47:52.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:52 smithi114 ceph-mon[40961]: pgmap v49: 89 pgs: 89 active+clean; 2.2 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-14T03:47:52.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:52 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:52 smithi114 ceph-mon[29589]: mgrmap e27: x(active, since 91s), standbys: y 2024-09-14T03:47:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:52 smithi114 ceph-mon[29589]: pgmap v49: 89 pgs: 89 active+clean; 2.2 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-14T03:47:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:52 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:52 smithi160 ceph-mon[32702]: mgrmap e27: x(active, since 91s), standbys: y 2024-09-14T03:47:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:52 smithi160 ceph-mon[32702]: pgmap v49: 89 pgs: 89 active+clean; 2.2 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-14T03:47:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:52 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:53.035 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:47:52] ENGINE Bus STOPPING 2024-09-14T03:47:53.035 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:47:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-14T03:47:53.036 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:47:52] ENGINE Bus STOPPED 2024-09-14T03:47:53.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: mgrmap e28: x(active, since 92s), standbys: y 2024-09-14T03:47:53.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:53.036 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:47:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:47:52.748+0000 7f75a6df3640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-14T03:47:53.036 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:47:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ignoring --setuser ceph since I am not root 2024-09-14T03:47:53.036 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:47:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ignoring --setgroup ceph since I am not root 2024-09-14T03:47:53.036 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:47:52 smithi160 ceph-mgr[68671]: -- 172.21.15.160:0/1826853641 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ecb7378ea0 con 0x55ecb734d000 2024-09-14T03:47:53.037 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:47:52 smithi160 ceph-mgr[68671]: -- 172.21.15.160:0/1826853641 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ecb736f0e0 con 0x55ecb734d000 2024-09-14T03:47:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: mgrmap e28: x(active, since 92s), standbys: y 2024-09-14T03:47:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:47:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:47:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: Activating manager daemon y 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: osdmap e70: 8 total, 8 up, 8 in 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.15207 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: mgrmap e29: y(active, starting, since 0.0244875s) 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"}]: dispatch 2024-09-14T03:47:53.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:47:53.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[40961]: Manager daemon y is now available 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: Activating manager daemon y 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: osdmap e70: 8 total, 8 up, 8 in 2024-09-14T03:47:53.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.15207 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: mgrmap e29: y(active, starting, since 0.0244875s) 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:47:53.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:47:53.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:47:53.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:47:53.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:47:53.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:47:53.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:47:53.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-mon[29589]: Manager daemon y is now available 2024-09-14T03:47:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: mgrmap e28: x(active, since 92s), standbys: y 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.15207 172.21.15.160:0/120688383' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: Activating manager daemon y 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: osdmap e70: 8 total, 8 up, 8 in 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.15207 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-14T03:47:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: mgrmap e29: y(active, starting, since 0.0244875s) 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-14T03:47:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-14T03:47:53.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-14T03:47:53.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-14T03:47:53.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-14T03:47:53.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:53 smithi160 ceph-mon[32702]: Manager daemon y is now available 2024-09-14T03:47:53.775 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:54.037 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:47:53] ENGINE Bus STARTING 2024-09-14T03:47:54.037 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:53.920+0000 7f4c9b97b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:47:54.037 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:53.920+0000 7f4c9b97b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:47:54.037 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:53.920+0000 7f4c9b97b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:47:54.037 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:53.920+0000 7f4c9b97b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:47:54.037 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:53.920+0000 7f4c9b97b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:47:54.037 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:53.924+0000 7f4c92569640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:47:54.037 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:53.924+0000 7f4c92569640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:47:54.038 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:53.924+0000 7f4c92569640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:47:54.038 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:53.924+0000 7f4c92569640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:47:54.038 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:53.924+0000 7f4c92569640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:47:54.303 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:54.303 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:54 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/462894334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:54.303 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:54 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:54.303 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:54 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:47:54.304 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:54 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:54.304 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:54 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:47:54.304 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:54 smithi160 ceph-mon[32702]: mgrmap e30: y(active, since 1.01143s) 2024-09-14T03:47:54.304 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:54 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:47:54.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:47:54] ENGINE Serving on http://:::9283 2024-09-14T03:47:54.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:47:54] ENGINE Bus STARTED 2024-09-14T03:47:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/462894334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:47:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:47:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[40961]: mgrmap e30: y(active, since 1.01143s) 2024-09-14T03:47:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:47:54.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/462894334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:54.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:47:54.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:47:54.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:47:54.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-14T03:47:54.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[29589]: mgrmap e30: y(active, since 1.01143s) 2024-09-14T03:47:54.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:54 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-14T03:47:55.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[40961]: from='client.25547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:55.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[40961]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:47:55.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[40961]: pgmap v3: 89 pgs: 89 active+clean; 2.0 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-09-14T03:47:55.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[40961]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:55.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:55.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:55.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[29589]: from='client.25547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:55.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[29589]: pgmap v3: 89 pgs: 89 active+clean; 2.0 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[29589]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:55.198+0000 7f4c77834640 -1 log_channel(cephadm) log [ERR] : [14/Sep/2024:03:47:55] ENGINE Error in HTTPServer.serve 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: Traceback (most recent call last): 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: self._connections.run(self.expiration_interval) 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: self._run(expiration_interval) 2024-09-14T03:47:55.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: s = self.context.wrap_socket( 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: return self.sslsocket_class._create( 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: self.do_handshake() 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: self._sslobj.do_handshake() 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:47:55.373 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:47:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:47:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:55 smithi160 ceph-mon[32702]: from='client.25547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:55 smithi160 ceph-mon[32702]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:47:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:55 smithi160 ceph-mon[32702]: pgmap v3: 89 pgs: 89 active+clean; 2.0 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-09-14T03:47:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:55 smithi160 ceph-mon[32702]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:55 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:55 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:56.055 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:56.055 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:56.271 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[40961]: [14/Sep/2024:03:47:54] ENGINE Bus STARTING 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[40961]: mgrmap e31: y(active, since 2s) 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[40961]: [14/Sep/2024:03:47:55] ENGINE Serving on http://172.21.15.114:8765 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[40961]: [14/Sep/2024:03:47:55] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[40961]: [14/Sep/2024:03:47:55] ENGINE Bus STARTED 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[40961]: [14/Sep/2024:03:47:55] ENGINE Error in HTTPServer.serve 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout: Traceback (most recent call last): 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._run(expiration_interval) 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:47:56.272 INFO:journalctl@ceph.mon.c.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: s = self.context.wrap_socket( 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: return self.sslsocket_class._create( 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: self.do_handshake() 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._sslobj.do_handshake() 2024-09-14T03:47:56.273 INFO:journalctl@ceph.mon.c.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[29589]: [14/Sep/2024:03:47:54] ENGINE Bus STARTING 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[29589]: mgrmap e31: y(active, since 2s) 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[29589]: [14/Sep/2024:03:47:55] ENGINE Serving on http://172.21.15.114:8765 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[29589]: [14/Sep/2024:03:47:55] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[29589]: [14/Sep/2024:03:47:55] ENGINE Bus STARTED 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:56 smithi114 ceph-mon[29589]: [14/Sep/2024:03:47:55] ENGINE Error in HTTPServer.serve 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:47:56.274 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-14T03:47:56.275 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:47:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:56 smithi160 ceph-mon[32702]: [14/Sep/2024:03:47:54] ENGINE Bus STARTING 2024-09-14T03:47:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:56 smithi160 ceph-mon[32702]: mgrmap e31: y(active, since 2s) 2024-09-14T03:47:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:56 smithi160 ceph-mon[32702]: [14/Sep/2024:03:47:55] ENGINE Serving on http://172.21.15.114:8765 2024-09-14T03:47:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:56 smithi160 ceph-mon[32702]: [14/Sep/2024:03:47:55] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:47:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:56 smithi160 ceph-mon[32702]: [14/Sep/2024:03:47:55] ENGINE Bus STARTED 2024-09-14T03:47:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:56 smithi160 ceph-mon[32702]: [14/Sep/2024:03:47:55] ENGINE Error in HTTPServer.serve 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: Traceback (most recent call last): 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._run(expiration_interval) 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: s = self.context.wrap_socket( 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: return self.sslsocket_class._create( 2024-09-14T03:47:56.409 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:47:56.410 INFO:journalctl@ceph.mon.b.smithi160.stdout: self.do_handshake() 2024-09-14T03:47:56.410 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:47:56.410 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._sslobj.do_handshake() 2024-09-14T03:47:56.410 INFO:journalctl@ceph.mon.b.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:47:57.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:57 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1222441482' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:57.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:57 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1222441482' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:57.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:57 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1222441482' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:47:57.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:47:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:47:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:47:57.423 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:47:58.268 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:58 smithi160 ceph-mon[32702]: pgmap v5: 89 pgs: 89 active+clean; 2.0 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-09-14T03:47:58.268 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:58 smithi160 ceph-mon[32702]: mgrmap e32: y(active, since 4s) 2024-09-14T03:47:58.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:58 smithi114 ceph-mon[29589]: pgmap v5: 89 pgs: 89 active+clean; 2.0 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-09-14T03:47:58.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:58 smithi114 ceph-mon[29589]: mgrmap e32: y(active, since 4s) 2024-09-14T03:47:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:58 smithi114 ceph-mon[40961]: pgmap v5: 89 pgs: 89 active+clean; 2.0 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail 2024-09-14T03:47:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:58 smithi114 ceph-mon[40961]: mgrmap e32: y(active, since 4s) 2024-09-14T03:47:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:59 smithi160 ceph-mon[32702]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:59 smithi160 ceph-mon[32702]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:59 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:47:59 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:59.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:59 smithi114 ceph-mon[29589]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:59.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:59 smithi114 ceph-mon[29589]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:59 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:47:59 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:59 smithi114 ceph-mon[40961]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:59 smithi114 ceph-mon[40961]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:47:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:59 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:47:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:47:59 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:00.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:00 smithi114 ceph-mon[40961]: pgmap v6: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 238 op/s 2024-09-14T03:48:00.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:00 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:00.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:00 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:00 smithi114 ceph-mon[29589]: pgmap v6: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 238 op/s 2024-09-14T03:48:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:00 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:00 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:00.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:00 smithi160 ceph-mon[32702]: pgmap v6: 89 pgs: 89 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 238 op/s 2024-09-14T03:48:00.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:00 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:00.413 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:00 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:00.912 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:01 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1754993093' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:01 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:01 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:01.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:01 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1754993093' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:01.249 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:01 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:01.249 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:01 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:01.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:01 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1754993093' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:01.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:01 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:01.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:01 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:01.625 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:48:01] ENGINE Bus STARTING 2024-09-14T03:48:01.625 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: CherryPy Checker: 2024-09-14T03:48:01.625 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: The Application mounted at '' has an empty config. 2024-09-14T03:48:01.625 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: 2024-09-14T03:48:01.625 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:48:01] ENGINE Serving on http://:::9283 2024-09-14T03:48:01.625 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: [14/Sep/2024:03:48:01] ENGINE Bus STARTED 2024-09-14T03:48:02.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[29589]: pgmap v7: 89 pgs: 89 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 25 KiB/s rd, 295 B/s wr, 514 op/s 2024-09-14T03:48:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[29589]: from='client.25586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[29589]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[29589]: Standby manager daemon x started 2024-09-14T03:48:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:48:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:48:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:48:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:48:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:02.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[40961]: pgmap v7: 89 pgs: 89 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 25 KiB/s rd, 295 B/s wr, 514 op/s 2024-09-14T03:48:02.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[40961]: from='client.25586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:02.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[40961]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:02.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[40961]: Standby manager daemon x started 2024-09-14T03:48:02.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:48:02.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:48:02.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:48:02.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:48:02.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:02.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:02 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:02 smithi160 ceph-mon[32702]: pgmap v7: 89 pgs: 89 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 25 KiB/s rd, 295 B/s wr, 514 op/s 2024-09-14T03:48:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:02 smithi160 ceph-mon[32702]: from='client.25586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:02 smithi160 ceph-mon[32702]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:02 smithi160 ceph-mon[32702]: Standby manager daemon x started 2024-09-14T03:48:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:02 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:48:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:02 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:48:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:02 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:48:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:02 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/1025085506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:48:02.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:02 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:02.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:02 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:03.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:03 smithi114 ceph-mon[29589]: mgrmap e33: y(active, since 9s), standbys: x 2024-09-14T03:48:03.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:03 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:48:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:03 smithi114 ceph-mon[40961]: mgrmap e33: y(active, since 9s), standbys: x 2024-09-14T03:48:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:03 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:48:03.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:03 smithi160 ceph-mon[32702]: mgrmap e33: y(active, since 9s), standbys: x 2024-09-14T03:48:03.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:03 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-14T03:48:04.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:04 smithi114 ceph-mon[29589]: pgmap v8: 89 pgs: 89 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 229 B/s wr, 399 op/s 2024-09-14T03:48:04.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:04 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3922720564' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:04.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:04 smithi114 ceph-mon[40961]: pgmap v8: 89 pgs: 89 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 229 B/s wr, 399 op/s 2024-09-14T03:48:04.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:04 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3922720564' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:04.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:04 smithi160 ceph-mon[32702]: pgmap v8: 89 pgs: 89 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 20 KiB/s rd, 229 B/s wr, 399 op/s 2024-09-14T03:48:04.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:04 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3922720564' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:04.482 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:06.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[40961]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[40961]: pgmap v9: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 16 KiB/s rd, 187 B/s wr, 676 op/s 2024-09-14T03:48:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[40961]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:48:06.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[29589]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:06.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[29589]: pgmap v9: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 16 KiB/s rd, 187 B/s wr, 676 op/s 2024-09-14T03:48:06.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[29589]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:06.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:06.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:06.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:06.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:06 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:48:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:06 smithi160 ceph-mon[32702]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:06 smithi160 ceph-mon[32702]: pgmap v9: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 16 KiB/s rd, 187 B/s wr, 676 op/s 2024-09-14T03:48:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:06 smithi160 ceph-mon[32702]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:06 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:06 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:06 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:06 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:48:07.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:06 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:07.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:48:07.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:48:07.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:48:07.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:48:07.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:48:07.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:48:07.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:48:07.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:07 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:48:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:48:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:48:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:48:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:48:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:48:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:48:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:07.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:07 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:08.058 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:08.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:08 smithi160 ceph-mon[32702]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:48:08.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:08 smithi160 ceph-mon[32702]: pgmap v10: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 170 B/s wr, 616 op/s 2024-09-14T03:48:08.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:08 smithi160 ceph-mon[32702]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-14T03:48:08.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:08 smithi160 ceph-mon[32702]: Reconfiguring daemon prometheus.a on smithi160 2024-09-14T03:48:08.343 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:08 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1812794803' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:08.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:08 smithi114 ceph-mon[40961]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:48:08.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:08 smithi114 ceph-mon[40961]: pgmap v10: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 170 B/s wr, 616 op/s 2024-09-14T03:48:08.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:08 smithi114 ceph-mon[40961]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-14T03:48:08.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:08 smithi114 ceph-mon[40961]: Reconfiguring daemon prometheus.a on smithi160 2024-09-14T03:48:08.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:08 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1812794803' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:08.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:08 smithi114 ceph-mon[29589]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:48:08.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:08 smithi114 ceph-mon[29589]: pgmap v10: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 170 B/s wr, 616 op/s 2024-09-14T03:48:08.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:08 smithi114 ceph-mon[29589]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-14T03:48:08.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:08 smithi114 ceph-mon[29589]: Reconfiguring daemon prometheus.a on smithi160 2024-09-14T03:48:08.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:08 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1812794803' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:08.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[68646]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:48:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.51.0" 2024-09-14T03:48:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:09 smithi160 ceph-mon[32702]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:09 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:48:09.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:09 smithi114 ceph-mon[40961]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:09.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:09 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:48:09.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:09 smithi114 ceph-mon[29589]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:09.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:09 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:48:10.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:10.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:10 smithi160 ceph-mon[32702]: from='client.25619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:10.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:10 smithi160 ceph-mon[32702]: pgmap v11: 89 pgs: 89 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 170 B/s wr, 680 op/s 2024-09-14T03:48:10.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:10 smithi114 ceph-mon[40961]: from='client.25619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:10.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:10 smithi114 ceph-mon[40961]: pgmap v11: 89 pgs: 89 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 170 B/s wr, 680 op/s 2024-09-14T03:48:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:10 smithi114 ceph-mon[29589]: from='client.25619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:10 smithi114 ceph-mon[29589]: pgmap v11: 89 pgs: 89 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 170 B/s wr, 680 op/s 2024-09-14T03:48:11.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 systemd[1]: Stopping Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:48:11.657 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:11.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.594Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.594Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.594Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.594Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.594Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.594Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.594Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.594Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.595Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.596Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.596Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-14T03:48:11.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[76659]: ts=2024-09-14T03:48:11.596Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-14T03:48:11.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:11 smithi160 podman[79352]: 2024-09-14 03:48:11.607134133 +0000 UTC m=+0.150777891 container died 11966e02e35850cd153d1646c21a0d7b95035e80945d0ded7e2298c35e2219e9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:48:12.302 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:12 smithi160 ceph-mon[32702]: pgmap v12: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 170 B/s wr, 870 op/s 2024-09-14T03:48:12.302 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:12 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/246448176' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:12.302 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:12 smithi160 podman[79352]: 2024-09-14 03:48:12.024362147 +0000 UTC m=+0.568005903 container cleanup 11966e02e35850cd153d1646c21a0d7b95035e80945d0ded7e2298c35e2219e9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:48:12.302 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:12 smithi160 bash[79352]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a 2024-09-14T03:48:12.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:12 smithi114 ceph-mon[29589]: pgmap v12: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 170 B/s wr, 870 op/s 2024-09-14T03:48:12.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:12 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/246448176' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:12.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:12 smithi114 ceph-mon[40961]: pgmap v12: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 170 B/s wr, 870 op/s 2024-09-14T03:48:12.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:12 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/246448176' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:12.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:12 smithi160 podman[79368]: 2024-09-14 03:48:12.301102573 +0000 UTC m=+0.691524851 container remove 11966e02e35850cd153d1646c21a0d7b95035e80945d0ded7e2298c35e2219e9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:48:12.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:12 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-14T03:48:12.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:12 smithi160 systemd[1]: Stopped Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:48:12.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:12 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.600s CPU time. 2024-09-14T03:48:12.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:12 smithi160 systemd[1]: Starting Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:48:13.144 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:12 smithi160 podman[79447]: 2024-09-14 03:48:12.796714359 +0000 UTC m=+0.037378051 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-14T03:48:13.145 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 podman[79447]: 2024-09-14 03:48:13.119779503 +0000 UTC m=+0.360443186 container create 1b09e4c45bc1b7660d5d7d70676e48091df8354470af858ed276fe501987d09b (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:48:13.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-mon[32702]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:13.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-mon[32702]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:13.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:13 smithi114 ceph-mon[29589]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:13.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:13 smithi114 ceph-mon[29589]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:13 smithi114 ceph-mon[40961]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:13 smithi114 ceph-mon[40961]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 podman[79447]: 2024-09-14 03:48:13.431580693 +0000 UTC m=+0.672244379 container init 1b09e4c45bc1b7660d5d7d70676e48091df8354470af858ed276fe501987d09b (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:48:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 podman[79447]: 2024-09-14 03:48:13.439595438 +0000 UTC m=+0.680259123 container start 1b09e4c45bc1b7660d5d7d70676e48091df8354470af858ed276fe501987d09b (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T03:48:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.473Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-14T03:48:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.473Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-14T03:48:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.474Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi160 (none))" 2024-09-14T03:48:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.474Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.474Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.480Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.481Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.482Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.482Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.485Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.485Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=5.898µs 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.485Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.486Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.527Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 bash[79447]: 1b09e4c45bc1b7660d5d7d70676e48091df8354470af858ed276fe501987d09b 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.542Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-14T03:48:13.910 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.542Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-14T03:48:13.911 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.542Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=50.689µs wal_replay_duration=57.039323ms wbl_replay_duration=126ns total_replay_duration=57.348503ms 2024-09-14T03:48:13.911 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 systemd[1]: Started Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:48:13.911 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.550Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-14T03:48:13.911 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.550Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-14T03:48:13.911 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.550Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-14T03:48:13.911 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.574Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.073194ms db_storage=1.385µs remote_storage=1.504µs web_handler=887ns query_engine=969ns scrape=378.882µs scrape_sd=189.718µs notify=17.024µs notify_sd=18.184µs rules=22.860348ms tracing=8.66µs 2024-09-14T03:48:13.911 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.575Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-14T03:48:13.911 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:13.575Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-14T03:48:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:14 smithi160 ceph-mon[32702]: pgmap v13: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 671 op/s 2024-09-14T03:48:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:14 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:14 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:14 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:48:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:14 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:48:14.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:14 smithi114 ceph-mon[40961]: pgmap v13: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 671 op/s 2024-09-14T03:48:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:14 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:14 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:14 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:48:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:14 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:48:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:14 smithi114 ceph-mon[29589]: pgmap v13: 89 pgs: 89 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 671 op/s 2024-09-14T03:48:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:14 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:14 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:14 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:48:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:14 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:48:15.195 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:15 smithi160 ceph-mon[32702]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:48:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:15 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1390427381' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:15.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:15 smithi114 ceph-mon[40961]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:48:15.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:15 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1390427381' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:15.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:15 smithi114 ceph-mon[29589]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-14T03:48:15.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:15 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1390427381' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:16.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:16 smithi114 ceph-mon[40961]: pgmap v14: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:48:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:16 smithi114 ceph-mon[40961]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:16 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:48:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:16 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:48:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:16 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:16 smithi114 ceph-mon[29589]: pgmap v14: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:48:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:16 smithi114 ceph-mon[29589]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:16 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:48:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:16 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:48:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:16 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:16 smithi160 ceph-mon[32702]: pgmap v14: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-14T03:48:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:16 smithi160 ceph-mon[32702]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:16 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-14T03:48:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:16 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-14T03:48:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:16 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:17.275 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:17.276 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:17 smithi160 ceph-mon[32702]: from='client.25643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:17.276 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:17 smithi160 ceph-mon[32702]: Upgrade: Updating mgr.x 2024-09-14T03:48:17.276 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:17 smithi160 ceph-mon[32702]: Deploying daemon mgr.x on smithi160 2024-09-14T03:48:17.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:17 smithi114 ceph-mon[40961]: from='client.25643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:17.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:17 smithi114 ceph-mon[40961]: Upgrade: Updating mgr.x 2024-09-14T03:48:17.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:17 smithi114 ceph-mon[40961]: Deploying daemon mgr.x on smithi160 2024-09-14T03:48:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:17 smithi114 ceph-mon[29589]: from='client.25643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:17 smithi114 ceph-mon[29589]: Upgrade: Updating mgr.x 2024-09-14T03:48:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:17 smithi114 ceph-mon[29589]: Deploying daemon mgr.x on smithi160 2024-09-14T03:48:18.255 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:18 smithi160 ceph-mon[32702]: pgmap v15: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 711 op/s 2024-09-14T03:48:18.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:18 smithi114 ceph-mon[29589]: pgmap v15: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 711 op/s 2024-09-14T03:48:18.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:18 smithi114 ceph-mon[40961]: pgmap v15: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 711 op/s 2024-09-14T03:48:18.781 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:19.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:18 smithi160 systemd[1]: Stopping Ceph mgr.x for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:48:19.579 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:19 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1969310994' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:19.579 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:19 smithi160 podman[79763]: 2024-09-14 03:48:19.270040412 +0000 UTC m=+0.206916118 container died e4ad5ee26d6afba939042057254af181ac8495816a9a680868fe0f31f166d10c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-a278231, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-14T03:48:19.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:19 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1969310994' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:19.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:19 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1969310994' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:19.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:19 smithi160 podman[79763]: 2024-09-14 03:48:19.79374121 +0000 UTC m=+0.730616915 container cleanup e4ad5ee26d6afba939042057254af181ac8495816a9a680868fe0f31f166d10c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-a278231, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-14T03:48:19.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:19 smithi160 bash[79763]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x 2024-09-14T03:48:20.299 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:20 smithi160 ceph-mon[32702]: from='client.25655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:20.299 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:20 smithi160 ceph-mon[32702]: pgmap v16: 89 pgs: 89 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T03:48:20.299 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:20 smithi160 ceph-mon[32702]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:20.300 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:20.300 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:20 smithi160 podman[79778]: 2024-09-14 03:48:20.045864881 +0000 UTC m=+0.767421838 container remove e4ad5ee26d6afba939042057254af181ac8495816a9a680868fe0f31f166d10c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, RELEASE=main-a278231, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T03:48:20.300 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:20 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-14T03:48:20.300 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:20 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-14T03:48:20.300 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:20 smithi160 systemd[1]: Stopped Ceph mgr.x for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:48:20.300 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:20 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.x.service: Consumed 29.490s CPU time. 2024-09-14T03:48:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:20 smithi114 ceph-mon[40961]: from='client.25655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:20 smithi114 ceph-mon[40961]: pgmap v16: 89 pgs: 89 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T03:48:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:20 smithi114 ceph-mon[40961]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:20 smithi114 ceph-mon[29589]: from='client.25655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:20 smithi114 ceph-mon[29589]: pgmap v16: 89 pgs: 89 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T03:48:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:20 smithi114 ceph-mon[29589]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:20.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:20 smithi160 systemd[1]: Starting Ceph mgr.x for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:48:21.349 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:20 smithi160 podman[79892]: 2024-09-14 03:48:20.830925003 +0000 UTC m=+0.034920389 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:48:21.648 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:21 smithi160 podman[79892]: 2024-09-14 03:48:21.347971153 +0000 UTC m=+0.551966531 container create 574eaed954d9adf878fd30c118bc756ac1e80b02d56a69fa1d935c358d4e9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=main-a278231, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-14T03:48:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:48:21] "GET /metrics HTTP/1.1" 200 36189 "" "Prometheus/2.51.0" 2024-09-14T03:48:21.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:21 smithi160 podman[79892]: 2024-09-14 03:48:21.801288026 +0000 UTC m=+1.005283407 container init 574eaed954d9adf878fd30c118bc756ac1e80b02d56a69fa1d935c358d4e9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=main-a278231) 2024-09-14T03:48:21.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:21 smithi160 podman[79892]: 2024-09-14 03:48:21.816445486 +0000 UTC m=+1.020440871 container start 574eaed954d9adf878fd30c118bc756ac1e80b02d56a69fa1d935c358d4e9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-a278231, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-14T03:48:21.908 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:21 smithi160 ceph-mgr[79926]: -- 172.21.15.160:0/1377774534 <== mon.2 v2:172.21.15.114:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c2075070e0 con 0x55c2074e4800 2024-09-14T03:48:22.207 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:21 smithi160 bash[79892]: 574eaed954d9adf878fd30c118bc756ac1e80b02d56a69fa1d935c358d4e9dd9 2024-09-14T03:48:22.207 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:21 smithi160 systemd[1]: Started Ceph mgr.x for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:48:22.282 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:22 smithi114 ceph-mon[40961]: pgmap v17: 89 pgs: 89 active+clean; 837 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 997 op/s 2024-09-14T03:48:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:22 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/860528710' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:22 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:22 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:22 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:48:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:22 smithi114 ceph-mon[29589]: pgmap v17: 89 pgs: 89 active+clean; 837 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 997 op/s 2024-09-14T03:48:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:22 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/860528710' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:22 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:22 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:22 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:48:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:22 smithi160 ceph-mon[32702]: pgmap v17: 89 pgs: 89 active+clean; 837 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 997 op/s 2024-09-14T03:48:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:22 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/860528710' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:22 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:22 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:22.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:22 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:48:23.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:23 smithi160 ceph-mon[32702]: from='client.15822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:23 smithi114 ceph-mon[40961]: from='client.15822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:23 smithi114 ceph-mon[29589]: from='client.15822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:24.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:24 smithi114 ceph-mon[40961]: from='client.15828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:24.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:24 smithi114 ceph-mon[40961]: pgmap v18: 89 pgs: 89 active+clean; 837 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-09-14T03:48:24.275 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:24 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:48:24.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:24 smithi114 ceph-mon[29589]: from='client.15828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:24.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:24 smithi114 ceph-mon[29589]: pgmap v18: 89 pgs: 89 active+clean; 837 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-09-14T03:48:24.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:24 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:48:24.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:24.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:24 smithi160 ceph-mon[32702]: from='client.15828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:24.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:24 smithi160 ceph-mon[32702]: pgmap v18: 89 pgs: 89 active+clean; 837 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-09-14T03:48:24.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:24 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:48:25.809 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:26.133 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:25 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:26.133 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:25 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:26 smithi160 ceph-mon[32702]: pgmap v19: 89 pgs: 89 active+clean; 585 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-14T03:48:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:26 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2778533932' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:26.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:26 smithi114 ceph-mon[29589]: pgmap v19: 89 pgs: 89 active+clean; 585 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-14T03:48:26.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:26 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/2778533932' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:26.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:26 smithi114 ceph-mon[40961]: pgmap v19: 89 pgs: 89 active+clean; 585 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-14T03:48:26.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:26 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2778533932' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:27.243 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:26.950Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:27.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:27 smithi114 ceph-mon[29589]: from='client.25697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:27.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:27 smithi114 ceph-mon[29589]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:27.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:27 smithi114 ceph-mon[40961]: from='client.25697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:27.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:27 smithi114 ceph-mon[40961]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:27 smithi160 ceph-mon[32702]: from='client.25697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:27 smithi160 ceph-mon[32702]: from='client.15843 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:28 smithi114 ceph-mon[40961]: pgmap v20: 89 pgs: 89 active+clean; 585 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 686 op/s 2024-09-14T03:48:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:28 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:28 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:28 smithi114 ceph-mon[29589]: pgmap v20: 89 pgs: 89 active+clean; 585 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 686 op/s 2024-09-14T03:48:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:28 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:28 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:28 smithi160 ceph-mon[32702]: pgmap v20: 89 pgs: 89 active+clean; 585 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 686 op/s 2024-09-14T03:48:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:28 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:28 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:29.199 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:29 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1405207189' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:29 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:29 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:29.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:29 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/1405207189' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:29.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:29 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:29.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:29 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:29.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:29 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1405207189' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:29.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:29 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:29.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:29 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:30.261 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:30.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:30 smithi160 ceph-mon[32702]: pgmap v21: 89 pgs: 89 active+clean; 571 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-14T03:48:30.605 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:30 smithi160 ceph-mon[32702]: from='client.15852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:30.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:30 smithi114 ceph-mon[29589]: pgmap v21: 89 pgs: 89 active+clean; 571 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-14T03:48:30.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:30 smithi114 ceph-mon[29589]: from='client.15852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:30.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:30 smithi114 ceph-mon[40961]: pgmap v21: 89 pgs: 89 active+clean; 571 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-14T03:48:30.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:30 smithi114 ceph-mon[40961]: from='client.15852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:30.910 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: [14/Sep/2024:03:48:30] ENGINE Bus STARTING 2024-09-14T03:48:30.910 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: CherryPy Checker: 2024-09-14T03:48:30.911 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: The Application mounted at '' has an empty config. 2024-09-14T03:48:30.911 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: 2024-09-14T03:48:30.911 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: [14/Sep/2024:03:48:30] ENGINE Serving on http://:::9283 2024-09-14T03:48:30.911 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: [14/Sep/2024:03:48:30] ENGINE Bus STARTED 2024-09-14T03:48:31.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:48:31] "GET /metrics HTTP/1.1" 200 36187 "" "Prometheus/2.51.0" 2024-09-14T03:48:31.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[29589]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:31.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[29589]: Standby manager daemon x restarted 2024-09-14T03:48:31.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[29589]: Standby manager daemon x started 2024-09-14T03:48:31.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:48:31.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:48:31.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:48:31.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[29589]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:48:31.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[40961]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:31.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[40961]: Standby manager daemon x restarted 2024-09-14T03:48:31.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[40961]: Standby manager daemon x started 2024-09-14T03:48:31.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:48:31.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:48:31.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:48:31.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:31 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:48:31.630 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:31 smithi160 ceph-mon[32702]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:31.630 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:31 smithi160 ceph-mon[32702]: Standby manager daemon x restarted 2024-09-14T03:48:31.630 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:31 smithi160 ceph-mon[32702]: Standby manager daemon x started 2024-09-14T03:48:31.630 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:31 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:48:31.630 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:31 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:48:31.630 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:31 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:48:31.630 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:31 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/4025960318' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:48:32.618 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:32.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:32 smithi114 ceph-mon[40961]: pgmap v22: 89 pgs: 89 active+clean; 357 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 968 op/s 2024-09-14T03:48:32.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:32 smithi114 ceph-mon[40961]: mgrmap e34: y(active, since 38s), standbys: x 2024-09-14T03:48:32.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:32 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3420659382' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:32.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:32 smithi114 ceph-mon[29589]: pgmap v22: 89 pgs: 89 active+clean; 357 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 968 op/s 2024-09-14T03:48:32.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:32 smithi114 ceph-mon[29589]: mgrmap e34: y(active, since 38s), standbys: x 2024-09-14T03:48:32.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:32 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/3420659382' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:32.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:32 smithi160 ceph-mon[32702]: pgmap v22: 89 pgs: 89 active+clean; 357 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 968 op/s 2024-09-14T03:48:32.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:32 smithi160 ceph-mon[32702]: mgrmap e34: y(active, since 38s), standbys: x 2024-09-14T03:48:32.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:32 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3420659382' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:34.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:34 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:34 smithi160 ceph-mon[32702]: from='client.25730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:34 smithi160 ceph-mon[32702]: pgmap v23: 89 pgs: 89 active+clean; 357 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 639 op/s 2024-09-14T03:48:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:34 smithi160 ceph-mon[32702]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:34 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:34 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:34 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:34 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:48:34.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:34 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:34.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[29589]: from='client.25730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[29589]: pgmap v23: 89 pgs: 89 active+clean; 357 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 639 op/s 2024-09-14T03:48:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[29589]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:48:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:34.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[40961]: from='client.25730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:34.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[40961]: pgmap v23: 89 pgs: 89 active+clean; 357 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 639 op/s 2024-09-14T03:48:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[40961]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-14T03:48:34.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:34 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:35.547 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:48:35.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-14T03:48:35.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: Upgrade: Setting container_image for all mgr 2024-09-14T03:48:35.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:35.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-14T03:48:35.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-14T03:48:35.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-14T03:48:35.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-14T03:48:35.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-14T03:48:35.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-14T03:48:35.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[40961]: Upgrade: It appears safe to stop mon.a 2024-09-14T03:48:35.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:48:35.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-14T03:48:35.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: Upgrade: Setting container_image for all mgr 2024-09-14T03:48:35.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:35.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-14T03:48:35.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-14T03:48:35.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-14T03:48:35.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-14T03:48:35.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-14T03:48:35.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-14T03:48:35.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:35 smithi114 ceph-mon[29589]: Upgrade: It appears safe to stop mon.a 2024-09-14T03:48:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-14T03:48:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-14T03:48:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: Upgrade: Setting container_image for all mgr 2024-09-14T03:48:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-14T03:48:35.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-14T03:48:35.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-14T03:48:35.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-14T03:48:35.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-14T03:48:35.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-14T03:48:35.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-mon[32702]: Upgrade: It appears safe to stop mon.a 2024-09-14T03:48:35.960 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 131678 objects 2024-09-14T03:48:35.961 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :159.637 2024-09-14T03:48:36.101 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:36.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:36.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:36 smithi114 ceph-mon[29589]: pgmap v24: 89 pgs: 89 active+clean; 114 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 964 op/s 2024-09-14T03:48:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:36 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/4129514190' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:36 smithi114 ceph-mon[40961]: pgmap v24: 89 pgs: 89 active+clean; 114 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 964 op/s 2024-09-14T03:48:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:36 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/4129514190' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:36.388 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-14T03:48:36.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-14T03:48:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:36 smithi160 ceph-mon[32702]: pgmap v24: 89 pgs: 89 active+clean; 114 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 964 op/s 2024-09-14T03:48:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:36 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/4129514190' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:36.672 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:48:37.344 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:37.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[29589]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:37.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[29589]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:37.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:37.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:48:37.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:48:37.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:37.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[40961]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:37.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[40961]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:37.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:37.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:48:37.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:48:37.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:37 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:37 smithi160 ceph-mon[32702]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:37 smithi160 ceph-mon[32702]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:37 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' 2024-09-14T03:48:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:37 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-14T03:48:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:37 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-14T03:48:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:37 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-14T03:48:38.604 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:38 smithi114 ceph-mon[40961]: Upgrade: Updating mon.a 2024-09-14T03:48:38.604 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:38 smithi114 ceph-mon[40961]: Deploying daemon mon.a on smithi114 2024-09-14T03:48:38.604 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:38 smithi114 ceph-mon[40961]: pgmap v25: 89 pgs: 89 active+clean; 114 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 627 op/s 2024-09-14T03:48:38.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:38 smithi114 ceph-mon[29589]: Upgrade: Updating mon.a 2024-09-14T03:48:38.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:38 smithi114 ceph-mon[29589]: Deploying daemon mon.a on smithi114 2024-09-14T03:48:38.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:38 smithi114 ceph-mon[29589]: pgmap v25: 89 pgs: 89 active+clean; 114 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 627 op/s 2024-09-14T03:48:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:38 smithi160 ceph-mon[32702]: Upgrade: Updating mon.a 2024-09-14T03:48:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:38 smithi160 ceph-mon[32702]: Deploying daemon mon.a on smithi114 2024-09-14T03:48:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:38 smithi160 ceph-mon[32702]: pgmap v25: 89 pgs: 89 active+clean; 114 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 627 op/s 2024-09-14T03:48:39.020 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:39.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:39 smithi114 ceph-mon[29589]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:48:39.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:39 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/442275648' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:39 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:48:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:39 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/442275648' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:39 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-14T03:48:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:39 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/442275648' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:40.366 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_0' removed 2024-09-14T03:48:40.447 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:48:40] ENGINE Bus STOPPING 2024-09-14T03:48:40.447 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-mon[40961]: pgmap v26: 89 pgs: 89 active+clean; 107 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 637 op/s 2024-09-14T03:48:40.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-mon[40961]: from='client.25766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:40.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-mon[40961]: from='client.15906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:40.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/326906320' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-14T03:48:40.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-14T03:48:40.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 systemd[1]: Stopping Ceph mon.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:48:40.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-mon[29589]: pgmap v26: 89 pgs: 89 active+clean; 107 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 637 op/s 2024-09-14T03:48:40.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-mon[29589]: from='client.25766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:40.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-mon[29589]: from='client.15906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:40.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-mon[29589]: from='client.? 172.21.15.114:0/326906320' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-14T03:48:40.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-mon[29589]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-14T03:48:40.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[29566]: 2024-09-14T03:48:40.388+0000 7f446a5df640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T03:48:40.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[29566]: 2024-09-14T03:48:40.388+0000 7f446a5df640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-14T03:48:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:40 smithi160 ceph-mon[32702]: pgmap v26: 89 pgs: 89 active+clean; 107 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 637 op/s 2024-09-14T03:48:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:40 smithi160 ceph-mon[32702]: from='client.25766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:40 smithi160 ceph-mon[32702]: from='client.15906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:40 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/326906320' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-14T03:48:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:40 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-14T03:48:40.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 podman[102545]: 2024-09-14 03:48:40.608110916 +0000 UTC m=+0.308039418 container died 92df6a472c268de312420180dc902f911c2aa33af1a55a86220a7ed28249113c (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-eb16b97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-14T03:48:40.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:48:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-14T03:48:40.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:48:40] ENGINE Bus STOPPED 2024-09-14T03:48:40.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:48:40] ENGINE Bus STARTING 2024-09-14T03:48:40.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:48:40] ENGINE Serving on http://:::9283 2024-09-14T03:48:40.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:48:40] ENGINE Bus STARTED 2024-09-14T03:48:41.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 podman[102545]: 2024-09-14 03:48:40.995235129 +0000 UTC m=+0.695163627 container cleanup 92df6a472c268de312420180dc902f911c2aa33af1a55a86220a7ed28249113c (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=quincy-eb16b97, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-14T03:48:41.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:40 smithi114 bash[102545]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a 2024-09-14T03:48:41.650 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-14T03:48:41.651 INFO:tasks.radosbench:Beginning radosbench... 2024-09-14T03:48:41.651 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-14T03:48:41.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-14T03:48:41.869 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:48:41] "GET /metrics HTTP/1.1" 200 36188 "" "Prometheus/2.51.0" 2024-09-14T03:48:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:48:41.977 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:42.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:41 smithi114 podman[102572]: 2024-09-14 03:48:41.866492784 +0000 UTC m=+1.442889476 container remove 92df6a472c268de312420180dc902f911c2aa33af1a55a86220a7ed28249113c (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=quincy-eb16b97, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-14T03:48:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:41 smithi114 bash[102682]: Error: no container with name or ID "ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a" found: no such container 2024-09-14T03:48:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:41 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-14T03:48:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:41 smithi114 systemd[1]: Stopped Ceph mon.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:48:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:41 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.a.service: Consumed 14.235s CPU time. 2024-09-14T03:48:42.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:42 smithi114 systemd[1]: Starting Ceph mon.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:48:43.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:42 smithi114 podman[102903]: 2024-09-14 03:48:42.738830659 +0000 UTC m=+0.347078204 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:48:43.704 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 podman[102903]: 2024-09-14 03:48:43.484881278 +0000 UTC m=+1.093128814 container create 2ca35e38ecf565399265397d6adaf8efb8940587ecad29b42f482e94c49dd58a (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a278231, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:48:43.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 podman[102903]: 2024-09-14 03:48:43.946827365 +0000 UTC m=+1.555074920 container init 2ca35e38ecf565399265397d6adaf8efb8940587ecad29b42f482e94c49dd58a (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-a278231) 2024-09-14T03:48:43.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 podman[102903]: 2024-09-14 03:48:43.95174579 +0000 UTC m=+1.559993332 container start 2ca35e38ecf565399265397d6adaf8efb8940587ecad29b42f482e94c49dd58a (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, RELEASE=main-a278231, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-14T03:48:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: set uid:gid to 167:167 (ceph:ceph) 2024-09-14T03:48:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev), process ceph-mon, pid 7 2024-09-14T03:48:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: pidfile_write: ignore empty --pid-file 2024-09-14T03:48:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: load: jerasure load: lrc 2024-09-14T03:48:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: RocksDB version: 7.9.2 2024-09-14T03:48:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Git sha 0 2024-09-14T03:48:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Compile date 2024-09-12 20:09:22 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: DB SUMMARY 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: DB Session ID: 7ZFWXBQYGRK0OH59ZZO3 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: CURRENT file: CURRENT 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: IDENTITY file: IDENTITY 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1179 Bytes 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000033.sst 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000031.log size: 1278695 ; 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.error_if_exists: 0 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.create_if_missing: 0 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.paranoid_checks: 1 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-14T03:48:44.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.env: 0x560e3752d120 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.fs: PosixFileSystem 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.info_log: 0x560e37daf700 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_file_opening_threads: 16 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.statistics: (nil) 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.use_fsync: 0 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_log_file_size: 0 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.keep_log_file_num: 1000 2024-09-14T03:48:44.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.recycle_log_file_num: 0 2024-09-14T03:48:44.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.allow_fallocate: 1 2024-09-14T03:48:44.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.allow_mmap_reads: 0 2024-09-14T03:48:44.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.allow_mmap_writes: 0 2024-09-14T03:48:44.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.use_direct_reads: 0 2024-09-14T03:48:44.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-14T03:48:44.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.create_missing_column_families: 0 2024-09-14T03:48:44.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.db_log_dir: 2024-09-14T03:48:44.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.wal_dir: 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.advise_random_on_open: 1 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.db_write_buffer_size: 0 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.write_buffer_manager: 0x560e37db21e0 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-14T03:48:44.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.rate_limiter: (nil) 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.wal_recovery_mode: 2 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.enable_thread_tracking: 0 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.enable_pipelined_write: 0 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.unordered_write: 0 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-14T03:48:44.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.row_cache: None 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.wal_filter: None 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.allow_ingest_behind: 0 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.two_write_queues: 0 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.manual_wal_flush: 0 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.wal_compression: 0 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.atomic_flush: 0 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.log_readahead_size: 0 2024-09-14T03:48:44.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.best_efforts_recovery: 0 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.allow_data_in_errors: 0 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.db_host_id: __hostname__ 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_background_jobs: 2 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_background_compactions: -1 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_subcompactions: 1 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-14T03:48:44.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_total_wal_size: 0 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_open_files: -1 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bytes_per_sync: 0 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_readahead_size: 0 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_background_flushes: -1 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Compression algorithms supported: 2024-09-14T03:48:44.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: kZSTD supported: 0 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: kXpressCompression supported: 0 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: kBZip2Compression supported: 0 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: kLZ4Compression supported: 1 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: kZlibCompression supported: 1 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: kLZ4HCCompression supported: 1 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: kSnappyCompression supported: 1 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.merge_operator: 2024-09-14T03:48:44.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_filter: None 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_filter_factory: None 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.sst_partitioner_factory: None 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560e37daf320) 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_type: 0 2024-09-14T03:48:44.382 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_index_type: 0 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_shortening: 1 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: checksum: 4 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: no_block_cache: 0 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache: 0x560e37da3610 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_options: 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: capacity : 536870912 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_shard_bits : 4 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: strict_capacity_limit : 0 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_compressed: (nil) 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: persistent_cache: (nil) 2024-09-14T03:48:44.383 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size: 4096 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size_deviation: 10 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_restart_interval: 16 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_block_restart_interval: 1 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: metadata_block_size: 4096 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: partition_filters: 0 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: use_delta_encoding: 1 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: filter_policy: bloomfilter 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: whole_key_filtering: 1 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: verify_compression: 0 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: format_version: 5 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_index_compression: 1 2024-09-14T03:48:44.384 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_align: 0 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout: max_auto_readahead_size: 262144 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout: prepopulate_block_cache: 0 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout: initial_auto_readahead_size: 8192 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.write_buffer_size: 33554432 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_write_buffer_number: 2 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compression: NoCompression 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bottommost_compression: Disabled 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.prefix_extractor: nullptr 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.num_levels: 7 2024-09-14T03:48:44.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compression_opts.level: 32767 2024-09-14T03:48:44.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compression_opts.strategy: 0 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compression_opts.enabled: false 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.target_file_size_base: 67108864 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-14T03:48:44.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.arena_block_size: 1048576 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-14T03:48:44.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.disable_auto_compactions: 0 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-14T03:48:44.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.inplace_update_support: 0 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.bloom_locality: 0 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.max_successive_merges: 0 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.paranoid_file_checks: 0 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.force_consistency_checks: 1 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.report_bg_io_stats: 0 2024-09-14T03:48:44.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.ttl: 2592000 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.enable_blob_files: false 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.min_blob_size: 0 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.blob_file_size: 268435456 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.blob_file_starting_level: 0 2024-09-14T03:48:44.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 12186, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d111feed-7c54-45cb-be1d-17d4b0df22db 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285723978437, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285723982946, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 1183647, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12187, "largest_seqno": 12879, "table_properties": {"data_size": 1179590, "index_size": 2121, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 901, "raw_key_size": 7912, "raw_average_key_size": 23, "raw_value_size": 1172351, "raw_average_value_size": 3499, "num_data_blocks": 96, "num_entries": 335, "num_filter_entries": 335, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726285723, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d111feed-7c54-45cb-be1d-17d4b0df22db", "db_session_id": "7ZFWXBQYGRK0OH59ZZO3", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285723983056, "job": 1, "event": "recovery_finished"} 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-14T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560e37dd2e00 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: DB pointer 0x560e37ed4000 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** DB Stats ** 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: L0 1/0 1.13 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 439.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: L6 1/0 7.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: Sum 2/0 8.24 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 439.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 439.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 439.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:48:44.394 INFO:journalctl@ceph.mon.a.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative compaction: 0.00 GB write, 165.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval compaction: 0.00 GB write, 165.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache BinnedLRUCache@0x560e37da3610#7 capacity: 512.00 MB usage: 3.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.03 KB,0.000196695%) IndexBlock(1,2.33 KB,0.000444055%) Misc(1,0.00 KB,0%) 2024-09-14T03:48:44.395 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: starting mon.a rank 0 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: mon.a@-1(???) e3 preinit fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: mon.a@-1(???).mds e9 new map 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: mon.a@-1(???).mds e9 print_map 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout: e9 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout: legacy client fscid: 1 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout: Filesystem 'foo' (1) 2024-09-14T03:48:44.396 INFO:journalctl@ceph.mon.a.smithi114.stdout: fs_name foo 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: epoch 8 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: created 2024-09-14T03:43:11.271900+0000 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: modified 2024-09-14T03:43:23.592848+0000 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: tableserver 0 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: root 0 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: session_timeout 60 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: session_autoclose 300 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: max_file_size 1099511627776 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: max_xattr_size 65536 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: required_client_features {} 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: last_failure 0 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: last_failure_osd_epoch 0 2024-09-14T03:48:44.397 INFO:journalctl@ceph.mon.a.smithi114.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: max_mds 1 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: in 0 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: up {0=14658} 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: failed 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: damaged 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: stopped 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_pools [4] 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: metadata_pool 3 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: inline_data disabled 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: balancer 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: bal_rank_mask -1 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: standby_count_wanted 1 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: qdb_cluster leader: 0 members: 2024-09-14T03:48:44.398 INFO:journalctl@ceph.mon.a.smithi114.stdout: [mds.foo.smithi114.ebmrat{0:14658} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] compat {c=[1],r=[1],i=[7ff]}] 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout: Standby daemons: 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout: [mds.foo.smithi160.ywyjae{-1:24481} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.160:6832/2654584478,v1:172.21.15.160:6833/2654584478] compat {c=[1],r=[1],i=[7ff]}] 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: mon.a@-1(???).osd e71 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: mon.a@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: mon.a@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: mon.a@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:43 smithi114 ceph-mon[103029]: mon.a@-1(???).paxosservice(auth 1..19) refresh upgraded, format 0 -> 3 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:44 smithi114 bash[102903]: 2ca35e38ecf565399265397d6adaf8efb8940587ecad29b42f482e94c49dd58a 2024-09-14T03:48:44.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:44 smithi114 systemd[1]: Started Ceph mon.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:48:44.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:44.891 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:48:46.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:46.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:47.082 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_1' created 2024-09-14T03:48:47.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ignoring --setuser ceph since I am not root 2024-09-14T03:48:47.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ignoring --setgroup ceph since I am not root 2024-09-14T03:48:47.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mgr[93546]: -- 172.21.15.114:0/751572170 <== mon.2 v2:172.21.15.114:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5646b2a390e0 con 0x5646b2a17000 2024-09-14T03:48:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:48:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:48:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:48:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:48:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:48:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: pgmap v28: 73 pgs: 73 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1471917631' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='client.25790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='client.25796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: pgmap v29: 73 pgs: 73 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3925077831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-14T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/1672892901' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: pgmap v30: 73 pgs: 73 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='client.25814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='client.25820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: mon.a calling monitor election 2024-09-14T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:48:47.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='client.? ' entity='' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: monmap epoch 3 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: last_changed 2024-09-14T03:38:03.057799+0000 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: min_mon_release 17 (quincy) 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: election_strategy: 1 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: osdmap e71: 8 total, 8 up, 8 in 2024-09-14T03:48:47.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: mgrmap e34: y(active, since 54s), standbys: x 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: overall HEALTH_OK 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: from='mgr.25505 ' entity='' 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[40961]: mgrmap e35: y(active, since 54s), standbys: x 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: pgmap v28: 73 pgs: 73 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1471917631' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:47.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='client.25790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='client.25796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: pgmap v29: 73 pgs: 73 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3925077831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1672892901' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: pgmap v30: 73 pgs: 73 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='client.25814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='client.25820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: mon.a calling monitor election 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='client.? ' entity='' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: monmap epoch 3 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: last_changed 2024-09-14T03:38:03.057799+0000 2024-09-14T03:48:47.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: min_mon_release 17 (quincy) 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: election_strategy: 1 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: osdmap e71: 8 total, 8 up, 8 in 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: mgrmap e34: y(active, since 54s), standbys: x 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: overall HEALTH_OK 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: from='mgr.25505 ' entity='' 2024-09-14T03:48:47.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:47 smithi114 ceph-mon[103029]: mgrmap e35: y(active, since 54s), standbys: x 2024-09-14T03:48:47.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: ignoring --setuser ceph since I am not root 2024-09-14T03:48:47.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: ignoring --setgroup ceph since I am not root 2024-09-14T03:48:47.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mgr[79926]: -- 172.21.15.160:0/2790807027 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dee5d4b0e0 con 0x55dee5d29000 2024-09-14T03:48:47.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:46.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-14T03:48:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-14T03:48:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-14T03:48:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='mgr.25505 172.21.15.114:0/816956185' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:48:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:48:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: pgmap v28: 73 pgs: 73 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1471917631' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='client.25790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='client.25796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: pgmap v29: 73 pgs: 73 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3925077831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/1672892901' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: pgmap v30: 73 pgs: 73 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='client.25814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='client.25820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: mon.a calling monitor election 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='client.? ' entity='' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: monmap epoch 3 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:48:47.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: last_changed 2024-09-14T03:38:03.057799+0000 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: min_mon_release 17 (quincy) 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: election_strategy: 1 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: osdmap e71: 8 total, 8 up, 8 in 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: mgrmap e34: y(active, since 54s), standbys: x 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: overall HEALTH_OK 2024-09-14T03:48:47.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: from='mgr.25505 ' entity='' 2024-09-14T03:48:47.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:47 smithi160 ceph-mon[32702]: mgrmap e35: y(active, since 54s), standbys: x 2024-09-14T03:48:47.528 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev)": 1 2024-09-14T03:48:47.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-14T03:48:48.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:48 smithi114 ceph-mon[103029]: from='client.? ' entity='' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-14T03:48:48.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:48 smithi114 ceph-mon[103029]: osdmap e72: 8 total, 8 up, 8 in 2024-09-14T03:48:48.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:48 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2870805768' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:48:48.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:48 smithi114 ceph-mon[40961]: from='client.? ' entity='' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-14T03:48:48.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:48 smithi114 ceph-mon[40961]: osdmap e72: 8 total, 8 up, 8 in 2024-09-14T03:48:48.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:48 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2870805768' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:48:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:48 smithi160 ceph-mon[32702]: from='client.? ' entity='' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-14T03:48:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:48 smithi160 ceph-mon[32702]: osdmap e72: 8 total, 8 up, 8 in 2024-09-14T03:48:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:48 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2870805768' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:48:48.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch ps' 2024-09-14T03:48:48.636 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:48:49.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:49 smithi114 ceph-mon[103029]: osdmap e73: 8 total, 8 up, 8 in 2024-09-14T03:48:49.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:49 smithi114 ceph-mon[40961]: osdmap e73: 8 total, 8 up, 8 in 2024-09-14T03:48:49.375 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:48:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:49 smithi160 ceph-mon[32702]: osdmap e73: 8 total, 8 up, 8 in 2024-09-14T03:48:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:50 smithi114 ceph-mon[103029]: osdmap e74: 8 total, 8 up, 8 in 2024-09-14T03:48:50.167 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:50 smithi114 ceph-mon[40961]: osdmap e74: 8 total, 8 up, 8 in 2024-09-14T03:48:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:50 smithi160 ceph-mon[32702]: osdmap e74: 8 total, 8 up, 8 in 2024-09-14T03:48:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:51.317 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-14T03:48:51.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:51 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/3343865977' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:48:51.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:51 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:48:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3343865977' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:48:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:51 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:48:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:51 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/3343865977' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-14T03:48:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:51 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:48:52.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: [14/Sep/2024:03:48:52] ENGINE Bus STARTING 2024-09-14T03:48:52.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: CherryPy Checker: 2024-09-14T03:48:52.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: The Application mounted at '' has an empty config. 2024-09-14T03:48:52.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: 2024-09-14T03:48:52.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: [14/Sep/2024:03:48:52] ENGINE Serving on http://:::9283 2024-09-14T03:48:52.408 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x[79904]: [14/Sep/2024:03:48:52] ENGINE Bus STARTED 2024-09-14T03:48:52.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:48:52.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: osdmap e75: 8 total, 8 up, 8 in 2024-09-14T03:48:52.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: Standby manager daemon x restarted 2024-09-14T03:48:52.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: Standby manager daemon x started 2024-09-14T03:48:52.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:48:52.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:48:52.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: Active manager daemon y restarted 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: Activating manager daemon y 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: mgrmap e36: y(active, starting, since 0.00580107s), standbys: x 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"} : dispatch 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"} : dispatch 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-14T03:48:52.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-14T03:48:52.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:52 smithi160 ceph-mon[32702]: Manager daemon y is now available 2024-09-14T03:48:52.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:48:52.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: osdmap e75: 8 total, 8 up, 8 in 2024-09-14T03:48:52.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: Standby manager daemon x restarted 2024-09-14T03:48:52.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: Standby manager daemon x started 2024-09-14T03:48:52.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:48:52.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:48:52.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:48:52.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: Active manager daemon y restarted 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: Activating manager daemon y 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: mgrmap e36: y(active, starting, since 0.00580107s), standbys: x 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"} : dispatch 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"} : dispatch 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-14T03:48:52.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-14T03:48:52.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-14T03:48:52.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-14T03:48:52.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-14T03:48:52.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-14T03:48:52.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-14T03:48:52.510 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-14T03:48:52.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:48:52.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: osdmap e75: 8 total, 8 up, 8 in 2024-09-14T03:48:52.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: Standby manager daemon x restarted 2024-09-14T03:48:52.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: Standby manager daemon x started 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.? 172.21.15.160:0/2005214135' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: Active manager daemon y restarted 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: Activating manager daemon y 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: mgrmap e36: y(active, starting, since 0.00580107s), standbys: x 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi114.ebmrat"} : dispatch 2024-09-14T03:48:52.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi160.ywyjae"} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-14T03:48:52.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-14T03:48:52.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-14T03:48:52.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-14T03:48:52.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[40961]: Manager daemon y is now available 2024-09-14T03:48:52.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-14T03:48:52.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-14T03:48:52.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-mon[103029]: Manager daemon y is now available 2024-09-14T03:48:52.767 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:52.617+0000 7fd3e6cf5640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:48:52.767 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:52.617+0000 7fd3e94fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:48:52.767 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:52.617+0000 7fd3e94fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:48:52.767 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:52.617+0000 7fd3e94fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:48:52.768 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:52.617+0000 7fd3e94fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:48:52.768 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:52.617+0000 7fd3e94fa640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:48:52.768 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:52.617+0000 7fd3e6cf5640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:48:52.768 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:52.617+0000 7fd3e6cf5640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:48:52.768 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:52.617+0000 7fd3e6cf5640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:48:52.768 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:52 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:52.617+0000 7fd3e6cf5640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-14T03:48:53.214 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-14T03:48:53.217 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-14T03:48:53.217 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-14T03:48:53.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-14T03:48:53.264 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-14T03:48:53.264 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_103911 2024-09-14T03:48:53.264 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:48:53.264 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-14T03:48:53.309 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-14T03:48:53.310 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.a smithi114 *:9093,9094 running (2m) 54s ago 6m 19.0M - 0.27.0 11f11916f8cd eaf39a5cb581 2024-09-14T03:48:53.310 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.a smithi160 *:3000 running (87s) 25s ago 6m 79.6M - 9.4.7 954c08fa6188 aa46fcb7cedc 2024-09-14T03:48:53.310 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi114.ebmrat smithi114 running (5m) 54s ago 5m 16.8M - 17.2.7-1629-geb16b97b ab51d1bebd12 00f4a6ea534e 2024-09-14T03:48:53.310 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi160.ywyjae smithi160 running (5m) 25s ago 5m 14.1M - 17.2.7-1629-geb16b97b ab51d1bebd12 a87b5ff7e907 2024-09-14T03:48:53.310 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x smithi160 *:8443,9283,8765 running (32s) 25s ago 10m 62.8M - 19.3.0-4907-ga2782315 8801d2529e43 574eaed954d9 2024-09-14T03:48:53.310 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y smithi114 *:8443,9283,8765 running (72s) 54s ago 12m 488M - 19.3.0-4907-ga2782315 8801d2529e43 4f6e65854aef 2024-09-14T03:48:53.310 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a smithi114 running (12m) 54s ago 12m 56.7M 2048M 17.2.7-1629-geb16b97b ab51d1bebd12 92df6a472c26 2024-09-14T03:48:53.310 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b smithi160 running (10m) 25s ago 10m 43.3M 2048M 17.2.7-1629-geb16b97b ab51d1bebd12 a30277b5b152 2024-09-14T03:48:53.310 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c smithi114 running (10m) 54s ago 10m 45.4M 2048M 17.2.7-1629-geb16b97b ab51d1bebd12 f8446e48bf23 2024-09-14T03:48:53.310 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.a smithi114 *:9100 running (113s) 54s ago 6m 10.4M - 1.7.0 72c9c2088986 ff1fe8404b33 2024-09-14T03:48:53.311 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.b smithi160 *:9100 running (80s) 25s ago 6m 10.3M - 1.7.0 72c9c2088986 289de68494ec 2024-09-14T03:48:53.311 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (10m) 54s ago 10m 377M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 ddb654f06efc 2024-09-14T03:48:53.311 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (9m) 54s ago 9m 248M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 a709e75520db 2024-09-14T03:48:53.311 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (9m) 54s ago 9m 315M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 a76e6ee3bc54 2024-09-14T03:48:53.311 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi114 running (9m) 54s ago 9m 172M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 b43e985ecf6c 2024-09-14T03:48:53.311 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi160 running (8m) 25s ago 8m 378M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 e89693bb3b76 2024-09-14T03:48:53.311 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi160 running (8m) 25s ago 8m 411M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 fb664f7f461a 2024-09-14T03:48:53.311 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 smithi160 running (7m) 25s ago 7m 214M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 0403446dedd1 2024-09-14T03:48:53.311 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 smithi160 running (7m) 25s ago 7m 257M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 39f0a617d104 2024-09-14T03:48:53.312 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.a smithi160 *:9095 running (40s) 25s ago 7m 44.9M - 2.51.0 1d3b7f56885b 1b09e4c45bc1 2024-09-14T03:48:53.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:48:53.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:48:53.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-14T03:48:53.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-14T03:48:53.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-14T03:48:53.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[103029]: mgrmap e37: y(active, since 1.01211s), standbys: x 2024-09-14T03:48:53.514 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:48:53.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:48:53.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-14T03:48:53.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-14T03:48:53.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-14T03:48:53.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-mon[40961]: mgrmap e37: y(active, since 1.01211s), standbys: x 2024-09-14T03:48:53.515 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:53.282+0000 7fd419828640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-14T03:48:53.515 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:48:53] ENGINE Bus STARTING 2024-09-14T03:48:53.515 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: CherryPy Checker: 2024-09-14T03:48:53.515 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: The Application mounted at '' has an empty config. 2024-09-14T03:48:53.515 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:53.516 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:48:53] ENGINE Serving on http://:::9283 2024-09-14T03:48:53.516 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:53 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:03:48:53] ENGINE Bus STARTED 2024-09-14T03:48:53.621 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:53 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:48:53.622 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:53 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:48:53.622 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:53 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-14T03:48:53.622 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:53 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-14T03:48:53.622 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:53 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-14T03:48:53.622 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:53 smithi160 ceph-mon[32702]: mgrmap e37: y(active, since 1.01211s), standbys: x 2024-09-14T03:48:54.264 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 2869 2853 178.304 178.312 0.00451785 0.00558474 2024-09-14T03:48:54.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch upgrade pause' 2024-09-14T03:48:54.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:48:54.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-mon[103029]: from='client.25874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:54.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-mon[103029]: pgmap v3: 89 pgs: 89 active+clean; 1.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:54.605 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-mon[40961]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:48:54.605 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-mon[40961]: from='client.25874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:54.606 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-mon[40961]: pgmap v3: 89 pgs: 89 active+clean; 1.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:54 smithi160 ceph-mon[32702]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-14T03:48:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:54 smithi160 ceph-mon[32702]: from='client.25874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:54 smithi160 ceph-mon[32702]: pgmap v3: 89 pgs: 89 active+clean; 1.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:54.873 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:54.602+0000 7fd3cd308640 -1 log_channel(cephadm) log [ERR] : [14/Sep/2024:03:48:54] ENGINE Error in HTTPServer.serve 2024-09-14T03:48:54.873 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: Traceback (most recent call last): 2024-09-14T03:48:54.873 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:48:54.873 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: self._connections.run(self.expiration_interval) 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: self._run(expiration_interval) 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: s = self.context.wrap_socket( 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: return self.sslsocket_class._create( 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: self.do_handshake() 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:48:54.874 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: self._sslobj.do_handshake() 2024-09-14T03:48:54.875 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:48:54.875 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:48:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:48:54.877 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:48:55.264 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 5857 5841 182.519 186.75 0.00446196 0.00546573 2024-09-14T03:48:55.564 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[40961]: mgrmap e38: y(active, since 2s), standbys: x 2024-09-14T03:48:55.564 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[40961]: [14/Sep/2024:03:48:54] ENGINE Bus STARTING 2024-09-14T03:48:55.564 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[40961]: [14/Sep/2024:03:48:54] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[40961]: [14/Sep/2024:03:48:54] ENGINE Error in HTTPServer.serve 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: Traceback (most recent call last): 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._run(expiration_interval) 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: s = self.context.wrap_socket( 2024-09-14T03:48:55.565 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.c.smithi114.stdout: return self.sslsocket_class._create( 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.c.smithi114.stdout: self.do_handshake() 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.c.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.c.smithi114.stdout: self._sslobj.do_handshake() 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.c.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[40961]: [14/Sep/2024:03:48:54] ENGINE Serving on http://172.21.15.114:8765 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[40961]: [14/Sep/2024:03:48:54] ENGINE Bus STARTED 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[103029]: mgrmap e38: y(active, since 2s), standbys: x 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[103029]: [14/Sep/2024:03:48:54] ENGINE Bus STARTING 2024-09-14T03:48:55.566 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[103029]: [14/Sep/2024:03:48:54] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[103029]: [14/Sep/2024:03:48:54] ENGINE Error in HTTPServer.serve 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:48:55.567 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-14T03:48:55.568 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:48:55.568 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-14T03:48:55.568 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:48:55.568 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-14T03:48:55.568 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:48:55.568 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-14T03:48:55.568 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:48:55.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[103029]: [14/Sep/2024:03:48:54] ENGINE Serving on http://172.21.15.114:8765 2024-09-14T03:48:55.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:55 smithi114 ceph-mon[103029]: [14/Sep/2024:03:48:54] ENGINE Bus STARTED 2024-09-14T03:48:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:55 smithi160 ceph-mon[32702]: mgrmap e38: y(active, since 2s), standbys: x 2024-09-14T03:48:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:55 smithi160 ceph-mon[32702]: [14/Sep/2024:03:48:54] ENGINE Bus STARTING 2024-09-14T03:48:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:55 smithi160 ceph-mon[32702]: [14/Sep/2024:03:48:54] ENGINE Serving on https://172.21.15.114:7150 2024-09-14T03:48:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:55 smithi160 ceph-mon[32702]: [14/Sep/2024:03:48:54] ENGINE Error in HTTPServer.serve 2024-09-14T03:48:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout: Traceback (most recent call last): 2024-09-14T03:48:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._run(expiration_interval) 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: s = self.context.wrap_socket( 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: return self.sslsocket_class._create( 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: self.do_handshake() 2024-09-14T03:48:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-14T03:48:55.660 INFO:journalctl@ceph.mon.b.smithi160.stdout: self._sslobj.do_handshake() 2024-09-14T03:48:55.660 INFO:journalctl@ceph.mon.b.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-14T03:48:55.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:55 smithi160 ceph-mon[32702]: [14/Sep/2024:03:48:54] ENGINE Serving on http://172.21.15.114:8765 2024-09-14T03:48:55.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:55 smithi160 ceph-mon[32702]: [14/Sep/2024:03:48:54] ENGINE Bus STARTED 2024-09-14T03:48:56.264 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 9552 9536 198.651 230.938 0.00331365 0.00502787 2024-09-14T03:48:57.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:48:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:48:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:48:57.264 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 12751 12735 198.968 199.938 0.00587402 0.00501939 2024-09-14T03:48:57.534 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:57 smithi114 ceph-mon[103029]: pgmap v5: 89 pgs: 89 active+clean; 1.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:57.534 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:57 smithi114 ceph-mon[103029]: mgrmap e39: y(active, since 4s), standbys: x 2024-09-14T03:48:57.534 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:57 smithi114 ceph-mon[40961]: pgmap v5: 89 pgs: 89 active+clean; 1.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:57.534 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:57 smithi114 ceph-mon[40961]: mgrmap e39: y(active, since 4s), standbys: x 2024-09-14T03:48:57.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:57 smithi160 ceph-mon[32702]: pgmap v5: 89 pgs: 89 active+clean; 1.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:57.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:57 smithi160 ceph-mon[32702]: mgrmap e39: y(active, since 4s), standbys: x 2024-09-14T03:48:58.128 INFO:teuthology.orchestra.run.smithi114.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:48:58.265 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 15625 15609 195.096 179.625 0.00606598 0.00512011 2024-09-14T03:48:59.265 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 18597 18581 193.536 185.75 0.00513895 0.00516089 2024-09-14T03:48:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:59 smithi160 ceph-mon[32702]: from='client.25895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:59 smithi160 ceph-mon[32702]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:48:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:59 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:48:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:59 smithi160 ceph-mon[32702]: pgmap v6: 89 pgs: 89 active+clean; 1.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:59 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:48:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:48:59 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:48:59.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[40961]: from='client.25895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:59.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[40961]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:48:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:48:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[40961]: pgmap v6: 89 pgs: 89 active+clean; 1.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:48:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:48:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[103029]: from='client.25895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:48:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[103029]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:48:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:48:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[103029]: pgmap v6: 89 pgs: 89 active+clean; 1.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:48:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:48:59.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:48:59 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:48:59.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'sleep 60' 2024-09-14T03:49:00.266 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 21336 21320 190.316 171.188 0.00438656 0.00524835 2024-09-14T03:49:00.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:00 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:00.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:00 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:00 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:00 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:00 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:00 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:00.659 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:01.266 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 23752 23736 185.393 151 0.00771668 0.00538772 2024-09-14T03:49:01.306 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:49:01.391 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:01 smithi160 ceph-mon[32702]: pgmap v7: 89 pgs: 89 active+clean; 979 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 25 KiB/s rd, 140 MiB/s wr, 2.25k op/s 2024-09-14T03:49:01.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:01 smithi114 ceph-mon[103029]: pgmap v7: 89 pgs: 89 active+clean; 979 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 25 KiB/s rd, 140 MiB/s wr, 2.25k op/s 2024-09-14T03:49:01.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:49:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:49:01] "GET /metrics HTTP/1.1" 200 33440 "" "Prometheus/2.51.0" 2024-09-14T03:49:01.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:01 smithi114 ceph-mon[40961]: pgmap v7: 89 pgs: 89 active+clean; 979 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 25 KiB/s rd, 140 MiB/s wr, 2.25k op/s 2024-09-14T03:49:02.266 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 26041 26025 180.688 143.062 0.00955527 0.00552788 2024-09-14T03:49:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:02 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:02 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:02 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:02 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:02 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:02 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:03.266 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 28384 28368 177.262 146.438 0.00390878 0.0056356 2024-09-14T03:49:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:03 smithi160 ceph-mon[32702]: pgmap v8: 89 pgs: 89 active+clean; 979 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 20 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-14T03:49:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:03 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:03 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:03.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:03 smithi114 ceph-mon[103029]: pgmap v8: 89 pgs: 89 active+clean; 979 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 20 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-14T03:49:03.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:03.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:03 smithi114 ceph-mon[40961]: pgmap v8: 89 pgs: 89 active+clean; 979 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 20 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-14T03:49:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:03 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:03 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:04.266 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 30683 30667 174.209 143.688 0.00905493 0.00573355 2024-09-14T03:49:04.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:04 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:05.266 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 32730 32714 170.352 127.938 0.00569678 0.00586481 2024-09-14T03:49:05.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:49:05.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:49:05.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[40961]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:49:05.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[40961]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:49:05.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:49:05.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:49:05.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[103029]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:49:05.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[103029]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:49:05.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:05 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:05 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:05 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:05 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:49:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:05 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:49:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:05 smithi160 ceph-mon[32702]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-14T03:49:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:05 smithi160 ceph-mon[32702]: Updating smithi160:/etc/ceph/ceph.conf 2024-09-14T03:49:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:05 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:05 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.019 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:06.019 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:06.266 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 35045 35029 168.377 144.688 0.0072467 0.00593397 2024-09-14T03:49:06.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: pgmap v9: 89 pgs: 89 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 16 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:49:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:49:06.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:49:06.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:49:06.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:49:06.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: pgmap v9: 89 pgs: 89 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 16 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-14T03:49:06.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:49:06.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:49:06.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:49:06.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:49:06.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:49:06.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:49:06.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:49:06.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:49:06.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: pgmap v9: 89 pgs: 89 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 16 KiB/s rd, 162 MiB/s wr, 2.59k op/s 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: Updating smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: Updating smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:49:06.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:49:06.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:49:07.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 37374 37358 166.747 145.562 0.00871661 0.00599256 2024-09-14T03:49:07.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:06 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:08.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 39628 39612 165.021 140.875 0.0041813 0.00605472 2024-09-14T03:49:08.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:08 smithi114 ceph-mon[103029]: pgmap v10: 89 pgs: 89 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-14T03:49:08.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:08.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:08 smithi114 ceph-mon[40961]: pgmap v10: 89 pgs: 89 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-14T03:49:08.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:08 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:08 smithi160 ceph-mon[32702]: pgmap v10: 89 pgs: 89 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-14T03:49:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:08 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:09.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 41819 41803 163.266 136.938 0.00691922 0.00611894 2024-09-14T03:49:10.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 43975 43959 161.588 134.75 0.0133942 0.00618394 2024-09-14T03:49:10.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:10 smithi114 ceph-mon[103029]: pgmap v11: 89 pgs: 89 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-14T03:49:10.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:10 smithi114 ceph-mon[40961]: pgmap v11: 89 pgs: 89 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-14T03:49:10.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:10.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:10 smithi160 ceph-mon[32702]: pgmap v11: 89 pgs: 89 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-09-14T03:49:11.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 46017 46001 159.701 127.625 0.00575829 0.00625658 2024-09-14T03:49:11.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:49:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:49:11] "GET /metrics HTTP/1.1" 200 36179 "" "Prometheus/2.51.0" 2024-09-14T03:49:12.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 48044 48028 157.963 126.688 0.00592726 0.00632523 2024-09-14T03:49:12.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:12 smithi114 ceph-mon[103029]: pgmap v12: 89 pgs: 89 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 15 KiB/s rd, 206 MiB/s wr, 3.31k op/s 2024-09-14T03:49:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:12 smithi114 ceph-mon[40961]: pgmap v12: 89 pgs: 89 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 15 KiB/s rd, 206 MiB/s wr, 3.31k op/s 2024-09-14T03:49:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:12 smithi160 ceph-mon[32702]: pgmap v12: 89 pgs: 89 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 15 KiB/s rd, 206 MiB/s wr, 3.31k op/s 2024-09-14T03:49:13.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:49:13.267000+0000 min lat: 0.00106836 max lat: 0.0986312 avg lat: 0.00640276 2024-09-14T03:49:13.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:49:13.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 49964 49948 156.064 120 0.0116663 0.00640276 2024-09-14T03:49:14.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 51903 51887 154.403 121.188 0.00766558 0.00647183 2024-09-14T03:49:14.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:14 smithi114 ceph-mon[103029]: pgmap v13: 89 pgs: 89 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 2k op/s 2024-09-14T03:49:14.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:14 smithi114 ceph-mon[40961]: pgmap v13: 89 pgs: 89 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 2k op/s 2024-09-14T03:49:14.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:14 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:14 smithi160 ceph-mon[32702]: pgmap v13: 89 pgs: 89 active+clean; 2.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 2k op/s 2024-09-14T03:49:15.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 53784 53768 152.728 117.562 0.0102191 0.0065426 2024-09-14T03:49:16.041 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:16.041 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:15.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:16.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 55612 55596 151.055 114.25 0.00530222 0.00661535 2024-09-14T03:49:16.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:16 smithi114 ceph-mon[103029]: pgmap v14: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 179 MiB/s wr, 2.86k op/s 2024-09-14T03:49:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:16 smithi114 ceph-mon[40961]: pgmap v14: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 179 MiB/s wr, 2.86k op/s 2024-09-14T03:49:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:16 smithi160 ceph-mon[32702]: pgmap v14: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 179 MiB/s wr, 2.86k op/s 2024-09-14T03:49:17.267 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 57433 57417 149.503 113.812 0.0102049 0.00668461 2024-09-14T03:49:17.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:18.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 59216 59200 147.98 111.438 0.0055902 0.00675215 2024-09-14T03:49:18.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:18 smithi114 ceph-mon[103029]: pgmap v15: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-14T03:49:18.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:18 smithi114 ceph-mon[40961]: pgmap v15: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-14T03:49:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:18 smithi160 ceph-mon[32702]: pgmap v15: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-14T03:49:19.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 60943 60927 146.439 107.938 0.0107016 0.00682391 2024-09-14T03:49:20.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 62612 62596 144.879 104.312 0.00605386 0.00689729 2024-09-14T03:49:20.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:20 smithi114 ceph-mon[103029]: pgmap v16: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-14T03:49:20.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:20 smithi114 ceph-mon[40961]: pgmap v16: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-14T03:49:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:20 smithi160 ceph-mon[32702]: pgmap v16: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 112 MiB/s wr, 1.80k op/s 2024-09-14T03:49:20.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:21.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 64282 64266 143.432 104.375 0.0123624 0.00696696 2024-09-14T03:49:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:49:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:49:21] "GET /metrics HTTP/1.1" 200 36179 "" "Prometheus/2.51.0" 2024-09-14T03:49:22.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 65890 65874 141.951 100.5 0.00898739 0.0070399 2024-09-14T03:49:22.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:22 smithi114 ceph-mon[103029]: pgmap v17: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-09-14T03:49:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:22 smithi114 ceph-mon[40961]: pgmap v17: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-09-14T03:49:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:22 smithi160 ceph-mon[32702]: pgmap v17: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-09-14T03:49:23.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 67458 67442 140.486 98 0.00577404 0.00711271 2024-09-14T03:49:23.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:23 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:23 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:24.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 15 68837 68822 138.736 86.25 0.00877652 0.00720301 2024-09-14T03:49:24.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:24 smithi114 ceph-mon[103029]: pgmap v18: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-14T03:49:24.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:24 smithi114 ceph-mon[40961]: pgmap v18: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-14T03:49:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:24 smithi160 ceph-mon[32702]: pgmap v18: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-14T03:49:24.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:24.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:25.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 70195 70179 137.051 84.8125 0.0108359 0.00729107 2024-09-14T03:49:26.078 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:25 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:26.079 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:25 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:25.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:26.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 71665 71649 135.682 91.875 0.00497666 0.00736504 2024-09-14T03:49:26.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:26 smithi114 ceph-mon[103029]: pgmap v19: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T03:49:26.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:26 smithi114 ceph-mon[40961]: pgmap v19: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T03:49:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:26 smithi160 ceph-mon[32702]: pgmap v19: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T03:49:27.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 73168 73152 134.454 93.9375 0.00618722 0.00743171 2024-09-14T03:49:27.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:28.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 74632 74616 133.226 91.5 0.0100607 0.00750071 2024-09-14T03:49:28.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:28 smithi114 ceph-mon[103029]: pgmap v20: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T03:49:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:28 smithi114 ceph-mon[40961]: pgmap v20: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T03:49:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:28 smithi160 ceph-mon[32702]: pgmap v20: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T03:49:29.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 76072 76056 132.025 90 0.00758415 0.00756871 2024-09-14T03:49:30.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 77495 77479 130.861 88.9375 0.00961302 0.00763666 2024-09-14T03:49:30.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:30 smithi114 ceph-mon[103029]: pgmap v21: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T03:49:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:30 smithi114 ceph-mon[40961]: pgmap v21: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T03:49:30.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:30.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:30 smithi160 ceph-mon[32702]: pgmap v21: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T03:49:31.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 78911 78895 129.746 88.5 0.0206753 0.00770241 2024-09-14T03:49:31.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:49:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:49:31] "GET /metrics HTTP/1.1" 200 36190 "" "Prometheus/2.51.0" 2024-09-14T03:49:32.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 80296 80280 128.638 86.5625 0.00618917 0.00776887 2024-09-14T03:49:32.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:32 smithi114 ceph-mon[103029]: pgmap v22: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-09-14T03:49:32.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:32 smithi114 ceph-mon[40961]: pgmap v22: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-09-14T03:49:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:32 smithi160 ceph-mon[32702]: pgmap v22: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-09-14T03:49:33.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:49:33.268763+0000 min lat: 0.00106836 max lat: 0.0986312 avg lat: 0.00783411 2024-09-14T03:49:33.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:49:33.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 81669 81653 127.568 85.8125 0.0080145 0.00783411 2024-09-14T03:49:34.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 83042 83026 126.549 85.8125 0.0107799 0.00789755 2024-09-14T03:49:34.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:34 smithi114 ceph-mon[103029]: pgmap v23: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-14T03:49:34.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:34 smithi114 ceph-mon[40961]: pgmap v23: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-14T03:49:34.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:34 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:34 smithi160 ceph-mon[32702]: pgmap v23: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-14T03:49:35.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 84384 84368 125.533 83.875 0.00781197 0.00796099 2024-09-14T03:49:36.107 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:36.107 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:36.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 85715 85699 124.548 83.1875 0.00953441 0.00802443 2024-09-14T03:49:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:36 smithi114 ceph-mon[103029]: pgmap v24: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-14T03:49:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:36 smithi114 ceph-mon[40961]: pgmap v24: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-14T03:49:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:36 smithi160 ceph-mon[32702]: pgmap v24: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-14T03:49:37.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 87035 87019 123.592 82.5 0.00908623 0.00808606 2024-09-14T03:49:37.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:38.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 88315 88299 122.623 80 0.0102938 0.0081503 2024-09-14T03:49:38.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:38 smithi114 ceph-mon[103029]: pgmap v25: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-14T03:49:38.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:38 smithi114 ceph-mon[40961]: pgmap v25: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-14T03:49:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:38 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:38 smithi160 ceph-mon[32702]: pgmap v25: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-14T03:49:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:38 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:39.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 89607 89591 121.713 80.75 0.0180764 0.00821145 2024-09-14T03:49:40.269 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 90871 90855 120.804 79 0.013275 0.00827267 2024-09-14T03:49:40.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:40 smithi114 ceph-mon[103029]: pgmap v26: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-14T03:49:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:40 smithi114 ceph-mon[40961]: pgmap v26: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-14T03:49:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:40 smithi160 ceph-mon[32702]: pgmap v26: 89 pgs: 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-14T03:49:41.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 92144 92128 119.945 79.5625 0.0102649 0.0083318 2024-09-14T03:49:41.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:41 smithi114 ceph-mon[103029]: pgmap v27: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-14T03:49:41.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:41 smithi114 ceph-mon[40961]: pgmap v27: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-14T03:49:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:41 smithi160 ceph-mon[32702]: pgmap v27: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-14T03:49:41.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:49:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:49:41] "GET /metrics HTTP/1.1" 200 36192 "" "Prometheus/2.51.0" 2024-09-14T03:49:42.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 93401 93385 119.1 78.5625 0.0217692 0.00839146 2024-09-14T03:49:43.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 94620 94604 118.242 76.1875 0.01406 0.00845142 2024-09-14T03:49:43.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:43 smithi114 ceph-mon[103029]: pgmap v28: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T03:49:43.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:43 smithi114 ceph-mon[40961]: pgmap v28: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T03:49:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:43 smithi160 ceph-mon[32702]: pgmap v28: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T03:49:44.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 95837 95821 117.415 76.0625 0.0185572 0.0085117 2024-09-14T03:49:44.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:45.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 96976 96960 116.525 71.1875 0.00423752 0.008577 2024-09-14T03:49:45.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:45 smithi114 ceph-mon[103029]: pgmap v29: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T03:49:45.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:45 smithi114 ceph-mon[40961]: pgmap v29: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T03:49:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:45 smithi160 ceph-mon[32702]: pgmap v29: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T03:49:46.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:46.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:46.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 98187 98171 115.755 75.6875 0.0119926 0.0086337 2024-09-14T03:49:47.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 99380 99364 114.992 74.5625 0.0186805 0.00869135 2024-09-14T03:49:47.348 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:47.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:47 smithi114 ceph-mon[103029]: pgmap v30: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-14T03:49:47.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:47 smithi114 ceph-mon[40961]: pgmap v30: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-14T03:49:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:47 smithi160 ceph-mon[32702]: pgmap v30: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-14T03:49:48.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 100563 100547 114.245 73.9375 0.0147362 0.00874786 2024-09-14T03:49:49.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 15 101728 101713 113.506 72.875 0.0124498 0.00880519 2024-09-14T03:49:49.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:49 smithi114 ceph-mon[103029]: pgmap v31: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-14T03:49:49.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:49 smithi114 ceph-mon[40961]: pgmap v31: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-14T03:49:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:49 smithi160 ceph-mon[32702]: pgmap v31: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-14T03:49:50.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 102873 102857 112.769 71.5 0.0145134 0.00886263 2024-09-14T03:49:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:51.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 104015 103999 112.056 71.375 0.0129057 0.00891922 2024-09-14T03:49:51.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:51 smithi114 ceph-mon[40961]: pgmap v32: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-14T03:49:51.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:49:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:49:51] "GET /metrics HTTP/1.1" 200 36192 "" "Prometheus/2.51.0" 2024-09-14T03:49:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:51 smithi114 ceph-mon[103029]: pgmap v32: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-14T03:49:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:51 smithi160 ceph-mon[32702]: pgmap v32: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-14T03:49:52.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 105156 105140 111.365 71.3125 0.0102777 0.0089746 2024-09-14T03:49:53.270 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:49:53.270513+0000 min lat: 0.00106836 max lat: 0.0986312 avg lat: 0.00903208 2024-09-14T03:49:53.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:49:53.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 106257 106241 110.656 68.8125 0.010225 0.00903208 2024-09-14T03:49:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:53 smithi160 ceph-mon[32702]: pgmap v33: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-14T03:49:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:53 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:53.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:53 smithi114 ceph-mon[40961]: pgmap v33: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-14T03:49:53.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:53 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:53 smithi114 ceph-mon[103029]: pgmap v33: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-14T03:49:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:49:54.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 107356 107340 109.968 68.6875 0.0132559 0.00908835 2024-09-14T03:49:54.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:55.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 108424 108408 109.27 66.75 0.00503061 0.00914609 2024-09-14T03:49:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:55 smithi160 ceph-mon[32702]: pgmap v34: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-14T03:49:55.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:55 smithi114 ceph-mon[103029]: pgmap v34: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-14T03:49:55.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:55 smithi114 ceph-mon[40961]: pgmap v34: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-14T03:49:56.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:56.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:56.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 109517 109501 108.62 68.3125 0.018243 0.00920122 2024-09-14T03:49:57.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 110614 110598 107.994 68.5625 0.00606829 0.00925486 2024-09-14T03:49:57.388 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:49:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:49:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:49:57.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:57 smithi160 ceph-mon[32702]: pgmap v35: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-14T03:49:57.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:57 smithi114 ceph-mon[40961]: pgmap v35: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-14T03:49:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:57 smithi114 ceph-mon[103029]: pgmap v35: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-14T03:49:58.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 111692 111676 107.369 67.375 0.00876173 0.00930827 2024-09-14T03:49:59.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 112752 112736 106.746 66.25 0.0125599 0.00936273 2024-09-14T03:49:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:49:59 smithi160 ceph-mon[32702]: pgmap v36: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-14T03:49:59.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:49:59 smithi114 ceph-mon[103029]: pgmap v36: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-14T03:49:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:49:59 smithi114 ceph-mon[40961]: pgmap v36: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-14T03:50:00.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 113803 113787 106.133 65.6875 0.00674804 0.00941686 2024-09-14T03:50:00.558 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:00 smithi160 ceph-mon[32702]: overall HEALTH_OK 2024-09-14T03:50:00.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:00 smithi114 ceph-mon[103029]: overall HEALTH_OK 2024-09-14T03:50:00.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:00 smithi114 ceph-mon[40961]: overall HEALTH_OK 2024-09-14T03:50:01.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 114851 114835 105.536 65.5 0.030147 0.00946979 2024-09-14T03:50:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:01 smithi160 ceph-mon[32702]: pgmap v37: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-14T03:50:01.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:01 smithi114 ceph-mon[103029]: pgmap v37: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-14T03:50:01.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:50:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:50:01] "GET /metrics HTTP/1.1" 200 36190 "" "Prometheus/2.51.0" 2024-09-14T03:50:01.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:01 smithi114 ceph-mon[40961]: pgmap v37: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-14T03:50:02.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 115890 115874 104.947 64.9375 0.0112561 0.00952373 2024-09-14T03:50:03.271 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 116940 116924 104.385 65.625 0.0052039 0.00957488 2024-09-14T03:50:03.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:03 smithi114 ceph-mon[103029]: pgmap v38: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-14T03:50:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:03 smithi114 ceph-mon[40961]: pgmap v38: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-14T03:50:03.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:03 smithi160 ceph-mon[32702]: pgmap v38: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-14T03:50:04.272 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 117969 117953 103.821 64.3125 0.0115613 0.00962618 2024-09-14T03:50:04.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:04 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:05.272 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 118983 118967 103.259 63.375 0.00509236 0.00967934 2024-09-14T03:50:05.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch upgrade resume' 2024-09-14T03:50:05.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:05 smithi114 ceph-mon[103029]: pgmap v39: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T03:50:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:05 smithi114 ceph-mon[40961]: pgmap v39: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T03:50:05.741 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:05 smithi160 ceph-mon[32702]: pgmap v39: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T03:50:05.847 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:50:06.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:06.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:06.272 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 119858 119842 102.594 54.6875 0.0093898 0.00974237 2024-09-14T03:50:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:06 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:06.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:06 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:06.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:07.272 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 120830 120814 102.028 60.75 0.0217524 0.00979563 2024-09-14T03:50:07.312 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:06 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:07.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:07 smithi114 ceph-mon[40961]: pgmap v40: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 56 MiB/s wr, 890 op/s 2024-09-14T03:50:07.532 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:07 smithi114 ceph-mon[103029]: pgmap v40: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 56 MiB/s wr, 890 op/s 2024-09-14T03:50:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:07 smithi160 ceph-mon[32702]: pgmap v40: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 56 MiB/s wr, 890 op/s 2024-09-14T03:50:08.272 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 121762 121746 101.444 58.25 0.0263695 0.00985207 2024-09-14T03:50:08.546 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:08 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:08.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:08.666 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:08 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:08.844 INFO:teuthology.orchestra.run.smithi114.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:50:09.272 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 122660 122644 100.848 56.125 0.0204229 0.00991078 2024-09-14T03:50:09.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'echo wait for majority of mons to upgrade' 2024-09-14T03:50:09.848 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:09 smithi114 ceph-mon[103029]: pgmap v41: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 890 op/s 2024-09-14T03:50:09.848 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:09 smithi114 ceph-mon[103029]: from='client.25898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:09.848 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:09 smithi114 ceph-mon[103029]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:50:09.849 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:09.849 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:09 smithi114 ceph-mon[40961]: pgmap v41: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 890 op/s 2024-09-14T03:50:09.849 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:09 smithi114 ceph-mon[40961]: from='client.25898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:09.849 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:09 smithi114 ceph-mon[40961]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:50:09.849 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:09 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:09 smithi160 ceph-mon[32702]: pgmap v41: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 890 op/s 2024-09-14T03:50:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:09 smithi160 ceph-mon[32702]: from='client.25898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:09 smithi160 ceph-mon[32702]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:50:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:09 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:10.272 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 123496 123480 100.217 52.25 0.00657536 0.00997339 2024-09-14T03:50:10.446 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:50:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:11.272 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 124413 124397 99.6667 57.3125 0.00554809 0.0100285 2024-09-14T03:50:11.747 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:50:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:50:11] "GET /metrics HTTP/1.1" 200 36185 "" "Prometheus/2.51.0" 2024-09-14T03:50:12.117 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:11 smithi114 ceph-mon[40961]: pgmap v42: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-14T03:50:12.118 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:11 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:12.118 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:11 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:12.118 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:11 smithi114 ceph-mon[103029]: pgmap v42: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-14T03:50:12.118 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:12.118 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:11 smithi160 ceph-mon[32702]: pgmap v42: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-14T03:50:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:11 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:11 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:12.272 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 125325 125309 99.1266 57 0.0108653 0.0100825 2024-09-14T03:50:12.570 INFO:teuthology.orchestra.run.smithi114.stdout:wait for majority of mons to upgrade 2024-09-14T03:50:13.272 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:50:13.272275+0000 min lat: 0.00106836 max lat: 0.119987 avg lat: 0.0101334 2024-09-14T03:50:13.273 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:50:13.273 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 126282 126266 98.6351 59.8125 0.0143474 0.0101334 2024-09-14T03:50:13.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'while ! ceph mon versions | grep $sha1 | egrep '"'"': [23]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-14T03:50:14.085 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:13 smithi114 ceph-mon[40961]: pgmap v43: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 52 MiB/s wr, 838 op/s 2024-09-14T03:50:14.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:13 smithi114 ceph-mon[103029]: pgmap v43: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 52 MiB/s wr, 838 op/s 2024-09-14T03:50:14.149 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:13 smithi160 ceph-mon[32702]: pgmap v43: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 52 MiB/s wr, 838 op/s 2024-09-14T03:50:14.273 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 127159 127143 98.094 54.8125 0.0139832 0.010189 2024-09-14T03:50:14.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:14 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:14.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:14.687 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:50:15.273 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 15 128031 128016 97.5626 54.5625 0.0266889 0.0102443 2024-09-14T03:50:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[103029]: pgmap v44: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-14T03:50:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:50:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:16.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:16.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:50:16.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-14T03:50:16.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-14T03:50:16.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[40961]: pgmap v44: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-14T03:50:16.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:16.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:16.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:16.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:50:16.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:16.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:16.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:50:16.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-14T03:50:16.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:15 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-14T03:50:16.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:16.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:15.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-mon[32702]: pgmap v44: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-14T03:50:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:16.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:16.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:50:16.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:16.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:16.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:50:16.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-14T03:50:16.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:15 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-14T03:50:16.273 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 128859 128843 97.0099 51.6875 0.0102169 0.010303 2024-09-14T03:50:16.854 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:16 smithi114 ceph-mon[103029]: Upgrade: It appears safe to stop mon.c 2024-09-14T03:50:16.854 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:16 smithi114 ceph-mon[40961]: Upgrade: It appears safe to stop mon.c 2024-09-14T03:50:17.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:17.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:16 smithi160 ceph-mon[32702]: Upgrade: It appears safe to stop mon.c 2024-09-14T03:50:17.273 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 129777 129761 96.538 57.375 0.00714731 0.0103537 2024-09-14T03:50:18.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:17 smithi114 ceph-mon[40961]: pgmap v45: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-14T03:50:18.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:17 smithi114 ceph-mon[103029]: pgmap v45: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-14T03:50:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:17 smithi160 ceph-mon[32702]: pgmap v45: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-14T03:50:18.273 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 130660 130644 96.0515 55.1875 0.0152325 0.0104059 2024-09-14T03:50:19.078 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:18 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2593813683' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:50:19.078 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:18 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/2593813683' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:50:19.078 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:50:19.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:18 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/2593813683' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-14T03:50:19.273 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 131452 131436 95.5101 49.5 0.0335488 0.0104642 2024-09-14T03:50:20.273 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 132322 132306 95.0373 54.375 0.0200553 0.0105171 2024-09-14T03:50:20.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[40961]: pgmap v46: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-14T03:50:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[40961]: Upgrade: Updating mon.c 2024-09-14T03:50:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:50:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:50:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[40961]: Deploying daemon mon.c on smithi114 2024-09-14T03:50:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[40961]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:20.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[103029]: pgmap v46: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-14T03:50:20.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[103029]: Upgrade: Updating mon.c 2024-09-14T03:50:20.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:20.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:50:20.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:50:20.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:20.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[103029]: Deploying daemon mon.c on smithi114 2024-09-14T03:50:20.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:20 smithi114 ceph-mon[103029]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:20.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:20 smithi160 ceph-mon[32702]: pgmap v46: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-14T03:50:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:20 smithi160 ceph-mon[32702]: Upgrade: Updating mon.c 2024-09-14T03:50:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:20 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:20 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:50:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:20 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:50:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:20 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:20 smithi160 ceph-mon[32702]: Deploying daemon mon.c on smithi114 2024-09-14T03:50:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:20 smithi160 ceph-mon[32702]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:21.273 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 133169 133153 94.5588 52.9375 0.0127517 0.0105687 2024-09-14T03:50:21.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:21 smithi114 ceph-mon[40961]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:21 smithi114 ceph-mon[103029]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:21 smithi160 ceph-mon[32702]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:50:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:50:21] "GET /metrics HTTP/1.1" 200 36185 "" "Prometheus/2.51.0" 2024-09-14T03:50:22.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 133936 133920 94.0349 47.9375 0.00745701 0.010629 2024-09-14T03:50:22.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:22 smithi114 ceph-mon[103029]: pgmap v47: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T03:50:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:22 smithi114 ceph-mon[40961]: pgmap v47: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T03:50:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:22 smithi160 ceph-mon[32702]: pgmap v47: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T03:50:22.835 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:50:23.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:22 smithi114 systemd[1]: Stopping Ceph mon.c for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:50:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:23 smithi114 ceph-mon[40961]: from='client.? 172.21.15.114:0/329865871' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:50:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:23 smithi114 ceph-mon[40961]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:23.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/329865871' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:50:23.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:23.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 90 4 134633 134629 93.4824 44.3125 0.0199597 0.0106927 2024-09-14T03:50:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:23 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/329865871' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:50:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:23 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:23.535 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:23 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c[40938]: 2024-09-14T03:50:23.319+0000 7fdc01d5e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T03:50:23.535 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:23 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c[40938]: 2024-09-14T03:50:23.319+0000 7fdc01d5e640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-14T03:50:23.836 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:23 smithi114 podman[106758]: 2024-09-14 03:50:23.533966677 +0000 UTC m=+0.351871720 container died f8446e48bf23137090eed5dc9a29f49a9a408c3811149877395dba8e4154f601 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, RELEASE=quincy-eb16b97, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-14T03:50:24.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:23 smithi114 podman[106758]: 2024-09-14 03:50:23.925902218 +0000 UTC m=+0.743807197 container cleanup f8446e48bf23137090eed5dc9a29f49a9a408c3811149877395dba8e4154f601 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, RELEASE=quincy-eb16b97, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-14T03:50:24.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:23 smithi114 bash[106758]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c 2024-09-14T03:50:24.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0143 2024-09-14T03:50:24.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 134633 2024-09-14T03:50:24.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-14T03:50:24.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-14T03:50:24.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 93.4803 2024-09-14T03:50:24.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 39.7418 2024-09-14T03:50:24.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 230.938 2024-09-14T03:50:24.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 44.3125 2024-09-14T03:50:24.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1495 2024-09-14T03:50:24.274 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 635.869 2024-09-14T03:50:24.275 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3695 2024-09-14T03:50:24.275 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 709 2024-09-14T03:50:24.275 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0106928 2024-09-14T03:50:24.275 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.00721189 2024-09-14T03:50:24.275 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 0.151917 2024-09-14T03:50:24.275 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00106836 2024-09-14T03:50:24.281 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-14T03:50:24.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:24.439 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:24 smithi114 podman[106776]: 2024-09-14 03:50:24.185901054 +0000 UTC m=+0.646054153 container remove f8446e48bf23137090eed5dc9a29f49a9a408c3811149877395dba8e4154f601 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-eb16b97, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-14T03:50:24.803 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:24 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-14T03:50:24.804 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:24 smithi114 systemd[1]: Stopped Ceph mon.c for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:50:24.804 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:24 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.c.service: Consumed 9.352s CPU time. 2024-09-14T03:50:25.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:24 smithi114 systemd[1]: Starting Ceph mon.c for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:50:25.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 podman[106899]: 2024-09-14 03:50:25.074143982 +0000 UTC m=+0.039760858 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:50:25.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 podman[106899]: 2024-09-14 03:50:25.187957258 +0000 UTC m=+0.153574125 container create 0d84986478c252d1c0c3825a10e139c5f5975678f330cdcd957cd7af280dd81e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-14T03:50:25.876 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 podman[106899]: 2024-09-14 03:50:25.50757622 +0000 UTC m=+0.473193095 container init 0d84986478c252d1c0c3825a10e139c5f5975678f330cdcd957cd7af280dd81e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, RELEASE=main-a278231, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T03:50:25.876 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 podman[106899]: 2024-09-14 03:50:25.523250135 +0000 UTC m=+0.488866999 container start 0d84986478c252d1c0c3825a10e139c5f5975678f330cdcd957cd7af280dd81e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-a278231) 2024-09-14T03:50:25.876 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: set uid:gid to 167:167 (ceph:ceph) 2024-09-14T03:50:25.876 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev), process ceph-mon, pid 7 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: pidfile_write: ignore empty --pid-file 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: load: jerasure load: lrc 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: RocksDB version: 7.9.2 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Git sha 0 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Compile date 2024-09-12 20:09:22 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: DB SUMMARY 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: DB Session ID: O7BSH2YGBHRQ9FYPC0SK 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: CURRENT file: CURRENT 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: IDENTITY file: IDENTITY 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1103 Bytes 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-09-14T03:50:25.877 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 5895161 ; 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.error_if_exists: 0 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.create_if_missing: 0 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.paranoid_checks: 1 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.env: 0x56018c1a0120 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.fs: PosixFileSystem 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.info_log: 0x56018df69700 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_file_opening_threads: 16 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.statistics: (nil) 2024-09-14T03:50:25.878 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.use_fsync: 0 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_log_file_size: 0 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.keep_log_file_num: 1000 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.recycle_log_file_num: 0 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.allow_fallocate: 1 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.allow_mmap_reads: 0 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.allow_mmap_writes: 0 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.use_direct_reads: 0 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.create_missing_column_families: 0 2024-09-14T03:50:25.879 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.db_log_dir: 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.wal_dir: 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.advise_random_on_open: 1 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.db_write_buffer_size: 0 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.write_buffer_manager: 0x56018df6c1e0 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-14T03:50:25.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.rate_limiter: (nil) 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.wal_recovery_mode: 2 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.enable_thread_tracking: 0 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.enable_pipelined_write: 0 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.unordered_write: 0 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.row_cache: None 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.wal_filter: None 2024-09-14T03:50:25.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.allow_ingest_behind: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.two_write_queues: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.manual_wal_flush: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.wal_compression: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.atomic_flush: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.log_readahead_size: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.best_efforts_recovery: 0 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-14T03:50:25.882 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.allow_data_in_errors: 0 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.db_host_id: __hostname__ 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_background_jobs: 2 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_background_compactions: -1 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_subcompactions: 1 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_total_wal_size: 0 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-14T03:50:25.883 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_open_files: -1 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bytes_per_sync: 0 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_readahead_size: 0 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_background_flushes: -1 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Compression algorithms supported: 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: kZSTD supported: 0 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: kXpressCompression supported: 0 2024-09-14T03:50:25.884 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: kBZip2Compression supported: 0 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: kLZ4Compression supported: 1 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: kZlibCompression supported: 1 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: kLZ4HCCompression supported: 1 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: kSnappyCompression supported: 1 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.merge_operator: 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_filter: None 2024-09-14T03:50:25.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_filter_factory: None 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.sst_partitioner_factory: None 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56018df69320) 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_type: 0 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout: data_block_index_type: 0 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_shortening: 1 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-14T03:50:25.886 INFO:journalctl@ceph.mon.c.smithi114.stdout: checksum: 4 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: no_block_cache: 0 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache: 0x56018df5d610 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_options: 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: capacity : 536870912 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: num_shard_bits : 4 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: strict_capacity_limit : 0 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_cache_compressed: (nil) 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: persistent_cache: (nil) 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_size: 4096 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_size_deviation: 10 2024-09-14T03:50:25.887 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_restart_interval: 16 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: index_block_restart_interval: 1 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: metadata_block_size: 4096 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: partition_filters: 0 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: use_delta_encoding: 1 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: filter_policy: bloomfilter 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: whole_key_filtering: 1 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: verify_compression: 0 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: format_version: 5 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: enable_index_compression: 1 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: block_align: 0 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: max_auto_readahead_size: 262144 2024-09-14T03:50:25.888 INFO:journalctl@ceph.mon.c.smithi114.stdout: prepopulate_block_cache: 0 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout: initial_auto_readahead_size: 8192 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.write_buffer_size: 33554432 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_write_buffer_number: 2 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compression: NoCompression 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bottommost_compression: Disabled 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.prefix_extractor: nullptr 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.num_levels: 7 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-14T03:50:25.889 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compression_opts.level: 32767 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compression_opts.strategy: 0 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-14T03:50:25.890 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compression_opts.enabled: false 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.target_file_size_base: 67108864 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-14T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.arena_block_size: 1048576 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-14T03:50:25.892 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.disable_auto_compactions: 0 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.inplace_update_support: 0 2024-09-14T03:50:25.893 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.bloom_locality: 0 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.max_successive_merges: 0 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.paranoid_file_checks: 0 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.force_consistency_checks: 1 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.report_bg_io_stats: 0 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.ttl: 2592000 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-14T03:50:25.894 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.enable_blob_files: false 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.min_blob_size: 0 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.blob_file_size: 268435456 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.blob_file_starting_level: 0 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-14T03:50:25.895 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12804, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2a6f2c61-5a28-4a8c-93dc-83e36b501fc4 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285825578664, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285825615106, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3695697, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12809, "largest_seqno": 14910, "table_properties": {"data_size": 3686623, "index_size": 5916, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2117, "raw_key_size": 18935, "raw_average_key_size": 22, "raw_value_size": 3668602, "raw_average_value_size": 4377, "num_data_blocks": 267, "num_entries": 838, "num_filter_entries": 838, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726285825, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2a6f2c61-5a28-4a8c-93dc-83e36b501fc4", "db_session_id": "O7BSH2YGBHRQ9FYPC0SK", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285825615296, "job": 1, "event": "recovery_finished"} 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56018df8ce00 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: DB pointer 0x56018e08e000 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-14T03:50:25.896 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** DB Stats ** 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-14T03:50:25.897 INFO:journalctl@ceph.mon.c.smithi114.stdout: L0 1/0 3.52 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 216.5 0.02 0.00 1 0.016 0 0 0.0 0.0 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: L6 1/0 7.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: Sum 2/0 10.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 216.5 0.02 0.00 1 0.016 0 0 0.0 0.0 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 216.5 0.02 0.00 1 0.016 0 0 0.0 0.0 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 216.5 0.02 0.00 1 0.016 0 0 0.0 0.0 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:50:25.898 INFO:journalctl@ceph.mon.c.smithi114.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: Cumulative compaction: 0.00 GB write, 70.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: Interval compaction: 0.00 GB write, 70.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: Block cache BinnedLRUCache@0x56018df5d610#7 capacity: 512.00 MB usage: 63.78 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: Block cache entry stats(count,size,portion): DataBlock(4,30.31 KB,0.00578165%) FilterBlock(2,11.31 KB,0.00215769%) IndexBlock(2,22.16 KB,0.00422597%) Misc(1,0.00 KB,0%) 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: starting mon.c rank 2 at public addrs [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] at bind addrs [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:25.899 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 ceph-mon[106969]: mon.c@-1(???) e3 preinit fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:25.900 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 bash[106899]: 0d84986478c252d1c0c3825a10e139c5f5975678f330cdcd957cd7af280dd81e 2024-09-14T03:50:25.900 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:25 smithi114 systemd[1]: Started Ceph mon.c for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:50:26.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:25 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:26.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:25 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:25.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:27.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:30 smithi160 ceph-mon[32702]: pgmap v48: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 46 MiB/s wr, 741 op/s 2024-09-14T03:50:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:30 smithi160 ceph-mon[32702]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:31.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:30 smithi114 ceph-mon[103029]: pgmap v48: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 46 MiB/s wr, 741 op/s 2024-09-14T03:50:31.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:30 smithi114 ceph-mon[103029]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:31.771 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:50:31] "GET /metrics HTTP/1.1" 200 36184 "" "Prometheus/2.51.0" 2024-09-14T03:50:32.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: mon.c calling monitor election 2024-09-14T03:50:32.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: mon.a calling monitor election 2024-09-14T03:50:32.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: mon.b calling monitor election 2024-09-14T03:50:32.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: monmap epoch 3 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: last_changed 2024-09-14T03:38:03.057799+0000 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: min_mon_release 17 (quincy) 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: election_strategy: 1 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:50:32.036 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: mgrmap e39: y(active, since 99s), standbys: x 2024-09-14T03:50:32.037 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-14T03:50:32.037 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:31 smithi160 ceph-mon[32702]: Cluster is now healthy 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: mon.c calling monitor election 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: mon.a calling monitor election 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: mon.b calling monitor election 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: monmap epoch 3 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: last_changed 2024-09-14T03:38:03.057799+0000 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: min_mon_release 17 (quincy) 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: election_strategy: 1 2024-09-14T03:50:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: mgrmap e39: y(active, since 99s), standbys: x 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: pgmap v48: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 46 MiB/s wr, 741 op/s 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: mon.c calling monitor election 2024-09-14T03:50:32.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: mon.a calling monitor election 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: mon.b calling monitor election 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: monmap epoch 3 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: last_changed 2024-09-14T03:38:03.057799+0000 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: min_mon_release 17 (quincy) 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: election_strategy: 1 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:50:32.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:50:32.125 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: mgrmap e39: y(active, since 99s), standbys: x 2024-09-14T03:50:32.125 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-14T03:50:32.125 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:31 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T03:50:32.506 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:50:33.337 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:32 smithi160 ceph-mon[32702]: overall HEALTH_OK 2024-09-14T03:50:33.337 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:32 smithi160 ceph-mon[32702]: mgrmap e40: y(active, since 99s), standbys: x 2024-09-14T03:50:33.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:32 smithi114 ceph-mon[106969]: overall HEALTH_OK 2024-09-14T03:50:33.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:32 smithi114 ceph-mon[106969]: mgrmap e40: y(active, since 99s), standbys: x 2024-09-14T03:50:33.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:32 smithi114 ceph-mon[103029]: overall HEALTH_OK 2024-09-14T03:50:33.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:32 smithi114 ceph-mon[103029]: mgrmap e40: y(active, since 99s), standbys: x 2024-09-14T03:50:34.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:33 smithi114 ceph-mon[106969]: pgmap v53: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 618 op/s 2024-09-14T03:50:34.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:33 smithi114 ceph-mon[106969]: from='client.25549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:34.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:33 smithi114 ceph-mon[106969]: from='client.25555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:34.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:33 smithi114 ceph-mon[103029]: pgmap v53: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 618 op/s 2024-09-14T03:50:34.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:33 smithi114 ceph-mon[103029]: from='client.25549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:34.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:33 smithi114 ceph-mon[103029]: from='client.25555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:34.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:34 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:33 smithi160 ceph-mon[32702]: pgmap v53: 89 pgs: 89 active+clean; 8.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 618 op/s 2024-09-14T03:50:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:33 smithi160 ceph-mon[32702]: from='client.25549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:33 smithi160 ceph-mon[32702]: from='client.25555 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:35.686 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev)": 2 2024-09-14T03:50:35.687 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:50:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:50:35.657+0000 7fd419828640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-14T03:50:36.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:35 smithi114 ceph-mon[106969]: pgmap v54: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 992 op/s 2024-09-14T03:50:36.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:35 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/106439398' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:50:36.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:35 smithi114 ceph-mon[103029]: pgmap v54: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 992 op/s 2024-09-14T03:50:36.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:35 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/106439398' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:50:36.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:36.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:35 smithi160 ceph-mon[32702]: pgmap v54: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 20 MiB/s wr, 992 op/s 2024-09-14T03:50:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:35 smithi160 ceph-mon[32702]: from='client.? 172.21.15.114:0/106439398' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:50:36.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch ps' 2024-09-14T03:50:37.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:37.073 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:37.174 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:50:37.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:37.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:37.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:37 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:37.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:37 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:37.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:38.088 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:38 smithi114 ceph-mon[103029]: pgmap v55: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 665 op/s 2024-09-14T03:50:38.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:38 smithi114 ceph-mon[106969]: pgmap v55: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 665 op/s 2024-09-14T03:50:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:38 smithi160 ceph-mon[32702]: pgmap v55: 89 pgs: 89 active+clean; 7.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 665 op/s 2024-09-14T03:50:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:38 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:38 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:38 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:40 smithi114 ceph-mon[103029]: pgmap v56: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 665 op/s 2024-09-14T03:50:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:40 smithi114 ceph-mon[106969]: pgmap v56: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 665 op/s 2024-09-14T03:50:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:40 smithi160 ceph-mon[32702]: pgmap v56: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 665 op/s 2024-09-14T03:50:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:40.737 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-14T03:50:40.737 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.a smithi114 *:9093,9094 running (3m) 4s ago 8m 18.0M - 0.27.0 11f11916f8cd eaf39a5cb581 2024-09-14T03:50:40.737 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.a smithi160 *:3000 running (3m) 29s ago 8m 85.1M - 9.4.7 954c08fa6188 aa46fcb7cedc 2024-09-14T03:50:40.737 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi114.ebmrat smithi114 running (7m) 4s ago 7m 17.9M - 17.2.7-1629-geb16b97b ab51d1bebd12 00f4a6ea534e 2024-09-14T03:50:40.737 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi160.ywyjae smithi160 running (7m) 29s ago 7m 14.5M - 17.2.7-1629-geb16b97b ab51d1bebd12 a87b5ff7e907 2024-09-14T03:50:40.737 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x smithi160 *:8443,9283,8765 running (2m) 29s ago 12m 453M - 19.3.0-4907-ga2782315 8801d2529e43 574eaed954d9 2024-09-14T03:50:40.737 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y smithi114 *:8443,9283,8765 running (3m) 4s ago 14m 507M - 19.3.0-4907-ga2782315 8801d2529e43 4f6e65854aef 2024-09-14T03:50:40.737 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a smithi114 running (117s) 4s ago 14m 40.6M 2048M 19.3.0-4907-ga2782315 8801d2529e43 2ca35e38ecf5 2024-09-14T03:50:40.737 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b smithi160 running (12m) 29s ago 12m 49.5M 2048M 17.2.7-1629-geb16b97b ab51d1bebd12 a30277b5b152 2024-09-14T03:50:40.737 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c smithi114 running (15s) 4s ago 12m 22.5M 2048M 19.3.0-4907-ga2782315 8801d2529e43 0d84986478c2 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.a smithi114 *:9100 running (3m) 4s ago 8m 10.4M - 1.7.0 72c9c2088986 ff1fe8404b33 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.b smithi160 *:9100 running (3m) 29s ago 8m 9.96M - 1.7.0 72c9c2088986 289de68494ec 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (12m) 4s ago 12m 380M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 ddb654f06efc 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (11m) 4s ago 11m 330M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 a709e75520db 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (11m) 4s ago 11m 465M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 a76e6ee3bc54 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi114 running (10m) 4s ago 10m 486M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 b43e985ecf6c 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi160 running (10m) 29s ago 10m 390M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 e89693bb3b76 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi160 running (10m) 29s ago 10m 416M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 fb664f7f461a 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 smithi160 running (9m) 29s ago 9m 280M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 0403446dedd1 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 smithi160 running (9m) 29s ago 9m 358M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 39f0a617d104 2024-09-14T03:50:40.738 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.a smithi160 *:9095 running (2m) 29s ago 8m 48.5M - 2.51.0 1d3b7f56885b 1b09e4c45bc1 2024-09-14T03:50:41.549 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:50:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:50:41] "GET /metrics HTTP/1.1" 200 36181 "" "Prometheus/2.51.0" 2024-09-14T03:50:41.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch upgrade pause' 2024-09-14T03:50:42.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:42 smithi114 ceph-mon[103029]: pgmap v57: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 1.02k op/s 2024-09-14T03:50:42.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:42 smithi114 ceph-mon[103029]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:42 smithi114 ceph-mon[106969]: pgmap v57: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 1.02k op/s 2024-09-14T03:50:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:42 smithi114 ceph-mon[106969]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:42 smithi160 ceph-mon[32702]: pgmap v57: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 85 B/s wr, 1.02k op/s 2024-09-14T03:50:42.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:42 smithi160 ceph-mon[32702]: from='client.44102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:42.484 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:50:44.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:44 smithi114 ceph-mon[103029]: pgmap v58: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 730 op/s 2024-09-14T03:50:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:44 smithi114 ceph-mon[106969]: pgmap v58: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 730 op/s 2024-09-14T03:50:44.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:44.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:44 smithi160 ceph-mon[32702]: pgmap v58: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 730 op/s 2024-09-14T03:50:45.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: pgmap v59: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:50:45.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:45.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:45.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:45.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:50:45.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:45.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:45.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:50:45.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-14T03:50:45.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-14T03:50:45.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[103029]: Upgrade: It appears safe to stop mon.b 2024-09-14T03:50:45.556 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: pgmap v59: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:50:45.556 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:45.556 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:45.556 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:45.556 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:50:45.556 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:45.557 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:45.557 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:50:45.557 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-14T03:50:45.557 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-14T03:50:45.557 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:45 smithi114 ceph-mon[106969]: Upgrade: It appears safe to stop mon.b 2024-09-14T03:50:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: pgmap v59: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:50:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:45.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:45.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:45.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:50:45.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:45.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:45.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:50:45.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-14T03:50:45.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-14T03:50:45.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-mon[32702]: Upgrade: It appears safe to stop mon.b 2024-09-14T03:50:46.076 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:46.076 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:46.089 INFO:teuthology.orchestra.run.smithi114.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:50:46.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'sleep 60' 2024-09-14T03:50:47.273 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:50:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[106969]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[106969]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:50:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[106969]: Upgrade: Updating mon.b 2024-09-14T03:50:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:50:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:50:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[106969]: Deploying daemon mon.b on smithi160 2024-09-14T03:50:47.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[103029]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:47.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[103029]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:50:47.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:47.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[103029]: Upgrade: Updating mon.b 2024-09-14T03:50:47.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:47.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:50:47.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:50:47.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:47.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:47 smithi114 ceph-mon[103029]: Deploying daemon mon.b on smithi160 2024-09-14T03:50:47.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:47 smithi160 ceph-mon[32702]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:50:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:47 smithi160 ceph-mon[32702]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:50:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:47 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:47 smithi160 ceph-mon[32702]: Upgrade: Updating mon.b 2024-09-14T03:50:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:47 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:47 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:50:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:47 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:50:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:47 smithi160 ceph-mon[32702]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:50:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:47 smithi160 ceph-mon[32702]: Deploying daemon mon.b on smithi160 2024-09-14T03:50:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:48 smithi160 ceph-mon[32702]: pgmap v60: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 719 op/s 2024-09-14T03:50:48.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:48 smithi114 ceph-mon[103029]: pgmap v60: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 719 op/s 2024-09-14T03:50:48.598 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:48 smithi114 ceph-mon[106969]: pgmap v60: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 719 op/s 2024-09-14T03:50:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:48 smithi160 systemd[1]: Stopping Ceph mon.b for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:50:49.310 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b[32679]: 2024-09-14T03:50:48.960+0000 7fa952263640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T03:50:49.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b[32679]: 2024-09-14T03:50:48.960+0000 7fa952263640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-14T03:50:49.311 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:49 smithi160 podman[88234]: 2024-09-14 03:50:49.034483657 +0000 UTC m=+0.210322726 container died a30277b5b152463ff7ddd7c9ab8c3067a2a9f7d5bc83262baadf3ce513dd5b2d (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, RELEASE=quincy-eb16b97, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-14T03:50:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:49 smithi160 podman[88234]: 2024-09-14 03:50:49.724716407 +0000 UTC m=+0.900555483 container cleanup a30277b5b152463ff7ddd7c9ab8c3067a2a9f7d5bc83262baadf3ce513dd5b2d (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T03:50:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:49 smithi160 bash[88234]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b 2024-09-14T03:50:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:50 smithi160 podman[88249]: 2024-09-14 03:50:50.401861392 +0000 UTC m=+1.369869888 container remove a30277b5b152463ff7ddd7c9ab8c3067a2a9f7d5bc83262baadf3ce513dd5b2d (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-eb16b97, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T03:50:51.141 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:50 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-14T03:50:51.141 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:50 smithi160 systemd[1]: Stopped Ceph mon.b for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:50:51.141 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:50 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.b.service: Consumed 10.667s CPU time. 2024-09-14T03:50:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 systemd[1]: Starting Ceph mon.b for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:50:51.749 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 podman[88372]: 2024-09-14 03:50:51.312800354 +0000 UTC m=+0.028035010 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:50:51.750 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 podman[88372]: 2024-09-14 03:50:51.461736541 +0000 UTC m=+0.176971184 container create c470c18a013d7d2f41fd3a1d637d2a29b75acf750a93afb161e42f2dbdcc51f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, RELEASE=main-a278231, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:50:51.750 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 podman[88372]: 2024-09-14 03:50:51.748539472 +0000 UTC m=+0.463774125 container init c470c18a013d7d2f41fd3a1d637d2a29b75acf750a93afb161e42f2dbdcc51f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-a278231, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T03:50:51.794 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 podman[88372]: 2024-09-14 03:50:51.756060621 +0000 UTC m=+0.471295268 container start c470c18a013d7d2f41fd3a1d637d2a29b75acf750a93afb161e42f2dbdcc51f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-a278231, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-14T03:50:51.794 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: set uid:gid to 167:167 (ceph:ceph) 2024-09-14T03:50:51.794 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev), process ceph-mon, pid 7 2024-09-14T03:50:51.794 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: pidfile_write: ignore empty --pid-file 2024-09-14T03:50:51.794 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: load: jerasure load: lrc 2024-09-14T03:50:51.794 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: RocksDB version: 7.9.2 2024-09-14T03:50:51.794 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Git sha 0 2024-09-14T03:50:51.794 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Compile date 2024-09-12 20:09:22 2024-09-14T03:50:51.794 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: DB SUMMARY 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: DB Session ID: ZUJPCAB9KIXPVYAEXZGS 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: CURRENT file: CURRENT 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: IDENTITY file: IDENTITY 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1103 Bytes 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 6802976 ; 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.error_if_exists: 0 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.create_if_missing: 0 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.paranoid_checks: 1 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-14T03:50:51.795 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.env: 0x55c1c090a120 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.fs: PosixFileSystem 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.info_log: 0x55c1c2671700 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_file_opening_threads: 16 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.statistics: (nil) 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.use_fsync: 0 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_log_file_size: 0 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.keep_log_file_num: 1000 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.recycle_log_file_num: 0 2024-09-14T03:50:51.796 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.allow_fallocate: 1 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.allow_mmap_reads: 0 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.allow_mmap_writes: 0 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.use_direct_reads: 0 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.create_missing_column_families: 0 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.db_log_dir: 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.wal_dir: 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-14T03:50:51.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.advise_random_on_open: 1 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.db_write_buffer_size: 0 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.write_buffer_manager: 0x55c1c26741e0 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.rate_limiter: (nil) 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.wal_recovery_mode: 2 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.enable_thread_tracking: 0 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.enable_pipelined_write: 0 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.unordered_write: 0 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-14T03:50:51.798 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.row_cache: None 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.wal_filter: None 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.allow_ingest_behind: 0 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.two_write_queues: 0 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.manual_wal_flush: 0 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.wal_compression: 0 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.atomic_flush: 0 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-14T03:50:51.799 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.log_readahead_size: 0 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.best_efforts_recovery: 0 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.allow_data_in_errors: 0 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.db_host_id: __hostname__ 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_background_jobs: 2 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_background_compactions: -1 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_subcompactions: 1 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-14T03:50:51.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-14T03:50:51.801 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_total_wal_size: 0 2024-09-14T03:50:51.801 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-14T03:50:51.801 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-14T03:50:51.801 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-14T03:50:51.801 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-14T03:50:51.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:50:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:50:51] "GET /metrics HTTP/1.1" 200 36181 "" "Prometheus/2.51.0" 2024-09-14T03:50:52.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_open_files: -1 2024-09-14T03:50:52.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bytes_per_sync: 0 2024-09-14T03:50:52.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-14T03:50:52.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-14T03:50:52.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_readahead_size: 0 2024-09-14T03:50:52.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_background_flushes: -1 2024-09-14T03:50:52.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Compression algorithms supported: 2024-09-14T03:50:52.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: kZSTD supported: 0 2024-09-14T03:50:52.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: kXpressCompression supported: 0 2024-09-14T03:50:52.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: kBZip2Compression supported: 0 2024-09-14T03:50:52.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-14T03:50:52.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: kLZ4Compression supported: 1 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: kZlibCompression supported: 1 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: kLZ4HCCompression supported: 1 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: kSnappyCompression supported: 1 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.merge_operator: 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_filter: None 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_filter_factory: None 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.sst_partitioner_factory: None 2024-09-14T03:50:52.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c1c2671320) 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: cache_index_and_filter_blocks: 1 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: pin_top_level_index_and_filter: 1 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_type: 0 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: data_block_index_type: 0 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_shortening: 1 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: checksum: 4 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: no_block_cache: 0 2024-09-14T03:50:52.161 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache: 0x55c1c2665610 2024-09-14T03:50:52.162 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_name: BinnedLRUCache 2024-09-14T03:50:52.162 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_options: 2024-09-14T03:50:52.162 INFO:journalctl@ceph.mon.b.smithi160.stdout: capacity : 536870912 2024-09-14T03:50:52.162 INFO:journalctl@ceph.mon.b.smithi160.stdout: num_shard_bits : 4 2024-09-14T03:50:52.162 INFO:journalctl@ceph.mon.b.smithi160.stdout: strict_capacity_limit : 0 2024-09-14T03:50:52.162 INFO:journalctl@ceph.mon.b.smithi160.stdout: high_pri_pool_ratio: 0.000 2024-09-14T03:50:52.162 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_cache_compressed: (nil) 2024-09-14T03:50:52.162 INFO:journalctl@ceph.mon.b.smithi160.stdout: persistent_cache: (nil) 2024-09-14T03:50:52.162 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_size: 4096 2024-09-14T03:50:52.162 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_size_deviation: 10 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_restart_interval: 16 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: index_block_restart_interval: 1 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: metadata_block_size: 4096 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: partition_filters: 0 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: use_delta_encoding: 1 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: filter_policy: bloomfilter 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: whole_key_filtering: 1 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: verify_compression: 0 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: read_amp_bytes_per_bit: 0 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: format_version: 5 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: enable_index_compression: 1 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: block_align: 0 2024-09-14T03:50:52.163 INFO:journalctl@ceph.mon.b.smithi160.stdout: max_auto_readahead_size: 262144 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout: prepopulate_block_cache: 0 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout: initial_auto_readahead_size: 8192 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout: num_file_reads_for_auto_readahead: 2 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.write_buffer_size: 33554432 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_write_buffer_number: 2 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compression: NoCompression 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bottommost_compression: Disabled 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.prefix_extractor: nullptr 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.num_levels: 7 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-14T03:50:52.164 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-14T03:50:52.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-14T03:50:52.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-14T03:50:52.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-14T03:50:52.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-14T03:50:52.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:50:52.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-14T03:50:52.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-14T03:50:52.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-14T03:50:52.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-14T03:50:52.165 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-14T03:50:52.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compression_opts.level: 32767 2024-09-14T03:50:52.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compression_opts.strategy: 0 2024-09-14T03:50:52.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-14T03:50:52.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-14T03:50:52.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-14T03:50:52.166 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-14T03:50:52.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compression_opts.enabled: false 2024-09-14T03:50:52.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-14T03:50:52.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-14T03:50:52.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-14T03:50:52.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-14T03:50:52.167 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.target_file_size_base: 67108864 2024-09-14T03:50:52.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-14T03:50:52.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-14T03:50:52.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-14T03:50:52.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-14T03:50:52.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-14T03:50:52.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-14T03:50:52.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-14T03:50:52.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-14T03:50:52.168 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.arena_block_size: 1048576 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.disable_auto_compactions: 0 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-14T03:50:52.169 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.inplace_update_support: 0 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.bloom_locality: 0 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.max_successive_merges: 0 2024-09-14T03:50:52.170 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.paranoid_file_checks: 0 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.force_consistency_checks: 1 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.report_bg_io_stats: 0 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.ttl: 2592000 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.enable_blob_files: false 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.min_blob_size: 0 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.blob_file_size: 268435456 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-14T03:50:52.171 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-14T03:50:52.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-14T03:50:52.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.blob_file_starting_level: 0 2024-09-14T03:50:52.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-14T03:50:52.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-14T03:50:52.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-14T03:50:52.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12880, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-14T03:50:52.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-14T03:50:52.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 79076f5f-dc04-4aa6-ad78-fefb2da7b4ab 2024-09-14T03:50:52.172 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285851794283, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-14T03:50:52.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-14T03:50:52.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285851816274, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 4235750, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12885, "largest_seqno": 15619, "table_properties": {"data_size": 4224793, "index_size": 6838, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3077, "raw_key_size": 26365, "raw_average_key_size": 21, "raw_value_size": 4200699, "raw_average_value_size": 3420, "num_data_blocks": 308, "num_entries": 1228, "num_filter_entries": 1228, "num_deletions": 253, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726285851, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "79076f5f-dc04-4aa6-ad78-fefb2da7b4ab", "db_session_id": "ZUJPCAB9KIXPVYAEXZGS", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-14T03:50:52.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726285851816570, "job": 1, "event": "recovery_finished"} 2024-09-14T03:50:52.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-14T03:50:52.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-14T03:50:52.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-14T03:50:52.173 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c1c2694e00 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: DB pointer 0x55c1c2796000 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** DB Stats ** 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-14T03:50:52.174 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** Compaction Stats [default] ** 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: L0 1/0 4.04 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 452.4 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: L6 1/0 7.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: Sum 2/0 11.15 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 452.4 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 452.4 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** Compaction Stats [default] ** 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-14T03:50:52.175 INFO:journalctl@ceph.mon.b.smithi160.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 452.4 0.01 0.00 1 0.009 0 0 0.0 0.0 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: Cumulative compaction: 0.00 GB write, 139.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: Interval compaction: 0.00 GB write, 139.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: Block cache BinnedLRUCache@0x55c1c2665610#7 capacity: 512.00 MB usage: 11.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-09-14T03:50:52.176 INFO:journalctl@ceph.mon.b.smithi160.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,3.28 KB,0.000625849%) IndexBlock(1,8.08 KB,0.00154078%) Misc(1,0.00 KB,0%) 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: starting mon.b rank 1 at public addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] at bind addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: mon.b@-1(???) e3 preinit fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: mon.b@-1(???).mds e9 new map 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: mon.b@-1(???).mds e9 print_map 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout: e9 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-14T03:50:52.177 INFO:journalctl@ceph.mon.b.smithi160.stdout: legacy client fscid: 1 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: Filesystem 'foo' (1) 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: fs_name foo 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: epoch 8 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: created 2024-09-14T03:43:11.271900+0000 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: modified 2024-09-14T03:43:23.592848+0000 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: tableserver 0 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: root 0 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: session_timeout 60 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: session_autoclose 300 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: max_file_size 1099511627776 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: max_xattr_size 65536 2024-09-14T03:50:52.178 INFO:journalctl@ceph.mon.b.smithi160.stdout: required_client_features {} 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: last_failure 0 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: last_failure_osd_epoch 0 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: max_mds 1 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: in 0 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: up {0=14658} 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: failed 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: damaged 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: stopped 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: data_pools [4] 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: metadata_pool 3 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: inline_data disabled 2024-09-14T03:50:52.179 INFO:journalctl@ceph.mon.b.smithi160.stdout: balancer 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: bal_rank_mask -1 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: standby_count_wanted 1 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: qdb_cluster leader: 0 members: 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: [mds.foo.smithi114.ebmrat{0:14658} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.114:6834/4270322853,v1:172.21.15.114:6835/4270322853] compat {c=[1],r=[1],i=[7ff]}] 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: Standby daemons: 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout: [mds.foo.smithi160.ywyjae{-1:24481} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.160:6832/2654584478,v1:172.21.15.160:6833/2654584478] compat {c=[1],r=[1],i=[7ff]}] 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: mon.b@-1(???).osd e76 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: mon.b@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: mon.b@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-09-14T03:50:52.180 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: mon.b@-1(???).osd e76 crush map has features 432629239337189376, adjusting msgr requires 2024-09-14T03:50:52.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 ceph-mon[88407]: mon.b@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-09-14T03:50:52.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 bash[88372]: c470c18a013d7d2f41fd3a1d637d2a29b75acf750a93afb161e42f2dbdcc51f6 2024-09-14T03:50:52.181 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:51 smithi160 systemd[1]: Started Ceph mon.b for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:50:54.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:54.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:56.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:56.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:57.120 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:50:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:50:56.836+0000 7fd419828640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-14T03:50:57.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:50:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:50:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:50:57.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-14T03:50:57.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: mon.b calling monitor election 2024-09-14T03:50:57.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: mon.a calling monitor election 2024-09-14T03:50:57.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-14T03:50:57.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: pgmap v63: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: pgmap v64: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: pgmap v65: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: monmap epoch 4 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: last_changed 2024-09-14T03:50:52.081398+0000 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: min_mon_release 19 (squid) 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: election_strategy: 1 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:50:57.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: mon.c (rank 2) addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] is down (out of quorum) 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:57.412 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:57.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-14T03:50:57.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: mon.b calling monitor election 2024-09-14T03:50:57.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: mon.a calling monitor election 2024-09-14T03:50:57.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-14T03:50:57.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-14T03:50:57.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: pgmap v63: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: pgmap v64: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: pgmap v65: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: monmap epoch 4 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: last_changed 2024-09-14T03:50:52.081398+0000 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: min_mon_release 19 (squid) 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: election_strategy: 1 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:50:57.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:50:57.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:50:57.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:50:57.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-14T03:50:57.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-14T03:50:57.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-14T03:50:57.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-14T03:50:57.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: mon.c (rank 2) addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] is down (out of quorum) 2024-09-14T03:50:57.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:57.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:57.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: mon.b calling monitor election 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: mon.a calling monitor election 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: pgmap v63: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: pgmap v64: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: pgmap v65: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: monmap epoch 4 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: last_changed 2024-09-14T03:50:52.081398+0000 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: min_mon_release 19 (squid) 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: election_strategy: 1 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: mon.c (rank 2) addr [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] is down (out of quorum) 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:59.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: mon.c calling monitor election 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: mon.b calling monitor election 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: mon.a calling monitor election 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: monmap epoch 4 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: last_changed 2024-09-14T03:50:52.081398+0000 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: min_mon_release 19 (squid) 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: election_strategy: 1 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:50:59.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:50:59.625 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:50:59.625 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:50:59.625 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:50:59.625 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-14T03:50:59.625 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-14T03:50:59.625 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T03:50:59.625 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[106969]: overall HEALTH_OK 2024-09-14T03:50:59.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: mon.c calling monitor election 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: mon.b calling monitor election 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: mon.a calling monitor election 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: monmap epoch 4 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: last_changed 2024-09-14T03:50:52.081398+0000 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: min_mon_release 19 (squid) 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: election_strategy: 1 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:50:59.626 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:50:59.627 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-14T03:50:59.627 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-14T03:50:59.627 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T03:50:59.627 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:50:59 smithi114 ceph-mon[103029]: overall HEALTH_OK 2024-09-14T03:50:59.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: mon.c calling monitor election 2024-09-14T03:50:59.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: mon.b calling monitor election 2024-09-14T03:50:59.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: mon.a calling monitor election 2024-09-14T03:50:59.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-14T03:50:59.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: monmap epoch 4 2024-09-14T03:50:59.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T03:50:59.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: last_changed 2024-09-14T03:50:52.081398+0000 2024-09-14T03:50:59.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: created 2024-09-14T03:36:27.521144+0000 2024-09-14T03:50:59.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: min_mon_release 19 (squid) 2024-09-14T03:50:59.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: election_strategy: 1 2024-09-14T03:50:59.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-14T03:50:59.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: 1: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.b 2024-09-14T03:50:59.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: 2: [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0] mon.c 2024-09-14T03:50:59.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: fsmap foo:1 {0=foo.smithi114.ebmrat=up:active} 1 up:standby 2024-09-14T03:50:59.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: osdmap e76: 8 total, 8 up, 8 in 2024-09-14T03:50:59.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: mgrmap e40: y(active, since 2m), standbys: x 2024-09-14T03:50:59.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-14T03:50:59.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T03:50:59.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:50:59 smithi160 ceph-mon[88407]: overall HEALTH_OK 2024-09-14T03:51:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:01.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:01 smithi160 ceph-mon[88407]: pgmap v67: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-14T03:51:01.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:01 smithi114 ceph-mon[106969]: pgmap v67: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-14T03:51:01.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:51:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:51:01] "GET /metrics HTTP/1.1" 200 36171 "" "Prometheus/2.51.0" 2024-09-14T03:51:01.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:01 smithi114 ceph-mon[103029]: pgmap v67: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-14T03:51:03.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:03 smithi114 ceph-mon[106969]: pgmap v68: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 761 op/s 2024-09-14T03:51:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:03 smithi114 ceph-mon[103029]: pgmap v68: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 761 op/s 2024-09-14T03:51:03.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.903 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:03 smithi160 ceph-mon[88407]: pgmap v68: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 761 op/s 2024-09-14T03:51:03.903 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.903 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.903 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:03.903 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:04.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:04 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:05.741 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:05 smithi160 ceph-mon[88407]: pgmap v69: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-14T03:51:05.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:05 smithi114 ceph-mon[106969]: pgmap v69: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-14T03:51:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:05 smithi114 ceph-mon[103029]: pgmap v69: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-14T03:51:06.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:06.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:07.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:06 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:07.863 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:07 smithi160 ceph-mon[88407]: pgmap v70: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 742 op/s 2024-09-14T03:51:07.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:07 smithi114 ceph-mon[106969]: pgmap v70: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 742 op/s 2024-09-14T03:51:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:07 smithi114 ceph-mon[103029]: pgmap v70: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 742 op/s 2024-09-14T03:51:08.857 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:08.857 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:08.857 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:08.858 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:08.858 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:51:08.858 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:08.858 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:51:08.858 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:51:08.858 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:08.858 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:08.858 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:08.858 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:08.858 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:08.859 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:51:08.859 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:08.859 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:51:08.859 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:51:08.859 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:51:08.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:08.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:51:08.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:51:08.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:09 smithi114 ceph-mon[106969]: pgmap v71: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 742 op/s 2024-09-14T03:51:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:09 smithi114 ceph-mon[106969]: Reconfiguring mon.a (monmap changed)... 2024-09-14T03:51:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:09 smithi114 ceph-mon[106969]: Reconfiguring daemon mon.a on smithi114 2024-09-14T03:51:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:09 smithi114 ceph-mon[103029]: pgmap v71: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 742 op/s 2024-09-14T03:51:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:09 smithi114 ceph-mon[103029]: Reconfiguring mon.a (monmap changed)... 2024-09-14T03:51:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:09 smithi114 ceph-mon[103029]: Reconfiguring daemon mon.a on smithi114 2024-09-14T03:51:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:09 smithi160 ceph-mon[88407]: pgmap v71: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 742 op/s 2024-09-14T03:51:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:09 smithi160 ceph-mon[88407]: Reconfiguring mon.a (monmap changed)... 2024-09-14T03:51:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:09 smithi160 ceph-mon[88407]: Reconfiguring daemon mon.a on smithi114 2024-09-14T03:51:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:11.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:11.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:11.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-14T03:51:11.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-14T03:51:11.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:11.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:11.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:11.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-14T03:51:11.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-14T03:51:11.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:11 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:11 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:11 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-14T03:51:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:11 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-14T03:51:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:11 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:11.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:51:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:51:11] "GET /metrics HTTP/1.1" 200 36191 "" "Prometheus/2.51.0" 2024-09-14T03:51:12.256 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:12 smithi114 ceph-mon[106969]: Reconfiguring mgr.y (monmap changed)... 2024-09-14T03:51:12.257 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:12 smithi114 ceph-mon[106969]: Reconfiguring daemon mgr.y on smithi114 2024-09-14T03:51:12.257 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:12 smithi114 ceph-mon[106969]: pgmap v72: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-14T03:51:12.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:12 smithi114 ceph-mon[103029]: Reconfiguring mgr.y (monmap changed)... 2024-09-14T03:51:12.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:12 smithi114 ceph-mon[103029]: Reconfiguring daemon mgr.y on smithi114 2024-09-14T03:51:12.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:12 smithi114 ceph-mon[103029]: pgmap v72: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-14T03:51:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:12 smithi160 ceph-mon[88407]: Reconfiguring mgr.y (monmap changed)... 2024-09-14T03:51:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:12 smithi160 ceph-mon[88407]: Reconfiguring daemon mgr.y on smithi114 2024-09-14T03:51:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:12 smithi160 ceph-mon[88407]: pgmap v72: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-14T03:51:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[106969]: pgmap v73: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-09-14T03:51:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[106969]: Reconfiguring mon.c (monmap changed)... 2024-09-14T03:51:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:51:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:51:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[106969]: Reconfiguring daemon mon.c on smithi114 2024-09-14T03:51:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[103029]: pgmap v73: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-09-14T03:51:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[103029]: Reconfiguring mon.c (monmap changed)... 2024-09-14T03:51:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:51:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:51:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:13 smithi114 ceph-mon[103029]: Reconfiguring daemon mon.c on smithi114 2024-09-14T03:51:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:13 smithi160 ceph-mon[88407]: pgmap v73: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-09-14T03:51:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:13 smithi160 ceph-mon[88407]: Reconfiguring mon.c (monmap changed)... 2024-09-14T03:51:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:51:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:51:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:13.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:13 smithi160 ceph-mon[88407]: Reconfiguring daemon mon.c on smithi114 2024-09-14T03:51:14.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:14 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:15.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:15.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:15.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[103029]: Reconfiguring osd.0 (monmap changed)... 2024-09-14T03:51:15.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-14T03:51:15.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:15.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[103029]: Reconfiguring daemon osd.0 on smithi114 2024-09-14T03:51:15.339 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[103029]: pgmap v74: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:15.339 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:15.339 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:15.339 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[106969]: Reconfiguring osd.0 (monmap changed)... 2024-09-14T03:51:15.339 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-14T03:51:15.339 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:15.339 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[106969]: Reconfiguring daemon osd.0 on smithi114 2024-09-14T03:51:15.339 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:15 smithi114 ceph-mon[106969]: pgmap v74: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:15 smithi160 ceph-mon[88407]: Reconfiguring osd.0 (monmap changed)... 2024-09-14T03:51:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-14T03:51:15.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:15.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:15 smithi160 ceph-mon[88407]: Reconfiguring daemon osd.0 on smithi114 2024-09-14T03:51:15.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:15 smithi160 ceph-mon[88407]: pgmap v74: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:16.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:16.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:15 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:15.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:17.290 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:16 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[103029]: Reconfiguring osd.1 (monmap changed)... 2024-09-14T03:51:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-14T03:51:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[103029]: Reconfiguring daemon osd.1 on smithi114 2024-09-14T03:51:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[103029]: pgmap v75: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-14T03:51:17.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:17.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:17.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[106969]: Reconfiguring osd.1 (monmap changed)... 2024-09-14T03:51:17.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-14T03:51:17.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:17.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[106969]: Reconfiguring daemon osd.1 on smithi114 2024-09-14T03:51:17.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:17 smithi114 ceph-mon[106969]: pgmap v75: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-14T03:51:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:17 smithi160 ceph-mon[88407]: Reconfiguring osd.1 (monmap changed)... 2024-09-14T03:51:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-14T03:51:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:17 smithi160 ceph-mon[88407]: Reconfiguring daemon osd.1 on smithi114 2024-09-14T03:51:17.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:17 smithi160 ceph-mon[88407]: pgmap v75: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-14T03:51:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[103029]: pgmap v76: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-14T03:51:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[103029]: Reconfiguring osd.2 (monmap changed)... 2024-09-14T03:51:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-14T03:51:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[103029]: Reconfiguring daemon osd.2 on smithi114 2024-09-14T03:51:20.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[106969]: pgmap v76: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-14T03:51:20.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:20.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:20.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[106969]: Reconfiguring osd.2 (monmap changed)... 2024-09-14T03:51:20.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-14T03:51:20.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:20.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:19 smithi114 ceph-mon[106969]: Reconfiguring daemon osd.2 on smithi114 2024-09-14T03:51:20.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:19 smithi160 ceph-mon[88407]: pgmap v76: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-14T03:51:20.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:19 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:20.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:19 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:20.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:19 smithi160 ceph-mon[88407]: Reconfiguring osd.2 (monmap changed)... 2024-09-14T03:51:20.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:19 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-14T03:51:20.215 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:19 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:20.215 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:19 smithi160 ceph-mon[88407]: Reconfiguring daemon osd.2 on smithi114 2024-09-14T03:51:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:21.793 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:51:21] "GET /metrics HTTP/1.1" 200 36191 "" "Prometheus/2.51.0" 2024-09-14T03:51:22.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-mon[106969]: pgmap v77: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:22.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:22.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:22.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-14T03:51:22.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:22.095 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-mon[103029]: pgmap v77: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:22.095 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:22.095 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:22.095 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-14T03:51:22.095 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:21 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:21 smithi160 ceph-mon[88407]: pgmap v77: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:21 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:21 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:22.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:21 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-14T03:51:22.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:21 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:23.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:22 smithi114 ceph-mon[103029]: Reconfiguring osd.3 (monmap changed)... 2024-09-14T03:51:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:22 smithi114 ceph-mon[103029]: Reconfiguring daemon osd.3 on smithi114 2024-09-14T03:51:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:22 smithi114 ceph-mon[106969]: Reconfiguring osd.3 (monmap changed)... 2024-09-14T03:51:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:22 smithi114 ceph-mon[106969]: Reconfiguring daemon osd.3 on smithi114 2024-09-14T03:51:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:22 smithi160 ceph-mon[88407]: Reconfiguring osd.3 (monmap changed)... 2024-09-14T03:51:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:22 smithi160 ceph-mon[88407]: Reconfiguring daemon osd.3 on smithi114 2024-09-14T03:51:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:24.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:23 smithi114 ceph-mon[106969]: pgmap v78: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-14T03:51:24.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:23 smithi114 ceph-mon[103029]: pgmap v78: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-14T03:51:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:23 smithi160 ceph-mon[88407]: pgmap v78: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-14T03:51:24.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:24 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[103029]: Reconfiguring mds.foo.smithi114.ebmrat (monmap changed)... 2024-09-14T03:51:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-14T03:51:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[103029]: Reconfiguring daemon mds.foo.smithi114.ebmrat on smithi114 2024-09-14T03:51:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[103029]: pgmap v79: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:25.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:25.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:25.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[106969]: Reconfiguring mds.foo.smithi114.ebmrat (monmap changed)... 2024-09-14T03:51:25.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-14T03:51:25.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:25.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[106969]: Reconfiguring daemon mds.foo.smithi114.ebmrat on smithi114 2024-09-14T03:51:25.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:25 smithi114 ceph-mon[106969]: pgmap v79: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:25 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:25 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:25 smithi160 ceph-mon[88407]: Reconfiguring mds.foo.smithi114.ebmrat (monmap changed)... 2024-09-14T03:51:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:25 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi114.ebmrat", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-14T03:51:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:25 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:25 smithi160 ceph-mon[88407]: Reconfiguring daemon mds.foo.smithi114.ebmrat on smithi114 2024-09-14T03:51:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:25 smithi160 ceph-mon[88407]: pgmap v79: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:26.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:25 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:26.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:25 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:27.083 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:26.950Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:27.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:27 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:27.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:27 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:27.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:27 smithi160 ceph-mon[88407]: Reconfiguring mon.b (monmap changed)... 2024-09-14T03:51:27.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:27 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:51:27.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:27 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:51:27.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:27 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:27.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:27 smithi160 ceph-mon[88407]: Reconfiguring daemon mon.b on smithi160 2024-09-14T03:51:27.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:27 smithi160 ceph-mon[88407]: pgmap v80: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s 2024-09-14T03:51:27.521 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:27.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:27.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[106969]: Reconfiguring mon.b (monmap changed)... 2024-09-14T03:51:27.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:51:27.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:51:27.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:27.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[106969]: Reconfiguring daemon mon.b on smithi160 2024-09-14T03:51:27.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[106969]: pgmap v80: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s 2024-09-14T03:51:27.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:27.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:27.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[103029]: Reconfiguring mon.b (monmap changed)... 2024-09-14T03:51:27.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-14T03:51:27.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-14T03:51:27.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:27.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[103029]: Reconfiguring daemon mon.b on smithi160 2024-09-14T03:51:27.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:27 smithi114 ceph-mon[103029]: pgmap v80: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s 2024-09-14T03:51:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:29 smithi160 ceph-mon[88407]: pgmap v81: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-09-14T03:51:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:29 smithi160 ceph-mon[88407]: Reconfiguring mgr.x (monmap changed)... 2024-09-14T03:51:29.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-14T03:51:29.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-14T03:51:29.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:29.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:29 smithi160 ceph-mon[88407]: Reconfiguring daemon mgr.x on smithi160 2024-09-14T03:51:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[106969]: pgmap v81: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-09-14T03:51:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[106969]: Reconfiguring mgr.x (monmap changed)... 2024-09-14T03:51:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-14T03:51:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-14T03:51:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[106969]: Reconfiguring daemon mgr.x on smithi160 2024-09-14T03:51:30.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[103029]: pgmap v81: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-09-14T03:51:30.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:30.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:30.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[103029]: Reconfiguring mgr.x (monmap changed)... 2024-09-14T03:51:30.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-14T03:51:30.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-14T03:51:30.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:30.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:29 smithi114 ceph-mon[103029]: Reconfiguring daemon mgr.x on smithi160 2024-09-14T03:51:30.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:31.589 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:31 smithi160 ceph-mon[88407]: pgmap v82: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:31.589 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:31.589 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:31.589 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:31 smithi160 ceph-mon[88407]: Reconfiguring osd.4 (monmap changed)... 2024-09-14T03:51:31.589 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-14T03:51:31.589 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:31.589 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:31 smithi160 ceph-mon[88407]: Reconfiguring daemon osd.4 on smithi160 2024-09-14T03:51:31.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[106969]: pgmap v82: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:31.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:31.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:31.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[106969]: Reconfiguring osd.4 (monmap changed)... 2024-09-14T03:51:31.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-14T03:51:31.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:31.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[106969]: Reconfiguring daemon osd.4 on smithi160 2024-09-14T03:51:31.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:51:31] "GET /metrics HTTP/1.1" 200 36186 "" "Prometheus/2.51.0" 2024-09-14T03:51:31.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[103029]: pgmap v82: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:31.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:31.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:31.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[103029]: Reconfiguring osd.4 (monmap changed)... 2024-09-14T03:51:31.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-14T03:51:31.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:31.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:31 smithi114 ceph-mon[103029]: Reconfiguring daemon osd.4 on smithi160 2024-09-14T03:51:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[106969]: Reconfiguring osd.5 (monmap changed)... 2024-09-14T03:51:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-14T03:51:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[106969]: Reconfiguring daemon osd.5 on smithi160 2024-09-14T03:51:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[106969]: pgmap v83: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 724 op/s 2024-09-14T03:51:33.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:33.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:33.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[103029]: Reconfiguring osd.5 (monmap changed)... 2024-09-14T03:51:33.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-14T03:51:33.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:33.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[103029]: Reconfiguring daemon osd.5 on smithi160 2024-09-14T03:51:33.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:33 smithi114 ceph-mon[103029]: pgmap v83: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 724 op/s 2024-09-14T03:51:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:33 smithi160 ceph-mon[88407]: Reconfiguring osd.5 (monmap changed)... 2024-09-14T03:51:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-14T03:51:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:33.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:33 smithi160 ceph-mon[88407]: Reconfiguring daemon osd.5 on smithi160 2024-09-14T03:51:33.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:33 smithi160 ceph-mon[88407]: pgmap v83: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 724 op/s 2024-09-14T03:51:34.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:34 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:35.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:35.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:35.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:35 smithi160 ceph-mon[88407]: pgmap v84: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:35.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:35 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:35.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:35 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:35.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:35 smithi160 ceph-mon[88407]: Reconfiguring osd.6 (monmap changed)... 2024-09-14T03:51:35.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:35 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-14T03:51:35.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:35 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:35.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:35 smithi160 ceph-mon[88407]: Reconfiguring daemon osd.6 on smithi160 2024-09-14T03:51:36.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[106969]: pgmap v84: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:36.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:36.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:36.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[106969]: Reconfiguring osd.6 (monmap changed)... 2024-09-14T03:51:36.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-14T03:51:36.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:36.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[106969]: Reconfiguring daemon osd.6 on smithi160 2024-09-14T03:51:36.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[103029]: pgmap v84: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:36.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:36.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:36.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[103029]: Reconfiguring osd.6 (monmap changed)... 2024-09-14T03:51:36.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-14T03:51:36.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:36.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:35 smithi114 ceph-mon[103029]: Reconfiguring daemon osd.6 on smithi160 2024-09-14T03:51:37.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:36.954Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:37 smithi160 ceph-mon[88407]: pgmap v85: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-09-14T03:51:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:37 smithi160 ceph-mon[88407]: Reconfiguring osd.7 (monmap changed)... 2024-09-14T03:51:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-14T03:51:38.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:38.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:37 smithi160 ceph-mon[88407]: Reconfiguring daemon osd.7 on smithi160 2024-09-14T03:51:38.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[103029]: pgmap v85: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-09-14T03:51:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[103029]: Reconfiguring osd.7 (monmap changed)... 2024-09-14T03:51:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-14T03:51:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[103029]: Reconfiguring daemon osd.7 on smithi160 2024-09-14T03:51:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:38.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[106969]: pgmap v85: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-09-14T03:51:38.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:38.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:38.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[106969]: Reconfiguring osd.7 (monmap changed)... 2024-09-14T03:51:38.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-14T03:51:38.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:38.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[106969]: Reconfiguring daemon osd.7 on smithi160 2024-09-14T03:51:38.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:39 smithi160 ceph-mon[88407]: pgmap v86: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-09-14T03:51:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:39 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:39 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:39 smithi160 ceph-mon[88407]: Reconfiguring mds.foo.smithi160.ywyjae (monmap changed)... 2024-09-14T03:51:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:39 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-14T03:51:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:39 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:40.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:39 smithi160 ceph-mon[88407]: Reconfiguring daemon mds.foo.smithi160.ywyjae on smithi160 2024-09-14T03:51:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[103029]: pgmap v86: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-09-14T03:51:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[103029]: Reconfiguring mds.foo.smithi160.ywyjae (monmap changed)... 2024-09-14T03:51:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-14T03:51:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[103029]: Reconfiguring daemon mds.foo.smithi160.ywyjae on smithi160 2024-09-14T03:51:40.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[106969]: pgmap v86: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-09-14T03:51:40.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:40.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:40.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[106969]: Reconfiguring mds.foo.smithi160.ywyjae (monmap changed)... 2024-09-14T03:51:40.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi160.ywyjae", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-14T03:51:40.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:40.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:39 smithi114 ceph-mon[106969]: Reconfiguring daemon mds.foo.smithi160.ywyjae on smithi160 2024-09-14T03:51:40.517 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:41.735 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:41 smithi114 ceph-mon[103029]: pgmap v87: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-14T03:51:41.735 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:41 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:41.735 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:41 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:41.735 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:41 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:51:41.736 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:51:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:51:41] "GET /metrics HTTP/1.1" 200 36190 "" "Prometheus/2.51.0" 2024-09-14T03:51:41.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:41 smithi114 ceph-mon[106969]: pgmap v87: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-14T03:51:41.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:41 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:41.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:41 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:41.736 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:41 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:51:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:41 smithi160 ceph-mon[88407]: pgmap v87: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-14T03:51:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:41 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:41 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:41 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:51:43.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:43 smithi114 ceph-mon[106969]: pgmap v88: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T03:51:43.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:43 smithi114 ceph-mon[103029]: pgmap v88: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T03:51:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:43 smithi160 ceph-mon[88407]: pgmap v88: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T03:51:44.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:45 smithi160 ceph-mon[88407]: pgmap v89: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:45.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:45.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:46.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:45 smithi114 ceph-mon[103029]: pgmap v89: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:46.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:45 smithi114 ceph-mon[106969]: pgmap v89: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:47.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:51:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:51:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:47.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:51:47.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:48.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:47 smithi114 ceph-mon[103029]: pgmap v90: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:51:48.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:47 smithi114 ceph-mon[106969]: pgmap v90: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:51:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:47 smithi160 ceph-mon[88407]: pgmap v90: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:51:50.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:49 smithi114 ceph-mon[103029]: pgmap v91: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:51:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:49 smithi114 ceph-mon[106969]: pgmap v91: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:51:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:49 smithi160 ceph-mon[88407]: pgmap v91: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 735 op/s 2024-09-14T03:51:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:51.717 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:51:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:51:51] "GET /metrics HTTP/1.1" 200 36190 "" "Prometheus/2.51.0" 2024-09-14T03:51:51.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch upgrade resume' 2024-09-14T03:51:52.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:51 smithi114 ceph-mon[103029]: pgmap v92: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:52.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:51 smithi114 ceph-mon[106969]: pgmap v92: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:52.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:51 smithi160 ceph-mon[88407]: pgmap v92: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-14T03:51:52.241 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:51:53.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:53.072 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:52 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:52 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:51:53.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:53 smithi114 ceph-mon[103029]: pgmap v93: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 723 op/s 2024-09-14T03:51:53.992 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:53 smithi114 ceph-mon[106969]: pgmap v93: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 723 op/s 2024-09-14T03:51:54.147 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:53 smithi160 ceph-mon[88407]: pgmap v93: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 723 op/s 2024-09-14T03:51:54.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:54.900 INFO:teuthology.orchestra.run.smithi114.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:51:55.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'echo wait for all mons to upgrade' 2024-09-14T03:51:56.000 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: pgmap v94: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:56.000 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='client.44111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:51:56.000 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:51:56.000 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.000 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: Upgrade: Setting container_image for all mon 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-14T03:51:56.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-14T03:51:56.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-14T03:51:56.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:51:56.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: Upgrade: Setting container_image for all crash 2024-09-14T03:51:56.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-14T03:51:56.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-14T03:51:56.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[106969]: Upgrade: osd.0 is safe to restart 2024-09-14T03:51:56.002 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: pgmap v94: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:56.002 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='client.44111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: Upgrade: Setting container_image for all mon 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-14T03:51:56.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-14T03:51:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-14T03:51:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-14T03:51:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-14T03:51:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-14T03:51:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:51:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: Upgrade: Setting container_image for all crash 2024-09-14T03:51:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-14T03:51:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-14T03:51:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:55 smithi114 ceph-mon[103029]: Upgrade: osd.0 is safe to restart 2024-09-14T03:51:56.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:56.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: pgmap v94: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-14T03:51:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='client.44111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: Upgrade: Setting container_image for all mon 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-14T03:51:56.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-14T03:51:56.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-14T03:51:56.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-14T03:51:56.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-14T03:51:56.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-14T03:51:56.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:51:56.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: Upgrade: Setting container_image for all crash 2024-09-14T03:51:56.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:56.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-14T03:51:56.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-14T03:51:56.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:55 smithi160 ceph-mon[88407]: Upgrade: osd.0 is safe to restart 2024-09-14T03:51:56.781 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:51:57.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:51:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:51:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:51:58.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:57 smithi114 ceph-mon[103029]: pgmap v95: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-14T03:51:58.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:58.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-14T03:51:58.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:58.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:57 smithi114 ceph-mon[106969]: pgmap v95: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-14T03:51:58.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:57 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:58.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:57 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-14T03:51:58.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:57 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:57 smithi160 ceph-mon[88407]: pgmap v95: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-14T03:51:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:51:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-14T03:51:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:51:59.145 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:58 smithi114 ceph-mon[106969]: Upgrade: Updating osd.0 2024-09-14T03:51:59.146 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:58 smithi114 ceph-mon[106969]: Deploying daemon osd.0 on smithi114 2024-09-14T03:51:59.146 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:58 smithi114 ceph-mon[103029]: Upgrade: Updating osd.0 2024-09-14T03:51:59.146 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:58 smithi114 ceph-mon[103029]: Deploying daemon osd.0 on smithi114 2024-09-14T03:51:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:58 smithi160 ceph-mon[88407]: Upgrade: Updating osd.0 2024-09-14T03:51:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:58 smithi160 ceph-mon[88407]: Deploying daemon osd.0 on smithi114 2024-09-14T03:51:59.502 INFO:teuthology.orchestra.run.smithi114.stdout:wait for all mons to upgrade 2024-09-14T03:52:00.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:51:59 smithi114 ceph-mon[106969]: pgmap v96: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-14T03:52:00.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:51:59 smithi114 ceph-mon[103029]: pgmap v96: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-14T03:52:00.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:51:59 smithi160 ceph-mon[88407]: pgmap v96: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-14T03:52:00.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'while ! ceph mon versions | grep $sha1 | grep '"'"': 3'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-14T03:52:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:52:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:52:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:52:01.105 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:52:01.495 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:52:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:52:01] "GET /metrics HTTP/1.1" 200 36186 "" "Prometheus/2.51.0" 2024-09-14T03:52:02.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:01 smithi114 ceph-mon[106969]: pgmap v97: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T03:52:02.002 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:01 smithi114 ceph-mon[103029]: pgmap v97: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T03:52:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:01 smithi160 ceph-mon[88407]: pgmap v97: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T03:52:04.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:03 smithi114 ceph-mon[103029]: pgmap v98: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-09-14T03:52:04.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:03 smithi114 ceph-mon[106969]: pgmap v98: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-09-14T03:52:04.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:52:04 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:52:04.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)\", cluster_addr=\"172.21.15.114\", device_class=\"ssd\", hostname=\"smithi114\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.114\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:52:04.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:03 smithi160 ceph-mon[88407]: pgmap v98: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 721 op/s 2024-09-14T03:52:04.416 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:04 smithi114 systemd[1]: Stopping Ceph osd.0 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:52:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev)": 3 2024-09-14T03:52:05.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:04 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3998674358' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:52:05.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:04 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3998674358' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:52:05.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:05 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[45233]: 2024-09-14T03:52:05.123+0000 7f7c08f93640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T03:52:05.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:05 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[45233]: 2024-09-14T03:52:05.123+0000 7f7c08f93640 -1 osd.0 76 *** Got signal Terminated *** 2024-09-14T03:52:05.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:05 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[45233]: 2024-09-14T03:52:05.123+0000 7f7c08f93640 -1 osd.0 76 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T03:52:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:04 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3998674358' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-14T03:52:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:05 smithi114 ceph-mon[103029]: pgmap v99: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-14T03:52:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:05 smithi114 ceph-mon[103029]: osd.0 marked itself down and dead 2024-09-14T03:52:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:05 smithi114 ceph-mon[106969]: pgmap v99: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-14T03:52:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:05 smithi114 ceph-mon[106969]: osd.0 marked itself down and dead 2024-09-14T03:52:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:05 smithi160 ceph-mon[88407]: pgmap v99: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-14T03:52:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:05 smithi160 ceph-mon[88407]: osd.0 marked itself down and dead 2024-09-14T03:52:06.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:52:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:52:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:52:06.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:52:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:52:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:52:06.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch ps' 2024-09-14T03:52:06.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:06 smithi114 podman[113115]: 2024-09-14 03:52:06.453029427 +0000 UTC m=+1.755678958 container died ddb654f06efc189d966c006924872a7ce307536d40bf1c18dafe64189a60c9c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-eb16b97, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-14T03:52:07.094 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:06 smithi114 ceph-mon[103029]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:52:07.094 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:06 smithi114 ceph-mon[103029]: osdmap e77: 8 total, 7 up, 8 in 2024-09-14T03:52:07.095 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:06 smithi114 ceph-mon[106969]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:52:07.095 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:06 smithi114 ceph-mon[106969]: osdmap e77: 8 total, 7 up, 8 in 2024-09-14T03:52:07.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:06 smithi114 podman[113115]: 2024-09-14 03:52:06.833051788 +0000 UTC m=+2.135701319 container cleanup ddb654f06efc189d966c006924872a7ce307536d40bf1c18dafe64189a60c9c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-eb16b97, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-14T03:52:07.095 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:06 smithi114 bash[113115]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0 2024-09-14T03:52:07.134 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:52:07.370 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:07 smithi114 podman[113181]: 2024-09-14 03:52:07.093436383 +0000 UTC m=+0.642651617 container remove ddb654f06efc189d966c006924872a7ce307536d40bf1c18dafe64189a60c9c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=quincy-eb16b97, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T03:52:07.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:06 smithi160 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:52:07.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:06 smithi160 ceph-mon[88407]: osdmap e77: 8 total, 7 up, 8 in 2024-09-14T03:52:07.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:52:06 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:52:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi114\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi114\", job=\"node\", machine=\"x86_64\", nodename=\"smithi114\", release=\"5.14.0-505.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:52:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:07 smithi114 ceph-mon[103029]: pgmap v101: 89 pgs: 13 stale+active+clean, 76 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 829 op/s 2024-09-14T03:52:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:07 smithi114 ceph-mon[103029]: osdmap e78: 8 total, 7 up, 8 in 2024-09-14T03:52:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:07 smithi114 ceph-mon[106969]: pgmap v101: 89 pgs: 13 stale+active+clean, 76 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 829 op/s 2024-09-14T03:52:08.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:07 smithi114 ceph-mon[106969]: osdmap e78: 8 total, 7 up, 8 in 2024-09-14T03:52:08.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:08.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:08.122 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:07 smithi114 podman[113251]: 2024-09-14 03:52:07.731203245 +0000 UTC m=+0.575099899 container died 00b6ab31d45ae5171687a31cc78a2e4dbe661bdd9f7925c646369696a4b20592 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=optimistic_brattain, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-14T03:52:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:07 smithi160 ceph-mon[88407]: pgmap v101: 89 pgs: 13 stale+active+clean, 76 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 829 op/s 2024-09-14T03:52:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:07 smithi160 ceph-mon[88407]: osdmap e78: 8 total, 7 up, 8 in 2024-09-14T03:52:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:09.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:08 smithi114 podman[113359]: 2024-09-14 03:52:08.848637533 +0000 UTC m=+0.267708264 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:52:09.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:08 smithi114 podman[113359]: 2024-09-14 03:52:08.981754673 +0000 UTC m=+0.400825398 container create 820f9011666c5d7ca0557bacdfe4730a5c9bb8e64e02d153eef04a254bcd7358 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-a278231, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:52:09.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:09 smithi114 podman[113359]: 2024-09-14 03:52:09.486659412 +0000 UTC m=+0.905730148 container init 820f9011666c5d7ca0557bacdfe4730a5c9bb8e64e02d153eef04a254bcd7358 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a278231, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-14T03:52:09.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:09 smithi114 podman[113359]: 2024-09-14 03:52:09.500571274 +0000 UTC m=+0.919642007 container start 820f9011666c5d7ca0557bacdfe4730a5c9bb8e64e02d153eef04a254bcd7358 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-a278231, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-14T03:52:09.763 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:09 smithi114 podman[113359]: 2024-09-14 03:52:09.611570142 +0000 UTC m=+1.030640872 container attach 820f9011666c5d7ca0557bacdfe4730a5c9bb8e64e02d153eef04a254bcd7358 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-a278231, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2) 2024-09-14T03:52:09.765 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-14T03:52:09.765 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.a smithi114 *:9093,9094 running (5m) 24s ago 10m 17.8M - 0.27.0 11f11916f8cd eaf39a5cb581 2024-09-14T03:52:09.765 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.a smithi160 *:3000 running (4m) 67s ago 9m 84.4M - 9.4.7 954c08fa6188 aa46fcb7cedc 2024-09-14T03:52:09.765 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi114.ebmrat smithi114 running (8m) 24s ago 8m 18.1M - 17.2.7-1629-geb16b97b ab51d1bebd12 00f4a6ea534e 2024-09-14T03:52:09.765 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi160.ywyjae smithi160 running (8m) 67s ago 8m 14.8M - 17.2.7-1629-geb16b97b ab51d1bebd12 a87b5ff7e907 2024-09-14T03:52:09.766 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x smithi160 *:8443,9283,8765 running (3m) 67s ago 13m 454M - 19.3.0-4907-ga2782315 8801d2529e43 574eaed954d9 2024-09-14T03:52:09.766 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y smithi114 *:8443,9283,8765 running (4m) 24s ago 15m 513M - 19.3.0-4907-ga2782315 8801d2529e43 4f6e65854aef 2024-09-14T03:52:09.766 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a smithi114 running (3m) 24s ago 15m 46.9M 2048M 19.3.0-4907-ga2782315 8801d2529e43 2ca35e38ecf5 2024-09-14T03:52:09.766 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b smithi160 running (78s) 67s ago 14m 25.0M 2048M 19.3.0-4907-ga2782315 8801d2529e43 c470c18a013d 2024-09-14T03:52:09.766 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c smithi114 running (104s) 24s ago 14m 35.2M 2048M 19.3.0-4907-ga2782315 8801d2529e43 0d84986478c2 2024-09-14T03:52:09.766 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.a smithi114 *:9100 running (5m) 24s ago 9m 10.4M - 1.7.0 72c9c2088986 ff1fe8404b33 2024-09-14T03:52:09.766 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.b smithi160 *:9100 running (4m) 67s ago 9m 11.5M - 1.7.0 72c9c2088986 289de68494ec 2024-09-14T03:52:09.766 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (13m) 24s ago 13m 388M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 ddb654f06efc 2024-09-14T03:52:09.767 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (13m) 24s ago 13m 343M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 a709e75520db 2024-09-14T03:52:09.767 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (12m) 24s ago 12m 483M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 a76e6ee3bc54 2024-09-14T03:52:09.767 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi114 running (12m) 24s ago 12m 506M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 b43e985ecf6c 2024-09-14T03:52:09.767 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi160 running (12m) 67s ago 12m 413M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 e89693bb3b76 2024-09-14T03:52:09.767 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi160 running (11m) 67s ago 11m 437M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 fb664f7f461a 2024-09-14T03:52:09.767 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 smithi160 running (11m) 67s ago 11m 309M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 0403446dedd1 2024-09-14T03:52:09.767 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 smithi160 running (10m) 67s ago 10m 392M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 39f0a617d104 2024-09-14T03:52:09.767 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.a smithi160 *:9095 running (3m) 67s ago 10m 51.6M - 2.51.0 1d3b7f56885b 1b09e4c45bc1 2024-09-14T03:52:10.033 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:09 smithi114 ceph-mon[106969]: pgmap v103: 89 pgs: 13 stale+active+clean, 76 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 510 op/s 2024-09-14T03:52:10.033 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:09 smithi114 ceph-mon[103029]: pgmap v103: 89 pgs: 13 stale+active+clean, 76 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 510 op/s 2024-09-14T03:52:10.034 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:09 smithi114 podman[113359]: 2024-09-14 03:52:09.791262397 +0000 UTC m=+1.210333125 container died 820f9011666c5d7ca0557bacdfe4730a5c9bb8e64e02d153eef04a254bcd7358 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a278231, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-09-14T03:52:10.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:09 smithi160 ceph-mon[88407]: pgmap v103: 89 pgs: 13 stale+active+clean, 76 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 510 op/s 2024-09-14T03:52:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:52:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:52:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:52:11.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:10 smithi114 ceph-mon[106969]: from='client.44123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:52:11.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:10 smithi114 ceph-mon[103029]: from='client.44123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:52:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:10 smithi160 ceph-mon[88407]: from='client.44123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:52:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:52:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:52:11] "GET /metrics HTTP/1.1" 200 36247 "" "Prometheus/2.51.0" 2024-09-14T03:52:11.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:11 smithi114 podman[113454]: 2024-09-14 03:52:11.311070664 +0000 UTC m=+1.511491203 container remove 820f9011666c5d7ca0557bacdfe4730a5c9bb8e64e02d153eef04a254bcd7358 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-14T03:52:11.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:11 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.0.service: Deactivated successfully. 2024-09-14T03:52:11.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:11 smithi114 systemd[1]: Stopped Ceph osd.0 for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:52:11.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:11 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.0.service: Consumed 11min 48.871s CPU time. 2024-09-14T03:52:11.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch upgrade pause' 2024-09-14T03:52:11.940 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:11 smithi114 systemd[1]: Starting Ceph osd.0 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:52:12.305 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:11 smithi160 ceph-mon[88407]: pgmap v104: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 969 op/s; 10956/132426 objects degraded (8.273%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:12.305 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:11 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 10956/132426 objects degraded (8.273%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:12.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:11 smithi114 ceph-mon[106969]: pgmap v104: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 969 op/s; 10956/132426 objects degraded (8.273%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:12.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:11 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 10956/132426 objects degraded (8.273%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:12.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:11 smithi114 ceph-mon[103029]: pgmap v104: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 969 op/s; 10956/132426 objects degraded (8.273%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:12.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:11 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 10956/132426 objects degraded (8.273%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:12.367 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:12 smithi114 podman[113585]: 2024-09-14 03:52:12.064676956 +0000 UTC m=+0.032201261 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:52:12.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:12 smithi114 podman[113585]: 2024-09-14 03:52:12.366166095 +0000 UTC m=+0.333690387 container create 2d5d76be7007be2bdaeca3c48087ed683a2494123f5455e91a0d8263d621b236 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-a278231, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T03:52:12.838 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:52:12.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:12 smithi114 podman[113585]: 2024-09-14 03:52:12.660791983 +0000 UTC m=+0.628316271 container init 2d5d76be7007be2bdaeca3c48087ed683a2494123f5455e91a0d8263d621b236 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-a278231, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T03:52:12.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:12 smithi114 podman[113585]: 2024-09-14 03:52:12.669915601 +0000 UTC m=+0.637439894 container start 2d5d76be7007be2bdaeca3c48087ed683a2494123f5455e91a0d8263d621b236 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-a278231, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T03:52:12.983 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:12 smithi114 podman[113585]: 2024-09-14 03:52:12.769574913 +0000 UTC m=+0.737099243 container attach 2d5d76be7007be2bdaeca3c48087ed683a2494123f5455e91a0d8263d621b236 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate, RELEASE=main-a278231, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-14T03:52:12.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:12 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate[113618]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:52:12.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:12 smithi114 bash[113585]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:52:12.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:12 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate[113618]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:52:12.984 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:12 smithi114 bash[113585]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:52:13.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate[113618]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-14T03:52:13.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 bash[113585]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-14T03:52:13.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate[113618]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-14T03:52:13.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 bash[113585]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-14T03:52:13.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate[113618]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-14T03:52:13.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 bash[113585]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-14T03:52:13.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate[113618]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-14T03:52:13.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 bash[113585]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-14T03:52:13.621 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate[113618]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-14T03:52:13.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 bash[113585]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-14T03:52:13.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate[113618]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-14T03:52:13.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 bash[113585]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-14T03:52:13.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate[113618]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-14T03:52:13.622 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 bash[113585]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-14T03:52:14.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:13 smithi114 ceph-mon[103029]: pgmap v105: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 459 op/s; 10956/132426 objects degraded (8.273%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:14.351 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:13 smithi114 ceph-mon[106969]: pgmap v105: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 459 op/s; 10956/132426 objects degraded (8.273%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:14.351 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:13 smithi114 podman[113585]: 2024-09-14 03:52:13.994405213 +0000 UTC m=+1.961929510 container died 2d5d76be7007be2bdaeca3c48087ed683a2494123f5455e91a0d8263d621b236 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-a278231, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-14T03:52:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:13 smithi160 ceph-mon[88407]: pgmap v105: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 459 op/s; 10956/132426 objects degraded (8.273%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:15.474 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:15 smithi114 podman[113817]: 2024-09-14 03:52:15.128411419 +0000 UTC m=+1.550559566 container remove 2d5d76be7007be2bdaeca3c48087ed683a2494123f5455e91a0d8263d621b236 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-activate, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a278231, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-14T03:52:15.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:15 smithi114 podman[113908]: 2024-09-14 03:52:15.769869199 +0000 UTC m=+0.261048814 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:52:15.985 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:15 smithi114 podman[113908]: 2024-09-14 03:52:15.884000369 +0000 UTC m=+0.375179983 container create bde17c62ee62fce117b7d483685503f059f9a5dea92f09d27b109e1b66b70b76 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a278231) 2024-09-14T03:52:15.985 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:15 smithi114 ceph-mon[106969]: pgmap v106: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 9736/117525 objects degraded (8.284%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:16.159 INFO:teuthology.orchestra.run.smithi114.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:52:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:15 smithi114 ceph-mon[103029]: pgmap v106: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 9736/117525 objects degraded (8.284%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:16.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:16 smithi114 podman[113908]: 2024-09-14 03:52:16.187335475 +0000 UTC m=+0.678515107 container init bde17c62ee62fce117b7d483685503f059f9a5dea92f09d27b109e1b66b70b76 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-a278231, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:52:16.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:16 smithi114 podman[113908]: 2024-09-14 03:52:16.198265005 +0000 UTC m=+0.689444625 container start bde17c62ee62fce117b7d483685503f059f9a5dea92f09d27b109e1b66b70b76 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0, org.label-schema.license=GPLv2, RELEASE=main-a278231, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-14T03:52:16.371 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:16 smithi114 ceph-osd[113984]: -- 172.21.15.114:0/1988435739 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563cd8117a40 con 0x563cd80f5c00 2024-09-14T03:52:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:15 smithi160 ceph-mon[88407]: pgmap v106: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 9736/117525 objects degraded (8.284%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:16.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'sleep 60' 2024-09-14T03:52:17.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:17 smithi114 bash[113908]: bde17c62ee62fce117b7d483685503f059f9a5dea92f09d27b109e1b66b70b76 2024-09-14T03:52:17.121 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:17 smithi114 systemd[1]: Started Ceph osd.0 for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:52:17.397 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:52:17.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:17 smithi114 ceph-mon[106969]: from='client.34239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:52:17.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:17 smithi114 ceph-mon[106969]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:52:17.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:17.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:17.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:17 smithi114 ceph-mon[103029]: from='client.34239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:52:17.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:17 smithi114 ceph-mon[103029]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:52:17.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:17.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:17 smithi160 ceph-mon[88407]: from='client.34239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:52:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:17 smithi160 ceph-mon[88407]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:52:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:18.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:18 smithi114 ceph-mon[103029]: pgmap v107: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 864 op/s; 9736/117525 objects degraded (8.284%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:18.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:18.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:52:18.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:18 smithi114 ceph-mon[106969]: pgmap v107: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 864 op/s; 9736/117525 objects degraded (8.284%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:18.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:18.344 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:52:18.344 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:18 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[113954]: 2024-09-14T03:52:18.089+0000 7fd5ec1dc640 -1 Falling back to public interface 2024-09-14T03:52:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:18 smithi160 ceph-mon[88407]: pgmap v107: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 864 op/s; 9736/117525 objects degraded (8.284%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:52:19.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:19 smithi114 ceph-mon[106969]: pgmap v108: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 9736/117525 objects degraded (8.284%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:19.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9736/117525 objects degraded (8.284%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:19 smithi114 ceph-mon[103029]: pgmap v108: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 9736/117525 objects degraded (8.284%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9736/117525 objects degraded (8.284%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:19 smithi160 ceph-mon[88407]: pgmap v108: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 9736/117525 objects degraded (8.284%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9736/117525 objects degraded (8.284%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:20.348 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:52:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:52:20.216Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:52:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:21 smithi160 ceph-mon[88407]: pgmap v109: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 8479/102456 objects degraded (8.276%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:21.776 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:52:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:52:21] "GET /metrics HTTP/1.1" 200 36247 "" "Prometheus/2.51.0" 2024-09-14T03:52:21.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:21 smithi114 ceph-mon[106969]: pgmap v109: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 8479/102456 objects degraded (8.276%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:21.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:21 smithi114 ceph-mon[103029]: pgmap v109: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 8479/102456 objects degraded (8.276%); 0 B/s, 0 objects/s recovering 2024-09-14T03:52:23.848 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:23 smithi114 ceph-mon[106969]: pgmap v110: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 8479/102456 objects degraded (8.276%) 2024-09-14T03:52:23.848 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:23.849 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:23.849 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:23.849 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:23.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:23 smithi114 ceph-mon[103029]: pgmap v110: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 8479/102456 objects degraded (8.276%) 2024-09-14T03:52:23.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:23.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:23.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:23.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:23 smithi160 ceph-mon[88407]: pgmap v110: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 2.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 8479/102456 objects degraded (8.276%) 2024-09-14T03:52:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:24.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8479/102456 objects degraded (8.276%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:24.874 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[113954]: 2024-09-14T03:52:24.776+0000 7fd5ec1dc640 -1 osd.0 76 log_to_monitors true 2024-09-14T03:52:24.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8479/102456 objects degraded (8.276%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:24.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8479/102456 objects degraded (8.276%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:25.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:25 smithi114 ceph-mon[106969]: pgmap v111: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 7336/88500 objects degraded (8.289%) 2024-09-14T03:52:25.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:25 smithi114 ceph-mon[106969]: from='osd.0 [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-14T03:52:25.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:25 smithi114 ceph-mon[106969]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-14T03:52:25.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:25 smithi114 ceph-mon[103029]: pgmap v111: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 7336/88500 objects degraded (8.289%) 2024-09-14T03:52:25.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:25 smithi114 ceph-mon[103029]: from='osd.0 [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-14T03:52:25.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:25 smithi114 ceph-mon[103029]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-14T03:52:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:25 smithi160 ceph-mon[88407]: pgmap v111: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 7336/88500 objects degraded (8.289%) 2024-09-14T03:52:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:25 smithi160 ceph-mon[88407]: from='osd.0 [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-14T03:52:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:25 smithi160 ceph-mon[88407]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-14T03:52:26.613 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[113954]: 2024-09-14T03:52:26.155+0000 7fd5dff7e640 -1 osd.0 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:52:26.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[106969]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-14T03:52:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[106969]: osdmap e79: 8 total, 7 up, 8 in 2024-09-14T03:52:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[106969]: from='osd.0 [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:52:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[106969]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:52:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[106969]: from='osd.0 ' entity='osd.0' 2024-09-14T03:52:26.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[103029]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-14T03:52:26.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[103029]: osdmap e79: 8 total, 7 up, 8 in 2024-09-14T03:52:26.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[103029]: from='osd.0 [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:52:26.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[103029]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:52:26.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:26.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:26.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:26 smithi114 ceph-mon[103029]: from='osd.0 ' entity='osd.0' 2024-09-14T03:52:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:26 smithi160 ceph-mon[88407]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-14T03:52:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:26 smithi160 ceph-mon[88407]: osdmap e79: 8 total, 7 up, 8 in 2024-09-14T03:52:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:26 smithi160 ceph-mon[88407]: from='osd.0 [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:52:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:26 smithi160 ceph-mon[88407]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:52:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:26 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:26.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:26 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:26.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:26 smithi160 ceph-mon[88407]: from='osd.0 ' entity='osd.0' 2024-09-14T03:52:27.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[106969]: pgmap v113: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 7336/88500 objects degraded (8.289%) 2024-09-14T03:52:27.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:27.880 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:27.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[106969]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:52:27.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[106969]: osd.0 [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658] boot 2024-09-14T03:52:27.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[106969]: osdmap e80: 8 total, 8 up, 8 in 2024-09-14T03:52:27.881 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-14T03:52:27.881 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[103029]: pgmap v113: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 7336/88500 objects degraded (8.289%) 2024-09-14T03:52:27.881 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:27.881 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:27.881 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[103029]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:52:27.881 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[103029]: osd.0 [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658] boot 2024-09-14T03:52:27.881 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[103029]: osdmap e80: 8 total, 8 up, 8 in 2024-09-14T03:52:27.882 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:27 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-14T03:52:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:27 smithi160 ceph-mon[88407]: pgmap v113: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 7336/88500 objects degraded (8.289%) 2024-09-14T03:52:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:27 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:27 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:27 smithi160 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:52:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:27 smithi160 ceph-mon[88407]: osd.0 [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658] boot 2024-09-14T03:52:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:27 smithi160 ceph-mon[88407]: osdmap e80: 8 total, 8 up, 8 in 2024-09-14T03:52:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:27 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-14T03:52:29.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:29 smithi114 ceph-mon[106969]: osdmap e81: 8 total, 8 up, 8 in 2024-09-14T03:52:29.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7336/88500 objects degraded (8.289%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:29.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:29 smithi114 ceph-mon[103029]: osdmap e81: 8 total, 8 up, 8 in 2024-09-14T03:52:29.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7336/88500 objects degraded (8.289%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:29 smithi160 ceph-mon[88407]: osdmap e81: 8 total, 8 up, 8 in 2024-09-14T03:52:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7336/88500 objects degraded (8.289%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:30.559 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:52:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:52:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:52:30.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:30 smithi160 ceph-mon[88407]: pgmap v116: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 7336/88500 objects degraded (8.289%) 2024-09-14T03:52:30.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:30 smithi160 ceph-mon[88407]: osdmap e82: 8 total, 8 up, 8 in 2024-09-14T03:52:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:30 smithi114 ceph-mon[106969]: pgmap v116: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 7336/88500 objects degraded (8.289%) 2024-09-14T03:52:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:30 smithi114 ceph-mon[106969]: osdmap e82: 8 total, 8 up, 8 in 2024-09-14T03:52:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:30 smithi114 ceph-mon[103029]: pgmap v116: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 7336/88500 objects degraded (8.289%) 2024-09-14T03:52:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:30 smithi114 ceph-mon[103029]: osdmap e82: 8 total, 8 up, 8 in 2024-09-14T03:52:31.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:52:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:52:31] "GET /metrics HTTP/1.1" 200 36313 "" "Prometheus/2.51.0" 2024-09-14T03:52:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:32 smithi160 ceph-mon[88407]: pgmap v118: 89 pgs: 4 remapped+peering, 9 active+undersized, 2 active+undersized+degraded, 74 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 3/78888 objects degraded (0.004%) 2024-09-14T03:52:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:52:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:52:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:52:32.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:52:32.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:52:32.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[103029]: pgmap v118: 89 pgs: 4 remapped+peering, 9 active+undersized, 2 active+undersized+degraded, 74 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 3/78888 objects degraded (0.004%) 2024-09-14T03:52:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:52:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:52:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:52:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:52:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:52:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[106969]: pgmap v118: 89 pgs: 4 remapped+peering, 9 active+undersized, 2 active+undersized+degraded, 74 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 3/78888 objects degraded (0.004%) 2024-09-14T03:52:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:52:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:52:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:52:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:52:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:52:32.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:33 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/78888 objects degraded (0.004%), 2 pgs degraded) 2024-09-14T03:52:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:33 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T03:52:33.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:33 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/78888 objects degraded (0.004%), 2 pgs degraded) 2024-09-14T03:52:33.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:33 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T03:52:33.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:33 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/78888 objects degraded (0.004%), 2 pgs degraded) 2024-09-14T03:52:33.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:33 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T03:52:34.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:34 smithi114 ceph-mon[103029]: pgmap v119: 89 pgs: 4 remapped+peering, 85 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 533 op/s 2024-09-14T03:52:34.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:34 smithi114 ceph-mon[106969]: pgmap v119: 89 pgs: 4 remapped+peering, 85 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 533 op/s 2024-09-14T03:52:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:34 smithi160 ceph-mon[88407]: pgmap v119: 89 pgs: 4 remapped+peering, 85 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 533 op/s 2024-09-14T03:52:35.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:35 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 3583/68232 objects degraded (5.251%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:35 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 3583/68232 objects degraded (5.251%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:35 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 3583/68232 objects degraded (5.251%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:36 smithi114 ceph-mon[103029]: pgmap v120: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 3583/68232 objects degraded (5.251%); 0 B/s, 33 objects/s recovering 2024-09-14T03:52:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:36 smithi114 ceph-mon[106969]: pgmap v120: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 3583/68232 objects degraded (5.251%); 0 B/s, 33 objects/s recovering 2024-09-14T03:52:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:36 smithi160 ceph-mon[88407]: pgmap v120: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 940 op/s; 3583/68232 objects degraded (5.251%); 0 B/s, 33 objects/s recovering 2024-09-14T03:52:38.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:38 smithi114 ceph-mon[103029]: pgmap v121: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 825 op/s; 3583/68232 objects degraded (5.251%); 0 B/s, 29 objects/s recovering 2024-09-14T03:52:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:38 smithi114 ceph-mon[106969]: pgmap v121: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 825 op/s; 3583/68232 objects degraded (5.251%); 0 B/s, 29 objects/s recovering 2024-09-14T03:52:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:38 smithi160 ceph-mon[88407]: pgmap v121: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 825 op/s; 3583/68232 objects degraded (5.251%); 0 B/s, 29 objects/s recovering 2024-09-14T03:52:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:52:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:40 smithi160 ceph-mon[88407]: pgmap v122: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 3583/68232 objects degraded (5.251%); 0 B/s, 24 objects/s recovering 2024-09-14T03:52:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:52:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:52:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:52:40.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:40 smithi114 ceph-mon[106969]: pgmap v122: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 3583/68232 objects degraded (5.251%); 0 B/s, 24 objects/s recovering 2024-09-14T03:52:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:40 smithi114 ceph-mon[103029]: pgmap v122: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 3583/68232 objects degraded (5.251%); 0 B/s, 24 objects/s recovering 2024-09-14T03:52:41.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:41 smithi114 ceph-mon[106969]: pgmap v123: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 3583/55095 objects degraded (6.503%); 0 B/s, 53 objects/s recovering 2024-09-14T03:52:41.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:41 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3583/55095 objects degraded (6.503%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:41.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:41 smithi114 ceph-mon[103029]: pgmap v123: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 3583/55095 objects degraded (6.503%); 0 B/s, 53 objects/s recovering 2024-09-14T03:52:41.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:41 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3583/55095 objects degraded (6.503%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:41 smithi160 ceph-mon[88407]: pgmap v123: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 3583/55095 objects degraded (6.503%); 0 B/s, 53 objects/s recovering 2024-09-14T03:52:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:41 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3583/55095 objects degraded (6.503%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:41.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:52:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:52:41] "GET /metrics HTTP/1.1" 200 36313 "" "Prometheus/2.51.0" 2024-09-14T03:52:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:43 smithi160 ceph-mon[88407]: pgmap v124: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 3583/55095 objects degraded (6.503%); 0 B/s, 50 objects/s recovering 2024-09-14T03:52:43.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:43 smithi114 ceph-mon[106969]: pgmap v124: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 3583/55095 objects degraded (6.503%); 0 B/s, 50 objects/s recovering 2024-09-14T03:52:43.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:43 smithi114 ceph-mon[103029]: pgmap v124: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.1 GiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 3583/55095 objects degraded (6.503%); 0 B/s, 50 objects/s recovering 2024-09-14T03:52:45.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:45 smithi114 ceph-mon[103029]: pgmap v125: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 871 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3583/41883 objects degraded (8.555%); 0 B/s, 82 objects/s recovering 2024-09-14T03:52:45.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:45 smithi114 ceph-mon[106969]: pgmap v125: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 871 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3583/41883 objects degraded (8.555%); 0 B/s, 82 objects/s recovering 2024-09-14T03:52:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:45 smithi160 ceph-mon[88407]: pgmap v125: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 871 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3583/41883 objects degraded (8.555%); 0 B/s, 82 objects/s recovering 2024-09-14T03:52:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:47 smithi114 ceph-mon[103029]: pgmap v126: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 871 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 3583/41883 objects degraded (8.555%); 0 B/s, 62 objects/s recovering 2024-09-14T03:52:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:47 smithi114 ceph-mon[103029]: osdmap e83: 8 total, 8 up, 8 in 2024-09-14T03:52:47.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:47 smithi114 ceph-mon[106969]: pgmap v126: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 871 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 3583/41883 objects degraded (8.555%); 0 B/s, 62 objects/s recovering 2024-09-14T03:52:47.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:47 smithi114 ceph-mon[106969]: osdmap e83: 8 total, 8 up, 8 in 2024-09-14T03:52:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:47 smithi160 ceph-mon[88407]: pgmap v126: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 871 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 3583/41883 objects degraded (8.555%); 0 B/s, 62 objects/s recovering 2024-09-14T03:52:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:47 smithi160 ceph-mon[88407]: osdmap e83: 8 total, 8 up, 8 in 2024-09-14T03:52:48.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:48 smithi114 ceph-mon[106969]: osdmap e84: 8 total, 8 up, 8 in 2024-09-14T03:52:48.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:48 smithi114 ceph-mon[103029]: osdmap e84: 8 total, 8 up, 8 in 2024-09-14T03:52:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:48 smithi160 ceph-mon[88407]: osdmap e84: 8 total, 8 up, 8 in 2024-09-14T03:52:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:49 smithi114 ceph-mon[103029]: pgmap v129: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 871 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 550 op/s; 3583/41883 objects degraded (8.555%); 0 B/s, 48 objects/s recovering 2024-09-14T03:52:49.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:49 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3583/41883 objects degraded (8.555%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:49 smithi114 ceph-mon[106969]: pgmap v129: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 871 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 550 op/s; 3583/41883 objects degraded (8.555%); 0 B/s, 48 objects/s recovering 2024-09-14T03:52:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:49 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3583/41883 objects degraded (8.555%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:49 smithi160 ceph-mon[88407]: pgmap v129: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 871 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 550 op/s; 3583/41883 objects degraded (8.555%); 0 B/s, 48 objects/s recovering 2024-09-14T03:52:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:49 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3583/41883 objects degraded (8.555%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:52:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:52:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:52:51.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:51 smithi114 ceph-mon[106969]: pgmap v130: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 655 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 983 op/s; 2391/31473 objects degraded (7.597%); 0 B/s, 91 objects/s recovering 2024-09-14T03:52:51.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:51 smithi114 ceph-mon[103029]: pgmap v130: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 655 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 983 op/s; 2391/31473 objects degraded (7.597%); 0 B/s, 91 objects/s recovering 2024-09-14T03:52:51.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:52:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:52:51] "GET /metrics HTTP/1.1" 200 36313 "" "Prometheus/2.51.0" 2024-09-14T03:52:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:51 smithi160 ceph-mon[88407]: pgmap v130: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 655 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 983 op/s; 2391/31473 objects degraded (7.597%); 0 B/s, 91 objects/s recovering 2024-09-14T03:52:53.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:53 smithi114 ceph-mon[103029]: pgmap v131: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 633 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 476 op/s; 2391/30432 objects degraded (7.857%); 0 B/s, 75 objects/s recovering 2024-09-14T03:52:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:53 smithi114 ceph-mon[106969]: pgmap v131: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 633 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 476 op/s; 2391/30432 objects degraded (7.857%); 0 B/s, 75 objects/s recovering 2024-09-14T03:52:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:53 smithi160 ceph-mon[88407]: pgmap v131: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 633 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 476 op/s; 2391/30432 objects degraded (7.857%); 0 B/s, 75 objects/s recovering 2024-09-14T03:52:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:52:54.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:54 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2391/30432 objects degraded (7.857%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:54.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:54 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2391/30432 objects degraded (7.857%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:54 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2391/30432 objects degraded (7.857%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:55.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:55 smithi114 ceph-mon[103029]: pgmap v132: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 379 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 2391/18249 objects degraded (13.102%); 0 B/s, 119 objects/s recovering 2024-09-14T03:52:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:55 smithi114 ceph-mon[106969]: pgmap v132: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 379 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 2391/18249 objects degraded (13.102%); 0 B/s, 119 objects/s recovering 2024-09-14T03:52:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:55 smithi160 ceph-mon[88407]: pgmap v132: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 379 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 2391/18249 objects degraded (13.102%); 0 B/s, 119 objects/s recovering 2024-09-14T03:52:57.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:57 smithi114 ceph-mon[103029]: pgmap v133: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 364 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 2391/17508 objects degraded (13.657%); 0 B/s, 96 objects/s recovering 2024-09-14T03:52:57.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:57 smithi114 ceph-mon[106969]: pgmap v133: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 364 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 2391/17508 objects degraded (13.657%); 0 B/s, 96 objects/s recovering 2024-09-14T03:52:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:57 smithi160 ceph-mon[88407]: pgmap v133: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 364 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 2391/17508 objects degraded (13.657%); 0 B/s, 96 objects/s recovering 2024-09-14T03:52:59.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:59 smithi114 ceph-mon[106969]: pgmap v134: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 364 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 2391/17508 objects degraded (13.657%); 0 B/s, 87 objects/s recovering 2024-09-14T03:52:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:52:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2391/17508 objects degraded (13.657%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:59.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:59 smithi114 ceph-mon[103029]: pgmap v134: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 364 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 2391/17508 objects degraded (13.657%); 0 B/s, 87 objects/s recovering 2024-09-14T03:52:59.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:52:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2391/17508 objects degraded (13.657%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T03:52:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:59 smithi160 ceph-mon[88407]: pgmap v134: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 364 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 2391/17508 objects degraded (13.657%); 0 B/s, 87 objects/s recovering 2024-09-14T03:52:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:52:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2391/17508 objects degraded (13.657%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T03:53:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:53:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:53:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:53:01.604 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 134633 objects 2024-09-14T03:53:01.604 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :157.323 2024-09-14T03:53:01.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:01 smithi114 ceph-mon[103029]: pgmap v135: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 118 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 2391/5694 objects degraded (41.992%); 0 B/s, 109 objects/s recovering 2024-09-14T03:53:01.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:01 smithi114 ceph-mon[103029]: osdmap e85: 8 total, 8 up, 8 in 2024-09-14T03:53:01.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:53:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:53:01] "GET /metrics HTTP/1.1" 200 36314 "" "Prometheus/2.51.0" 2024-09-14T03:53:01.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:01 smithi114 ceph-mon[106969]: pgmap v135: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 118 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 2391/5694 objects degraded (41.992%); 0 B/s, 109 objects/s recovering 2024-09-14T03:53:01.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:01 smithi114 ceph-mon[106969]: osdmap e85: 8 total, 8 up, 8 in 2024-09-14T03:53:01.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:01 smithi160 ceph-mon[88407]: pgmap v135: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 118 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 2391/5694 objects degraded (41.992%); 0 B/s, 109 objects/s recovering 2024-09-14T03:53:01.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:01 smithi160 ceph-mon[88407]: osdmap e85: 8 total, 8 up, 8 in 2024-09-14T03:53:02.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:02 smithi114 ceph-mon[103029]: osdmap e86: 8 total, 8 up, 8 in 2024-09-14T03:53:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:02 smithi114 ceph-mon[106969]: osdmap e86: 8 total, 8 up, 8 in 2024-09-14T03:53:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:02 smithi160 ceph-mon[88407]: osdmap e86: 8 total, 8 up, 8 in 2024-09-14T03:53:03.372 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-14T03:53:03.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-14T03:53:03.659 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:53:03.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:03 smithi114 ceph-mon[103029]: pgmap v138: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 101 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 555 op/s; 2391/4905 objects degraded (48.746%); 0 B/s, 45 objects/s recovering 2024-09-14T03:53:03.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:03 smithi114 ceph-mon[106969]: pgmap v138: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 101 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 555 op/s; 2391/4905 objects degraded (48.746%); 0 B/s, 45 objects/s recovering 2024-09-14T03:53:03.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:03 smithi160 ceph-mon[88407]: pgmap v138: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 101 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 555 op/s; 2391/4905 objects degraded (48.746%); 0 B/s, 45 objects/s recovering 2024-09-14T03:53:04.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:04 smithi114 ceph-mon[103029]: osdmap e87: 8 total, 8 up, 8 in 2024-09-14T03:53:04.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2391/4905 objects degraded (48.746%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T03:53:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:04 smithi114 ceph-mon[106969]: osdmap e87: 8 total, 8 up, 8 in 2024-09-14T03:53:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2391/4905 objects degraded (48.746%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T03:53:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:04 smithi160 ceph-mon[88407]: osdmap e87: 8 total, 8 up, 8 in 2024-09-14T03:53:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2391/4905 objects degraded (48.746%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T03:53:05.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:05 smithi114 ceph-mon[106969]: pgmap v140: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 87 active+clean; 646 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 0 B/s, 297 objects/s recovering 2024-09-14T03:53:05.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:05 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2391/4905 objects degraded (48.746%), 2 pgs degraded) 2024-09-14T03:53:05.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:05 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T03:53:05.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:05 smithi114 ceph-mon[106969]: osdmap e88: 8 total, 8 up, 8 in 2024-09-14T03:53:05.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:05 smithi114 ceph-mon[103029]: pgmap v140: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 87 active+clean; 646 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 0 B/s, 297 objects/s recovering 2024-09-14T03:53:05.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:05 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2391/4905 objects degraded (48.746%), 2 pgs degraded) 2024-09-14T03:53:05.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:05 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T03:53:05.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:05 smithi114 ceph-mon[103029]: osdmap e88: 8 total, 8 up, 8 in 2024-09-14T03:53:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:05 smithi160 ceph-mon[88407]: pgmap v140: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 87 active+clean; 646 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 0 B/s, 297 objects/s recovering 2024-09-14T03:53:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:05 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2391/4905 objects degraded (48.746%), 2 pgs degraded) 2024-09-14T03:53:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:05 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T03:53:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:05 smithi160 ceph-mon[88407]: osdmap e88: 8 total, 8 up, 8 in 2024-09-14T03:53:06.514 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_1' removed 2024-09-14T03:53:06.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2913574966' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T03:53:06.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2913574966' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T03:53:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2913574966' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T03:53:07.181 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-14T03:53:07.182 INFO:tasks.radosbench:Beginning radosbench... 2024-09-14T03:53:07.183 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-14T03:53:07.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-14T03:53:07.432 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:53:07.837 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:07 smithi114 ceph-mon[103029]: pgmap v142: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 87 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 318 op/s; 0 B/s, 241 objects/s recovering 2024-09-14T03:53:07.837 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:07 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2913574966' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T03:53:07.837 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:07 smithi114 ceph-mon[103029]: osdmap e89: 8 total, 8 up, 8 in 2024-09-14T03:53:07.838 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:07 smithi114 ceph-mon[106969]: pgmap v142: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 87 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 318 op/s; 0 B/s, 241 objects/s recovering 2024-09-14T03:53:07.838 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:07 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2913574966' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T03:53:07.838 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:07 smithi114 ceph-mon[106969]: osdmap e89: 8 total, 8 up, 8 in 2024-09-14T03:53:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:07 smithi160 ceph-mon[88407]: pgmap v142: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 87 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 318 op/s; 0 B/s, 241 objects/s recovering 2024-09-14T03:53:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:07 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2913574966' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T03:53:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:07 smithi160 ceph-mon[88407]: osdmap e89: 8 total, 8 up, 8 in 2024-09-14T03:53:08.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:08.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:09.605 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_2' created 2024-09-14T03:53:09.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:09 smithi114 ceph-mon[103029]: pgmap v144: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:09 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/797862581' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-14T03:53:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:09 smithi114 ceph-mon[106969]: pgmap v144: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:09 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/797862581' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-14T03:53:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:09 smithi160 ceph-mon[88407]: pgmap v144: 73 pgs: 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:09 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/797862581' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-14T03:53:10.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-14T03:53:10.414 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:53:10.616 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:53:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:53:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:53:10.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:10 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/797862581' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-14T03:53:10.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:10 smithi114 ceph-mon[103029]: osdmap e90: 8 total, 8 up, 8 in 2024-09-14T03:53:10.800 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:10 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/797862581' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-14T03:53:10.801 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:10 smithi114 ceph-mon[106969]: osdmap e90: 8 total, 8 up, 8 in 2024-09-14T03:53:10.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:10 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/797862581' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-14T03:53:10.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:10 smithi160 ceph-mon[88407]: osdmap e90: 8 total, 8 up, 8 in 2024-09-14T03:53:11.546 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:53:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:53:11] "GET /metrics HTTP/1.1" 200 33949 "" "Prometheus/2.51.0" 2024-09-14T03:53:11.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:11 smithi114 ceph-mon[106969]: pgmap v146: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:11.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:11 smithi114 ceph-mon[106969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:53:11.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:11 smithi114 ceph-mon[106969]: osdmap e91: 8 total, 8 up, 8 in 2024-09-14T03:53:11.820 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:11 smithi114 ceph-mon[103029]: pgmap v146: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:11.820 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:11 smithi114 ceph-mon[103029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:53:11.820 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:11 smithi114 ceph-mon[103029]: osdmap e91: 8 total, 8 up, 8 in 2024-09-14T03:53:11.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:11 smithi160 ceph-mon[88407]: pgmap v146: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:11.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:11 smithi160 ceph-mon[88407]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:53:11.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:11 smithi160 ceph-mon[88407]: osdmap e91: 8 total, 8 up, 8 in 2024-09-14T03:53:12.623 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-14T03:53:12.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:12 smithi160 ceph-mon[88407]: osdmap e92: 8 total, 8 up, 8 in 2024-09-14T03:53:12.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:12 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1875426701' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:53:12.927 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:12 smithi114 ceph-mon[106969]: osdmap e92: 8 total, 8 up, 8 in 2024-09-14T03:53:12.927 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:12 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1875426701' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:53:12.927 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:12 smithi114 ceph-mon[103029]: osdmap e92: 8 total, 8 up, 8 in 2024-09-14T03:53:12.927 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:12 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1875426701' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:53:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:13 smithi160 ceph-mon[88407]: pgmap v149: 89 pgs: 12 unknown, 77 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:13 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1875426701' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:53:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:13 smithi160 ceph-mon[88407]: osdmap e93: 8 total, 8 up, 8 in 2024-09-14T03:53:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:14.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:13 smithi114 ceph-mon[106969]: pgmap v149: 89 pgs: 12 unknown, 77 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:14.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:13 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1875426701' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:53:14.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:13 smithi114 ceph-mon[106969]: osdmap e93: 8 total, 8 up, 8 in 2024-09-14T03:53:14.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:14.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:13 smithi114 ceph-mon[103029]: pgmap v149: 89 pgs: 12 unknown, 77 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:14.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:13 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1875426701' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:53:14.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:13 smithi114 ceph-mon[103029]: osdmap e93: 8 total, 8 up, 8 in 2024-09-14T03:53:14.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:14.237 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-14T03:53:14.239 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-14T03:53:14.240 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-14T03:53:14.293 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-14T03:53:14.293 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-14T03:53:14.293 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_118301 2024-09-14T03:53:14.293 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:53:14.293 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-14T03:53:15.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:14 smithi114 ceph-mon[106969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:53:15.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:14 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T03:53:15.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:14 smithi114 ceph-mon[103029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:53:15.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:14 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T03:53:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:14 smithi160 ceph-mon[88407]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:53:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:14 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T03:53:15.293 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 3958 3942 246.36 246.375 0.00331976 0.0040478 2024-09-14T03:53:16.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:15 smithi114 ceph-mon[103029]: pgmap v151: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:16.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:15 smithi114 ceph-mon[106969]: pgmap v151: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:15 smithi160 ceph-mon[88407]: pgmap v151: 89 pgs: 89 active+clean; 582 KiB data, 2.2 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:53:16.293 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 7635 7619 238.072 229.812 0.00295644 0.00419063 2024-09-14T03:53:17.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 11092 11076 230.728 216.062 0.00596957 0.00432674 2024-09-14T03:53:18.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:17 smithi114 ceph-mon[103029]: pgmap v152: 89 pgs: 89 active+clean; 96 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s wr, 254 op/s 2024-09-14T03:53:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:17 smithi114 ceph-mon[106969]: pgmap v152: 89 pgs: 89 active+clean; 96 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s wr, 254 op/s 2024-09-14T03:53:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:17 smithi160 ceph-mon[88407]: pgmap v152: 89 pgs: 89 active+clean; 96 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 16 MiB/s wr, 254 op/s 2024-09-14T03:53:18.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 14244 14228 222.291 197 0.00814553 0.00449076 2024-09-14T03:53:19.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 15 17247 17232 215.379 187.75 0.00575716 0.00463717 2024-09-14T03:53:20.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:19 smithi114 ceph-mon[103029]: pgmap v153: 89 pgs: 89 active+clean; 96 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s wr, 197 op/s 2024-09-14T03:53:20.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:19 smithi114 ceph-mon[106969]: pgmap v153: 89 pgs: 89 active+clean; 96 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s wr, 197 op/s 2024-09-14T03:53:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:19 smithi160 ceph-mon[88407]: pgmap v153: 89 pgs: 89 active+clean; 96 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s wr, 197 op/s 2024-09-14T03:53:20.295 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 20036 20020 208.522 174.25 0.00395282 0.00479001 2024-09-14T03:53:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:53:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:53:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:53:21.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 22730 22714 202.784 168.375 0.00639247 0.00492566 2024-09-14T03:53:21.700 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:53:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:53:21] "GET /metrics HTTP/1.1" 200 33949 "" "Prometheus/2.51.0" 2024-09-14T03:53:22.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:21 smithi114 ceph-mon[103029]: pgmap v154: 89 pgs: 89 active+clean; 812 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-09-14T03:53:22.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:21 smithi114 ceph-mon[106969]: pgmap v154: 89 pgs: 89 active+clean; 812 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-09-14T03:53:22.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:21 smithi160 ceph-mon[88407]: pgmap v154: 89 pgs: 89 active+clean; 812 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-09-14T03:53:22.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 24745 24729 193.177 125.938 0.0263495 0.00516853 2024-09-14T03:53:22.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:22.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:22.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch upgrade resume' 2024-09-14T03:53:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:23.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 27087 27071 187.975 146.375 0.00801452 0.00531289 2024-09-14T03:53:23.435 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:53:23.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:23 smithi114 ceph-mon[103029]: pgmap v155: 89 pgs: 89 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-14T03:53:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:23 smithi114 ceph-mon[106969]: pgmap v155: 89 pgs: 89 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-14T03:53:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:23 smithi160 ceph-mon[88407]: pgmap v155: 89 pgs: 89 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-14T03:53:24.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 29077 29061 181.614 124.375 0.00303501 0.00549759 2024-09-14T03:53:25.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 31055 31039 176.341 123.625 0.00205566 0.00565784 2024-09-14T03:53:26.112 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:25 smithi114 ceph-mon[106969]: pgmap v156: 89 pgs: 89 active+clean; 1.6 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T03:53:26.112 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:25 smithi114 ceph-mon[103029]: pgmap v156: 89 pgs: 89 active+clean; 1.6 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T03:53:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:25 smithi160 ceph-mon[88407]: pgmap v156: 89 pgs: 89 active+clean; 1.6 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T03:53:26.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 33160 33144 172.609 131.562 0.0121168 0.00578778 2024-09-14T03:53:27.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 35163 35147 168.96 125.188 0.0100766 0.00591294 2024-09-14T03:53:27.978 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:27 smithi114 ceph-mon[106969]: pgmap v157: 89 pgs: 89 active+clean; 1.8 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-14T03:53:27.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:27 smithi114 ceph-mon[103029]: pgmap v157: 89 pgs: 89 active+clean; 1.8 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-14T03:53:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:27 smithi160 ceph-mon[88407]: pgmap v157: 89 pgs: 89 active+clean; 1.8 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-14T03:53:28.294 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 36959 36943 164.909 112.25 0.00356679 0.00605818 2024-09-14T03:53:28.550 INFO:teuthology.orchestra.run.smithi114.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:53:29.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'echo wait for half of osds to upgrade' 2024-09-14T03:53:29.297 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 38528 38512 160.447 98.0625 0.0244934 0.00622602 2024-09-14T03:53:29.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:29 smithi114 ceph-mon[103029]: pgmap v158: 89 pgs: 89 active+clean; 1.8 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-14T03:53:29.830 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:29 smithi114 ceph-mon[103029]: from='client.54136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:29.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:29 smithi114 ceph-mon[103029]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:53:29.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:29.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:53:29.831 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:29 smithi114 ceph-mon[106969]: pgmap v158: 89 pgs: 89 active+clean; 1.8 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-14T03:53:29.831 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:29 smithi114 ceph-mon[106969]: from='client.54136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:29.831 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:29 smithi114 ceph-mon[106969]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:53:29.831 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:29.831 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:53:29.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:29 smithi160 ceph-mon[88407]: pgmap v158: 89 pgs: 89 active+clean; 1.8 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-14T03:53:29.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:29 smithi160 ceph-mon[88407]: from='client.54136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:29.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:29 smithi160 ceph-mon[88407]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:53:29.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:29.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:53:30.295 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 40036 40020 156.309 94.25 0.00624133 0.00639084 2024-09-14T03:53:30.329 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:53:30.562 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:53:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:53:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:53:31.295 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 41569 41553 152.75 95.8125 0.00264818 0.00650737 2024-09-14T03:53:31.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:31 smithi114 ceph-mon[106969]: pgmap v159: 89 pgs: 89 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 180 MiB/s wr, 2.88k op/s 2024-09-14T03:53:31.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:53:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:53:31] "GET /metrics HTTP/1.1" 200 36296 "" "Prometheus/2.51.0" 2024-09-14T03:53:31.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:31 smithi114 ceph-mon[103029]: pgmap v159: 89 pgs: 89 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 180 MiB/s wr, 2.88k op/s 2024-09-14T03:53:31.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:31 smithi160 ceph-mon[88407]: pgmap v159: 89 pgs: 89 active+clean; 2.2 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 180 MiB/s wr, 2.88k op/s 2024-09-14T03:53:32.296 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 43468 43452 150.855 118.688 0.00555175 0.00662213 2024-09-14T03:53:33.129 INFO:teuthology.orchestra.run.smithi114.stdout:wait for half of osds to upgrade 2024-09-14T03:53:33.296 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 45175 45159 148.53 106.688 0.0139078 0.00672657 2024-09-14T03:53:33.752 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:33 smithi114 ceph-mon[106969]: pgmap v160: 89 pgs: 89 active+clean; 2.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-14T03:53:33.752 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:33 smithi114 ceph-mon[103029]: pgmap v160: 89 pgs: 89 active+clean; 2.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-14T03:53:33.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'while ! ceph osd versions | grep $sha1 | egrep '"'"': [45678]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-14T03:53:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:33 smithi160 ceph-mon[88407]: pgmap v160: 89 pgs: 89 active+clean; 2.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-14T03:53:34.298 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:53:34.297831+0000 min lat: 0.00128604 max lat: 0.116699 avg lat: 0.00681719 2024-09-14T03:53:34.298 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:53:34.298 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 46923 46907 146.549 109.25 0.00631655 0.00681719 2024-09-14T03:53:34.377 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:53:35.298 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 48556 48540 144.43 102.062 0.0126281 0.00691892 2024-09-14T03:53:36.045 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:35 smithi114 ceph-mon[106969]: pgmap v161: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 145 MiB/s wr, 2.31k op/s 2024-09-14T03:53:36.045 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:35 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:36.045 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:35 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:36.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:35 smithi114 ceph-mon[103029]: pgmap v161: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 145 MiB/s wr, 2.31k op/s 2024-09-14T03:53:36.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:35 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:36.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:35 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:35 smithi160 ceph-mon[88407]: pgmap v161: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 145 MiB/s wr, 2.31k op/s 2024-09-14T03:53:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:35 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:35 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:36.298 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 50169 50153 142.447 100.812 0.0070673 0.00701443 2024-09-14T03:53:37.298 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 51846 51830 140.811 104.812 0.00998222 0.00709635 2024-09-14T03:53:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:37 smithi114 ceph-mon[103029]: pgmap v162: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-14T03:53:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:37 smithi114 ceph-mon[106969]: pgmap v162: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-14T03:53:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:37 smithi160 ceph-mon[88407]: pgmap v162: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-14T03:53:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:38.298 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 53304 53288 138.74 91.125 0.0123397 0.00720198 2024-09-14T03:53:38.642 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:53:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1627303317' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1627303317' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1627303317' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:39.298 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 54647 54631 136.548 83.9375 0.00426118 0.00731663 2024-09-14T03:53:40.298 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 56056 56040 134.683 88.0625 0.0044493 0.00741944 2024-09-14T03:53:40.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: pgmap v163: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-09-14T03:53:40.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='client.44171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:40.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:40.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:40.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:53:40.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:53:40.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:40.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:53:40.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:53:40.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:53:40.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:53:40.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-14T03:53:40.333 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: pgmap v163: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-09-14T03:53:40.333 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='client.44171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:40.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:40.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:40.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:53:40.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:53:40.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:40.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:53:40.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:53:40.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:53:40.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:53:40.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-14T03:53:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:53:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:53:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: pgmap v163: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-09-14T03:53:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='client.44171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:40.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:53:40.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:53:40.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:40.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:53:40.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:53:40.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:53:40.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:53:40.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-14T03:53:41.299 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 15 57552 57537 133.16 93.5625 0.0142947 0.00750437 2024-09-14T03:53:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:41 smithi114 ceph-mon[103029]: from='client.34287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:41 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-14T03:53:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:41 smithi114 ceph-mon[103029]: Upgrade: osd.1 is safe to restart 2024-09-14T03:53:41.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:41 smithi114 ceph-mon[106969]: from='client.34287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:41.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:41 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-14T03:53:41.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:41 smithi114 ceph-mon[106969]: Upgrade: osd.1 is safe to restart 2024-09-14T03:53:41.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:41 smithi160 ceph-mon[88407]: from='client.34287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:41.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:41 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-14T03:53:41.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:41 smithi160 ceph-mon[88407]: Upgrade: osd.1 is safe to restart 2024-09-14T03:53:41.819 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:53:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:53:41] "GET /metrics HTTP/1.1" 200 36302 "" "Prometheus/2.51.0" 2024-09-14T03:53:42.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:42 smithi114 ceph-mon[106969]: pgmap v164: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-14T03:53:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1606346335' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:42 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:42 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-14T03:53:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:42 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:53:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:42 smithi114 ceph-mon[103029]: pgmap v164: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-14T03:53:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1606346335' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:42 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:42 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-14T03:53:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:42 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:53:42.301 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 58826 58810 131.233 79.5625 0.00960055 0.00761418 2024-09-14T03:53:42.348 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:53:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:42 smithi160 ceph-mon[88407]: pgmap v164: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-14T03:53:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1606346335' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:42 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:42 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-14T03:53:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:42 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:53:43.301 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 60183 60167 129.633 84.8125 0.00865747 0.00770885 2024-09-14T03:53:43.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:43 smithi114 ceph-mon[103029]: Upgrade: Updating osd.1 2024-09-14T03:53:43.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:43 smithi114 ceph-mon[103029]: Deploying daemon osd.1 on smithi114 2024-09-14T03:53:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:43 smithi114 ceph-mon[106969]: Upgrade: Updating osd.1 2024-09-14T03:53:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:43 smithi114 ceph-mon[106969]: Deploying daemon osd.1 on smithi114 2024-09-14T03:53:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:43 smithi160 ceph-mon[88407]: Upgrade: Updating osd.1 2024-09-14T03:53:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:43 smithi160 ceph-mon[88407]: Deploying daemon osd.1 on smithi114 2024-09-14T03:53:44.301 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 61581 61565 128.224 87.375 0.00884787 0.00779372 2024-09-14T03:53:44.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:44 smithi114 ceph-mon[103029]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:44 smithi114 ceph-mon[103029]: pgmap v165: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-14T03:53:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:44 smithi114 ceph-mon[103029]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:44 smithi114 ceph-mon[106969]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:44 smithi114 ceph-mon[106969]: pgmap v165: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-14T03:53:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:44 smithi114 ceph-mon[106969]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:44 smithi160 ceph-mon[88407]: from='client.34299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:44 smithi160 ceph-mon[88407]: pgmap v165: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-14T03:53:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:44 smithi160 ceph-mon[88407]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:45.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 62899 62883 126.745 82.375 0.0184627 0.00788288 2024-09-14T03:53:46.022 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:53:46.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 64100 64084 125.13 75.0625 0.00535231 0.00798585 2024-09-14T03:53:46.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:46 smithi114 ceph-mon[103029]: pgmap v166: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-14T03:53:46.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4152746953' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:46.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:46 smithi114 ceph-mon[106969]: pgmap v166: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-14T03:53:46.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4152746953' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:46 smithi160 ceph-mon[88407]: pgmap v166: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-14T03:53:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4152746953' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:47.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 65350 65334 123.706 78.125 0.0158064 0.00807817 2024-09-14T03:53:47.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:47 smithi114 ceph-mon[103029]: from='client.54166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:47 smithi114 ceph-mon[106969]: from='client.54166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:47.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:46 smithi114 systemd[1]: Stopping Ceph osd.1 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:53:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:47 smithi160 ceph-mon[88407]: from='client.54166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:47.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[51524]: 2024-09-14T03:53:47.588+0000 7f89082cb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T03:53:47.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[51524]: 2024-09-14T03:53:47.588+0000 7f89082cb640 -1 osd.1 93 *** Got signal Terminated *** 2024-09-14T03:53:47.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:47 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[51524]: 2024-09-14T03:53:47.588+0000 7f89082cb640 -1 osd.1 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T03:53:48.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 66501 66485 122.183 71.9375 0.00249853 0.00813189 2024-09-14T03:53:48.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:48 smithi114 ceph-mon[106969]: pgmap v167: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-14T03:53:48.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:48 smithi114 ceph-mon[106969]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:48.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:48 smithi114 ceph-mon[106969]: osd.1 marked itself down and dead 2024-09-14T03:53:48.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:48 smithi114 ceph-mon[103029]: pgmap v167: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-14T03:53:48.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:48 smithi114 ceph-mon[103029]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:48.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:48 smithi114 ceph-mon[103029]: osd.1 marked itself down and dead 2024-09-14T03:53:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:48 smithi160 ceph-mon[88407]: pgmap v167: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-14T03:53:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:48 smithi160 ceph-mon[88407]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:48 smithi160 ceph-mon[88407]: osd.1 marked itself down and dead 2024-09-14T03:53:49.092 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:48 smithi114 podman[119973]: 2024-09-14 03:53:48.67458573 +0000 UTC m=+1.425093492 container died a709e75520db79265c6fc3cd0fe4c0d06170ff2d416144d7df6d10b9cda25bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=quincy-eb16b97, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T03:53:49.241 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:53:49.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 66824 66808 119.269 20.1875 0.00415363 0.00837845 2024-09-14T03:53:49.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:49 smithi114 ceph-mon[103029]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:53:49.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:49 smithi114 ceph-mon[103029]: osdmap e94: 8 total, 7 up, 8 in 2024-09-14T03:53:49.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:49 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2660924096' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:49.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:49 smithi114 ceph-mon[103029]: osdmap e95: 8 total, 7 up, 8 in 2024-09-14T03:53:49.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:49 smithi114 ceph-mon[106969]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:53:49.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:49 smithi114 ceph-mon[106969]: osdmap e94: 8 total, 7 up, 8 in 2024-09-14T03:53:49.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:49 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2660924096' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:49.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:49 smithi114 ceph-mon[106969]: osdmap e95: 8 total, 7 up, 8 in 2024-09-14T03:53:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:49 smithi160 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:53:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:49 smithi160 ceph-mon[88407]: osdmap e94: 8 total, 7 up, 8 in 2024-09-14T03:53:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:49 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2660924096' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:49 smithi160 ceph-mon[88407]: osdmap e95: 8 total, 7 up, 8 in 2024-09-14T03:53:50.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 68482 68466 118.834 103.625 0.0113263 0.00840948 2024-09-14T03:53:50.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:50 smithi114 ceph-mon[106969]: pgmap v169: 89 pgs: 11 stale+active+clean, 78 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-14T03:53:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:50 smithi114 ceph-mon[103029]: pgmap v169: 89 pgs: 11 stale+active+clean, 78 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-14T03:53:50.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:50 smithi114 podman[119973]: 2024-09-14 03:53:50.167458331 +0000 UTC m=+2.917966091 container cleanup a709e75520db79265c6fc3cd0fe4c0d06170ff2d416144d7df6d10b9cda25bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=quincy-eb16b97, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-14T03:53:50.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:50 smithi114 bash[119973]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1 2024-09-14T03:53:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:50 smithi160 ceph-mon[88407]: pgmap v169: 89 pgs: 11 stale+active+clean, 78 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-14T03:53:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:53:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:53:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:53:50.870 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:50 smithi114 podman[120014]: 2024-09-14 03:53:50.452984967 +0000 UTC m=+1.771927671 container remove a709e75520db79265c6fc3cd0fe4c0d06170ff2d416144d7df6d10b9cda25bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-14T03:53:51.302 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 70099 70083 118.354 101.062 0.00502021 0.00844458 2024-09-14T03:53:51.325 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:51 smithi114 ceph-mon[103029]: from='client.44204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:51.325 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:51 smithi114 ceph-mon[103029]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:51.325 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:51 smithi114 ceph-mon[106969]: from='client.44204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:51.325 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:51 smithi114 ceph-mon[106969]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:51.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:51 smithi114 podman[120145]: 2024-09-14 03:53:50.920789467 +0000 UTC m=+0.047826724 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:53:51.325 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:51 smithi114 podman[120145]: 2024-09-14 03:53:51.054065646 +0000 UTC m=+0.181102889 container create dd263f9d59d3de8fdbb70266bee1cfd6c82469f7ba95f183568b0165f4fcd5d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a278231, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-14T03:53:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:51 smithi160 ceph-mon[88407]: from='client.44204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:51 smithi160 ceph-mon[88407]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:51.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:53:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:53:51] "GET /metrics HTTP/1.1" 200 36302 "" "Prometheus/2.51.0" 2024-09-14T03:53:51.621 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:51 smithi114 podman[120145]: 2024-09-14 03:53:51.324225894 +0000 UTC m=+0.451263143 container init dd263f9d59d3de8fdbb70266bee1cfd6c82469f7ba95f183568b0165f4fcd5d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-a278231, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-14T03:53:51.621 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:51 smithi114 podman[120145]: 2024-09-14 03:53:51.331666978 +0000 UTC m=+0.458704227 container start dd263f9d59d3de8fdbb70266bee1cfd6c82469f7ba95f183568b0165f4fcd5d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-a278231, org.label-schema.vendor=CentOS) 2024-09-14T03:53:51.621 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:51 smithi114 podman[120145]: 2024-09-14 03:53:51.440987447 +0000 UTC m=+0.568024695 container attach dd263f9d59d3de8fdbb70266bee1cfd6c82469f7ba95f183568b0165f4fcd5d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-a278231, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T03:53:51.973 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:51 smithi114 podman[120145]: 2024-09-14 03:53:51.632087309 +0000 UTC m=+0.759124553 container died dd263f9d59d3de8fdbb70266bee1cfd6c82469f7ba95f183568b0165f4fcd5d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=main-a278231, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-14T03:53:52.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 71631 71615 117.759 95.75 0.0081065 0.0084871 2024-09-14T03:53:52.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:52 smithi114 ceph-mon[106969]: pgmap v171: 89 pgs: 32 peering, 1 stale+active+clean, 56 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-14T03:53:52.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:52 smithi114 ceph-mon[106969]: Health check failed: Reduced data availability: 9 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-14T03:53:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:52 smithi114 ceph-mon[103029]: pgmap v171: 89 pgs: 32 peering, 1 stale+active+clean, 56 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-14T03:53:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:52 smithi114 ceph-mon[103029]: Health check failed: Reduced data availability: 9 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-14T03:53:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:52 smithi160 ceph-mon[88407]: pgmap v171: 89 pgs: 32 peering, 1 stale+active+clean, 56 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-14T03:53:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:52 smithi160 ceph-mon[88407]: Health check failed: Reduced data availability: 9 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-14T03:53:52.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:52 smithi114 podman[120184]: 2024-09-14 03:53:52.396702543 +0000 UTC m=+0.756547860 container remove dd263f9d59d3de8fdbb70266bee1cfd6c82469f7ba95f183568b0165f4fcd5d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-a278231, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0) 2024-09-14T03:53:52.754 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:52 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-14T03:53:52.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:52 smithi114 systemd[1]: Stopped Ceph osd.1 for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:53:52.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:52 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.1.service: Consumed 11min 21.548s CPU time. 2024-09-14T03:53:52.755 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:52 smithi114 systemd[1]: Starting Ceph osd.1 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:53:52.906 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:53:53.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 72926 72910 116.815 80.9375 0.00739448 0.00855235 2024-09-14T03:53:53.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1656415296' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:53.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1656415296' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:53.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:53.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:53.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:53 smithi114 podman[120331]: 2024-09-14 03:53:53.008433076 +0000 UTC m=+0.034152352 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:53:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1656415296' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:53:54.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:53:54.302433+0000 min lat: 0.00128604 max lat: 1.0368 avg lat: 0.00860272 2024-09-14T03:53:54.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:53:54.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 74355 74339 116.127 89.3125 0.0276604 0.00860272 2024-09-14T03:53:54.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:54 smithi114 ceph-mon[103029]: pgmap v172: 89 pgs: 3 active+undersized, 32 peering, 3 active+undersized+degraded, 51 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 58 MiB/s wr, 933 op/s; 8695/201684 objects degraded (4.311%); 8.0 KiB/s, 0 objects/s recovering 2024-09-14T03:53:54.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:54 smithi114 ceph-mon[103029]: from='client.44213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:54.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:54 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 8695/201684 objects degraded (4.311%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:53:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:54 smithi114 ceph-mon[106969]: pgmap v172: 89 pgs: 3 active+undersized, 32 peering, 3 active+undersized+degraded, 51 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 58 MiB/s wr, 933 op/s; 8695/201684 objects degraded (4.311%); 8.0 KiB/s, 0 objects/s recovering 2024-09-14T03:53:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:54 smithi114 ceph-mon[106969]: from='client.44213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:54 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 8695/201684 objects degraded (4.311%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:53:54.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:54 smithi114 podman[120331]: 2024-09-14 03:53:54.057690012 +0000 UTC m=+1.083409283 container create f1c758c8f5fe6e86e6a4d41b161c512dd147bbefa4c620de0ffd58badc0bfa6c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate, RELEASE=main-a278231, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T03:53:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:54 smithi160 ceph-mon[88407]: pgmap v172: 89 pgs: 3 active+undersized, 32 peering, 3 active+undersized+degraded, 51 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 58 MiB/s wr, 933 op/s; 8695/201684 objects degraded (4.311%); 8.0 KiB/s, 0 objects/s recovering 2024-09-14T03:53:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:54 smithi160 ceph-mon[88407]: from='client.44213 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:54 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 8695/201684 objects degraded (4.311%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T03:53:54.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:54 smithi114 podman[120331]: 2024-09-14 03:53:54.677620177 +0000 UTC m=+1.703339453 container init f1c758c8f5fe6e86e6a4d41b161c512dd147bbefa4c620de0ffd58badc0bfa6c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-a278231) 2024-09-14T03:53:54.787 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:54 smithi114 podman[120331]: 2024-09-14 03:53:54.68553825 +0000 UTC m=+1.711257521 container start f1c758c8f5fe6e86e6a4d41b161c512dd147bbefa4c620de0ffd58badc0bfa6c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=main-a278231, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-14T03:53:55.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:54 smithi114 podman[120331]: 2024-09-14 03:53:54.78576096 +0000 UTC m=+1.811480225 container attach f1c758c8f5fe6e86e6a4d41b161c512dd147bbefa4c620de0ffd58badc0bfa6c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-a278231, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-14T03:53:55.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate[120367]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:53:55.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:54 smithi114 bash[120331]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:53:55.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate[120367]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:53:55.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:54 smithi114 bash[120331]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:53:55.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 75910 75894 115.665 97.1875 0.004428 0.0086406 2024-09-14T03:53:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:55 smithi160 ceph-mon[88407]: from='client.54190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:55 smithi160 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 9 pgs peering) 2024-09-14T03:53:55.554 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-mon[106969]: from='client.54190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:55.554 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-mon[106969]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 9 pgs peering) 2024-09-14T03:53:55.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-mon[103029]: from='client.54190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:55.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-mon[103029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 9 pgs peering) 2024-09-14T03:53:55.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate[120367]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-14T03:53:55.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 bash[120331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-14T03:53:55.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate[120367]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-14T03:53:55.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 bash[120331]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-14T03:53:55.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate[120367]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-14T03:53:55.871 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 bash[120331]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-14T03:53:55.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate[120367]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-14T03:53:55.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 bash[120331]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-14T03:53:55.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate[120367]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-14T03:53:55.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 bash[120331]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-14T03:53:55.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate[120367]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-14T03:53:55.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 bash[120331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-14T03:53:55.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate[120367]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-14T03:53:55.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 bash[120331]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-14T03:53:55.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 conmon[120367]: conmon f1c758c8f5fe6e86e6a4 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-f1c758c8f5fe6e86e6a4d41b161c512dd147bbefa4c620de0ffd58badc0bfa6c.scope/memory.events 2024-09-14T03:53:55.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:55 smithi114 podman[120331]: 2024-09-14 03:53:55.656519057 +0000 UTC m=+2.682238338 container died f1c758c8f5fe6e86e6a4d41b161c512dd147bbefa4c620de0ffd58badc0bfa6c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-a278231, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-14T03:53:56.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 77278 77262 114.947 85.5 0.0115654 0.00869499 2024-09-14T03:53:56.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:56 smithi114 ceph-mon[106969]: pgmap v173: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 27145/217008 objects degraded (12.509%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:53:56.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:56 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3352022642' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:56.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:56 smithi114 ceph-mon[103029]: pgmap v173: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 27145/217008 objects degraded (12.509%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:53:56.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:56 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3352022642' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:56 smithi160 ceph-mon[88407]: pgmap v173: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 27145/217008 objects degraded (12.509%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:53:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:56 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3352022642' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:53:56.448 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:53:56.786 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:56 smithi114 podman[120557]: 2024-09-14 03:53:56.458982567 +0000 UTC m=+0.784849217 container remove f1c758c8f5fe6e86e6a4d41b161c512dd147bbefa4c620de0ffd58badc0bfa6c (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a278231, io.buildah.version=1.37.2) 2024-09-14T03:53:57.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:56 smithi114 podman[120664]: 2024-09-14 03:53:56.691584203 +0000 UTC m=+0.031024089 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:53:57.121 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:56 smithi114 podman[120664]: 2024-09-14 03:53:56.81017516 +0000 UTC m=+0.149615030 container create 724ed57646205c415e6778611e1f4e840bb9f432c21b7b8d95affedd2872f18e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=main-a278231, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-09-14T03:53:57.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 78643 78627 114.258 85.3125 0.0136285 0.00874752 2024-09-14T03:53:57.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:57 smithi114 podman[120664]: 2024-09-14 03:53:57.271992342 +0000 UTC m=+0.611432222 container init 724ed57646205c415e6778611e1f4e840bb9f432c21b7b8d95affedd2872f18e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=main-a278231) 2024-09-14T03:53:57.389 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:57 smithi114 podman[120664]: 2024-09-14 03:53:57.282728762 +0000 UTC m=+0.622168630 container start 724ed57646205c415e6778611e1f4e840bb9f432c21b7b8d95affedd2872f18e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a278231, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:53:57.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:57 smithi114 bash[120664]: 724ed57646205c415e6778611e1f4e840bb9f432c21b7b8d95affedd2872f18e 2024-09-14T03:53:57.872 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:57 smithi114 systemd[1]: Started Ceph osd.1 for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:53:58.303 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 79820 79804 113.333 73.5625 0.00331704 0.0088184 2024-09-14T03:53:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:58 smithi160 ceph-mon[88407]: pgmap v174: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s; 28078/222549 objects degraded (12.617%); 15 KiB/s, 0 objects/s recovering 2024-09-14T03:53:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:58 smithi160 ceph-mon[88407]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:58 smithi160 ceph-mon[88407]: from='client.44231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:58 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:58 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:58 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:53:58.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[106969]: pgmap v174: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s; 28078/222549 objects degraded (12.617%); 15 KiB/s, 0 objects/s recovering 2024-09-14T03:53:58.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[106969]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:58.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[106969]: from='client.44231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:58.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:58.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:58.422 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:53:58.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[103029]: pgmap v174: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 96 MiB/s wr, 1.53k op/s; 28078/222549 objects degraded (12.617%); 15 KiB/s, 0 objects/s recovering 2024-09-14T03:53:58.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[103029]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:58.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[103029]: from='client.44231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:53:58.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:58.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:53:58.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:53:59.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 81007 80991 112.46 74.1875 0.0137001 0.00888745 2024-09-14T03:53:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:53:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 28078/222549 objects degraded (12.617%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T03:53:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:53:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 28078/222549 objects degraded (12.617%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T03:53:59.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:53:58 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[120675]: 2024-09-14T03:53:58.944+0000 7f86e2892640 -1 Falling back to public interface 2024-09-14T03:53:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:53:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 28078/222549 objects degraded (12.617%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:00.054 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:00.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 82272 82256 111.734 79.0625 0.0224855 0.00894421 2024-09-14T03:54:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:00 smithi114 ceph-mon[103029]: pgmap v175: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s; 28078/222549 objects degraded (12.617%); 13 KiB/s, 0 objects/s recovering 2024-09-14T03:54:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4157434674' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:00 smithi114 ceph-mon[106969]: pgmap v175: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s; 28078/222549 objects degraded (12.617%); 13 KiB/s, 0 objects/s recovering 2024-09-14T03:54:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4157434674' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:00 smithi160 ceph-mon[88407]: pgmap v175: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s; 28078/222549 objects degraded (12.617%); 13 KiB/s, 0 objects/s recovering 2024-09-14T03:54:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4157434674' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:00.559 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:54:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:54:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:54:01.186 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:01 smithi114 ceph-mon[106969]: from='client.44240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:01.187 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:01 smithi114 ceph-mon[103029]: from='client.44240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:01.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 83657 83641 111.199 86.5625 0.00454414 0.00898815 2024-09-14T03:54:01.569 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:01 smithi160 ceph-mon[88407]: from='client.44240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:01.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:54:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:54:01] "GET /metrics HTTP/1.1" 200 36387 "" "Prometheus/2.51.0" 2024-09-14T03:54:02.246 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:02 smithi160 ceph-mon[88407]: pgmap v176: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 29789/237924 objects degraded (12.520%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:54:02.247 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:02 smithi160 ceph-mon[88407]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:02.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 85029 85013 110.668 85.75 0.0171575 0.00903082 2024-09-14T03:54:02.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:02 smithi114 ceph-mon[106969]: pgmap v176: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 29789/237924 objects degraded (12.520%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:54:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:02 smithi114 ceph-mon[106969]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:02 smithi114 ceph-mon[103029]: pgmap v176: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 29789/237924 objects degraded (12.520%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:54:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:02 smithi114 ceph-mon[103029]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:03.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 86013 85997 109.665 61.5 0.00735741 0.00911221 2024-09-14T03:54:03.650 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:03 smithi160 ceph-mon[88407]: pgmap v177: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 30599/242778 objects degraded (12.604%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:54:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:03 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1102255271' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:03.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:03 smithi114 ceph-mon[106969]: pgmap v177: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 30599/242778 objects degraded (12.604%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:54:03.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:03.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:03.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:03 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1102255271' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:03.876 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:03 smithi114 ceph-mon[103029]: pgmap v177: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 30599/242778 objects degraded (12.604%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:54:03.876 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:03.876 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:03.876 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:03 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1102255271' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:04.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 87020 87004 108.73 62.9375 0.00472366 0.00919186 2024-09-14T03:54:04.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:04 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:04 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:04 smithi114 ceph-mon[106969]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 30599/242778 objects degraded (12.604%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:04.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:04 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:04.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:04 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:04.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:04 smithi114 ceph-mon[103029]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:04.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 30599/242778 objects degraded (12.604%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:04 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:04 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:04 smithi160 ceph-mon[88407]: from='client.34395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 30599/242778 objects degraded (12.604%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:05.304 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 88306 88290 108.174 80.375 0.00968953 0.00923924 2024-09-14T03:54:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:05 smithi114 ceph-mon[103029]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:05 smithi114 ceph-mon[103029]: pgmap v178: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s; 32134/256494 objects degraded (12.528%); 5.3 KiB/s, 0 objects/s recovering 2024-09-14T03:54:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:05 smithi114 ceph-mon[106969]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:05 smithi114 ceph-mon[106969]: pgmap v178: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s; 32134/256494 objects degraded (12.528%); 5.3 KiB/s, 0 objects/s recovering 2024-09-14T03:54:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:05 smithi160 ceph-mon[88407]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:05 smithi160 ceph-mon[88407]: pgmap v178: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s; 32134/256494 objects degraded (12.528%); 5.3 KiB/s, 0 objects/s recovering 2024-09-14T03:54:06.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 89444 89428 107.462 71.125 0.011951 0.0092991 2024-09-14T03:54:07.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 90426 90410 106.592 61.375 0.00727313 0.00937618 2024-09-14T03:54:07.361 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:07.362 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:54:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[120675]: 2024-09-14T03:54:06.991+0000 7f86e2892640 -1 osd.1 93 log_to_monitors true 2024-09-14T03:54:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:07 smithi114 ceph-mon[106969]: pgmap v179: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s; 32880/261111 objects degraded (12.592%) 2024-09-14T03:54:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:07 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/88636530' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:07 smithi114 ceph-mon[106969]: from='osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-14T03:54:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:07 smithi114 ceph-mon[103029]: pgmap v179: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s; 32880/261111 objects degraded (12.592%) 2024-09-14T03:54:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:07 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/88636530' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:07.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:07 smithi114 ceph-mon[103029]: from='osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-14T03:54:07.622 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 03:54:07 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[120675]: 2024-09-14T03:54:07.579+0000 7f86da63c640 -1 osd.1 93 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:54:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:07 smithi160 ceph-mon[88407]: pgmap v179: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s; 32880/261111 objects degraded (12.592%) 2024-09-14T03:54:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:07 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/88636530' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:07 smithi160 ceph-mon[88407]: from='osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-14T03:54:08.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 91661 91645 106.048 77.1875 0.0123101 0.00942473 2024-09-14T03:54:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[106969]: from='client.44267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[106969]: from='osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-14T03:54:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[106969]: osdmap e96: 8 total, 7 up, 8 in 2024-09-14T03:54:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[106969]: from='osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:54:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[106969]: from='client.34413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[103029]: from='client.44267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[103029]: from='osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-14T03:54:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[103029]: osdmap e96: 8 total, 7 up, 8 in 2024-09-14T03:54:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[103029]: from='osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:54:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:08 smithi114 ceph-mon[103029]: from='client.34413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:08 smithi160 ceph-mon[88407]: from='client.44267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:08 smithi160 ceph-mon[88407]: from='osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-14T03:54:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:08 smithi160 ceph-mon[88407]: osdmap e96: 8 total, 7 up, 8 in 2024-09-14T03:54:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:08 smithi160 ceph-mon[88407]: from='osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:54:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:08.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:08 smithi160 ceph-mon[88407]: from='client.34413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:09.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 92012 91996 104.519 21.9375 0.00374691 0.00943748 2024-09-14T03:54:09.547 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:54:09.424+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-14T03:54:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: pgmap v181: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 32880/261111 objects degraded (12.592%) 2024-09-14T03:54:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:54:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706] boot 2024-09-14T03:54:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: osdmap e97: 8 total, 8 up, 8 in 2024-09-14T03:54:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-14T03:54:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 32880/261111 objects degraded (12.592%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: osdmap e98: 8 total, 8 up, 8 in 2024-09-14T03:54:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:54:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:54:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:54:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:54:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:54:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:54:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:09.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: pgmap v181: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 32880/261111 objects degraded (12.592%) 2024-09-14T03:54:09.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:54:09.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706] boot 2024-09-14T03:54:09.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: osdmap e97: 8 total, 8 up, 8 in 2024-09-14T03:54:09.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-14T03:54:09.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 32880/261111 objects degraded (12.592%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:09.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: osdmap e98: 8 total, 8 up, 8 in 2024-09-14T03:54:09.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:09.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:09.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:54:09.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:54:09.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:09.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:54:09.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:54:09.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:54:09.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:54:09.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: pgmap v181: 89 pgs: 28 active+undersized, 10 active+undersized+degraded, 51 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 32880/261111 objects degraded (12.592%) 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: osd.1 [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706] boot 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: osdmap e97: 8 total, 8 up, 8 in 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 32880/261111 objects degraded (12.592%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: osdmap e98: 8 total, 8 up, 8 in 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:54:09.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:09.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:54:09.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:54:09.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:54:09.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:54:09.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:10.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 92012 91996 102.652 0 - 0.00943748 2024-09-14T03:54:10.618 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:54:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:54:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:54:11.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 92805 92789 101.721 24.7812 0.0145077 0.00982555 2024-09-14T03:54:11.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:11 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:11.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:11 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-14T03:54:11.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:11 smithi114 ceph-mon[106969]: osdmap e99: 8 total, 8 up, 8 in 2024-09-14T03:54:11.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:11 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/732189368' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:11.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:11 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:11.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:11 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-14T03:54:11.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:11 smithi114 ceph-mon[103029]: osdmap e99: 8 total, 8 up, 8 in 2024-09-14T03:54:11.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:11 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/732189368' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:11 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:11 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-14T03:54:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:11 smithi160 ceph-mon[88407]: osdmap e99: 8 total, 8 up, 8 in 2024-09-14T03:54:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:11 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/732189368' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:11.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:54:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:54:11] "GET /metrics HTTP/1.1" 200 36224 "" "Prometheus/2.51.0" 2024-09-14T03:54:12.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 93785 93769 101.023 61.25 0.0123334 0.00989331 2024-09-14T03:54:12.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:12 smithi114 ceph-mon[103029]: pgmap v185: 89 pgs: 1 remapped+peering, 5 activating, 2 peering, 21 active+undersized, 9 active+undersized+degraded, 51 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 28438/273600 objects degraded (10.394%) 2024-09-14T03:54:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:12 smithi114 ceph-mon[103029]: from='client.44285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:12 smithi114 ceph-mon[103029]: from='client.54244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:12 smithi114 ceph-mon[106969]: pgmap v185: 89 pgs: 1 remapped+peering, 5 activating, 2 peering, 21 active+undersized, 9 active+undersized+degraded, 51 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 28438/273600 objects degraded (10.394%) 2024-09-14T03:54:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:12 smithi114 ceph-mon[106969]: from='client.44285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:12 smithi114 ceph-mon[106969]: from='client.54244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:12 smithi160 ceph-mon[88407]: pgmap v185: 89 pgs: 1 remapped+peering, 5 activating, 2 peering, 21 active+undersized, 9 active+undersized+degraded, 51 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 28438/273600 objects degraded (10.394%) 2024-09-14T03:54:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:12 smithi160 ceph-mon[88407]: from='client.44285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:12 smithi160 ceph-mon[88407]: from='client.54244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:13.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 94756 94740 100.339 60.6875 0.0285819 0.00995999 2024-09-14T03:54:14.288 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:14.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:54:14.305208+0000 min lat: 0.00128604 max lat: 1.89669 avg lat: 0.0100402 2024-09-14T03:54:14.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:54:14.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 95596 95580 99.5422 52.5 0.00946912 0.0100402 2024-09-14T03:54:14.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:14 smithi114 ceph-mon[106969]: pgmap v186: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 5 activating, 2 peering, 1 active+recovering+undersized+degraded+remapped, 19 active+undersized, 6 active+undersized+degraded, 54 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 20455/276129 objects degraded (7.408%); 251 KiB/s, 3 objects/s recovering 2024-09-14T03:54:14.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:14 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3796884738' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:14.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 20455/276129 objects degraded (7.408%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:14 smithi114 ceph-mon[103029]: pgmap v186: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 5 activating, 2 peering, 1 active+recovering+undersized+degraded+remapped, 19 active+undersized, 6 active+undersized+degraded, 54 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 20455/276129 objects degraded (7.408%); 251 KiB/s, 3 objects/s recovering 2024-09-14T03:54:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:14 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3796884738' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 20455/276129 objects degraded (7.408%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:14 smithi160 ceph-mon[88407]: pgmap v186: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 5 activating, 2 peering, 1 active+recovering+undersized+degraded+remapped, 19 active+undersized, 6 active+undersized+degraded, 54 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 20455/276129 objects degraded (7.408%); 251 KiB/s, 3 objects/s recovering 2024-09-14T03:54:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:14 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3796884738' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 20455/276129 objects degraded (7.408%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:15.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 96504 96488 98.8406 56.75 0.0169397 0.0101116 2024-09-14T03:54:16.305 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 97477 97461 98.2273 60.8125 0.0189827 0.0101749 2024-09-14T03:54:16.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:16 smithi114 ceph-mon[106969]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:16.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:16 smithi114 ceph-mon[106969]: pgmap v187: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 9550/283233 objects degraded (3.372%); 213 KiB/s, 3 objects/s recovering 2024-09-14T03:54:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:16 smithi114 ceph-mon[106969]: from='client.54250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:16 smithi114 ceph-mon[103029]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:16 smithi114 ceph-mon[103029]: pgmap v187: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 9550/283233 objects degraded (3.372%); 213 KiB/s, 3 objects/s recovering 2024-09-14T03:54:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:16 smithi114 ceph-mon[103029]: from='client.54250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:16 smithi160 ceph-mon[88407]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:16 smithi160 ceph-mon[88407]: pgmap v187: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 9550/283233 objects degraded (3.372%); 213 KiB/s, 3 objects/s recovering 2024-09-14T03:54:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:16 smithi160 ceph-mon[88407]: from='client.54250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:17.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 98379 98363 97.5629 56.375 0.0191093 0.0102445 2024-09-14T03:54:17.930 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:18.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 99201 99185 96.8412 51.375 0.0290096 0.0103202 2024-09-14T03:54:18.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:18 smithi114 ceph-mon[106969]: pgmap v188: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s; 9364/286857 objects degraded (3.264%); 1.6 MiB/s, 26 objects/s recovering 2024-09-14T03:54:18.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:18 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3905921036' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:18 smithi114 ceph-mon[103029]: pgmap v188: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s; 9364/286857 objects degraded (3.264%); 1.6 MiB/s, 26 objects/s recovering 2024-09-14T03:54:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:18 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3905921036' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:18 smithi160 ceph-mon[88407]: pgmap v188: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s; 9364/286857 objects degraded (3.264%); 1.6 MiB/s, 26 objects/s recovering 2024-09-14T03:54:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:18 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3905921036' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:19.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 100141 100125 96.2552 58.75 0.00930619 0.0103832 2024-09-14T03:54:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:19 smithi160 ceph-mon[88407]: from='client.54259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9364/286857 objects degraded (3.264%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:19.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:19 smithi114 ceph-mon[106969]: from='client.54259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:19.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9364/286857 objects degraded (3.264%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:19.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:19 smithi114 ceph-mon[103029]: from='client.54259 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:19.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9364/286857 objects degraded (3.264%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:20.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 101072 101056 95.6784 58.1875 0.0105948 0.0104466 2024-09-14T03:54:20.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:54:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:54:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:54:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:20 smithi160 ceph-mon[88407]: pgmap v189: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 36 MiB/s wr, 579 op/s; 9364/286857 objects degraded (3.264%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T03:54:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:20 smithi160 ceph-mon[88407]: from='client.54262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:20 smithi114 ceph-mon[106969]: pgmap v189: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 36 MiB/s wr, 579 op/s; 9364/286857 objects degraded (3.264%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T03:54:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:20 smithi114 ceph-mon[106969]: from='client.54262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:20 smithi114 ceph-mon[103029]: pgmap v189: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 36 MiB/s wr, 579 op/s; 9364/286857 objects degraded (3.264%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T03:54:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:20 smithi114 ceph-mon[103029]: from='client.54262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:21.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 101918 101902 95.0396 52.875 0.0234123 0.0105156 2024-09-14T03:54:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2976232020' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:21.432 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2976232020' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:21.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2976232020' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:21.594 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:54:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:54:21] "GET /metrics HTTP/1.1" 200 36224 "" "Prometheus/2.51.0" 2024-09-14T03:54:22.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 102786 102770 94.4397 54.25 0.0283722 0.0105834 2024-09-14T03:54:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:22 smithi160 ceph-mon[88407]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 751 op/s; 9364/296844 objects degraded (3.155%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T03:54:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:22 smithi114 ceph-mon[106969]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 751 op/s; 9364/296844 objects degraded (3.155%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T03:54:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:22 smithi114 ceph-mon[103029]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 751 op/s; 9364/296844 objects degraded (3.155%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T03:54:23.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 103704 103688 93.9025 57.375 0.00458436 0.0106438 2024-09-14T03:54:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:23 smithi160 ceph-mon[88407]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:23 smithi160 ceph-mon[88407]: from='client.44321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:23 smithi114 ceph-mon[106969]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:23.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:23 smithi114 ceph-mon[106969]: from='client.44321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:23.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:23.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:23 smithi114 ceph-mon[103029]: from='client.44315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:23 smithi114 ceph-mon[103029]: from='client.44321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:54:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:24.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 104584 104568 93.3468 55 0.041989 0.0107079 2024-09-14T03:54:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:24 smithi160 ceph-mon[88407]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 46 MiB/s wr, 739 op/s; 9154/300246 objects degraded (3.049%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9154/300246 objects degraded (3.049%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:24.443 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:24 smithi114 ceph-mon[106969]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 46 MiB/s wr, 739 op/s; 9154/300246 objects degraded (3.049%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:24.443 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9154/300246 objects degraded (3.049%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:24.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:24 smithi114 ceph-mon[103029]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 46 MiB/s wr, 739 op/s; 9154/300246 objects degraded (3.049%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:24.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9154/300246 objects degraded (3.049%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:24.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:54:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:54:24.441+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:25.237 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:25.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 105368 105352 92.7222 49 0.0138847 0.010779 2024-09-14T03:54:25.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:25 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:25.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4205609412' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:25 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4205609412' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:25 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4205609412' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:26.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 106232 106216 92.1843 54 0.0158908 0.0108422 2024-09-14T03:54:26.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:26 smithi114 ceph-mon[106969]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 59 MiB/s wr, 950 op/s; 9154/310341 objects degraded (2.950%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:54:26.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:26 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:26.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:26 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:26.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:26 smithi114 ceph-mon[103029]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 59 MiB/s wr, 950 op/s; 9154/310341 objects degraded (2.950%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:54:26.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:26 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:26.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:26 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:26 smithi160 ceph-mon[88407]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 59 MiB/s wr, 950 op/s; 9154/310341 objects degraded (2.950%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:54:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:26 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:26 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:27.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 107151 107135 91.7083 57.4375 0.0205456 0.0108984 2024-09-14T03:54:27.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:27 smithi114 ceph-mon[106969]: from='client.54280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:27.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:27 smithi114 ceph-mon[106969]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:27.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:27 smithi114 ceph-mon[103029]: from='client.54280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:27.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:27 smithi114 ceph-mon[103029]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:27 smithi160 ceph-mon[88407]: from='client.54280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:27 smithi160 ceph-mon[88407]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:28.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:28 smithi114 ceph-mon[103029]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 52 MiB/s wr, 839 op/s; 8949/313461 objects degraded (2.855%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T03:54:28.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:28 smithi114 ceph-mon[106969]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 52 MiB/s wr, 839 op/s; 8949/313461 objects degraded (2.855%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T03:54:28.306 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 107911 107895 91.1109 47.5 0.015987 0.0109695 2024-09-14T03:54:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:28 smithi160 ceph-mon[88407]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 52 MiB/s wr, 839 op/s; 8949/313461 objects degraded (2.855%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T03:54:28.890 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:29.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 108580 108564 90.4536 41.8125 0.0201688 0.0110498 2024-09-14T03:54:29.563 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:29 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3588531799' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:29.563 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:29 smithi114 ceph-mon[106969]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 738 op/s; 8949/313461 objects degraded (2.855%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:29.563 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:29 smithi114 ceph-mon[106969]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:29.563 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8949/313461 objects degraded (2.855%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:29.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:29 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3588531799' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:29.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:29 smithi114 ceph-mon[103029]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 738 op/s; 8949/313461 objects degraded (2.855%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:29.564 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:29 smithi114 ceph-mon[103029]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:29.564 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8949/313461 objects degraded (2.855%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:29 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3588531799' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:29 smithi160 ceph-mon[88407]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 738 op/s; 8949/313461 objects degraded (2.855%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:29 smithi160 ceph-mon[88407]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8949/313461 objects degraded (2.855%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:30.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 109434 109418 89.9657 53.375 0.0140272 0.0111099 2024-09-14T03:54:30.558 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:54:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:54:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:54:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:30 smithi160 ceph-mon[88407]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:30 smithi114 ceph-mon[106969]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:30 smithi114 ceph-mon[103029]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:31.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 110343 110327 89.5351 56.8125 0.0149539 0.011163 2024-09-14T03:54:31.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:31 smithi114 ceph-mon[106969]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1k op/s; 8949/322893 objects degraded (2.772%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:31.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:54:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:54:31] "GET /metrics HTTP/1.1" 200 36393 "" "Prometheus/2.51.0" 2024-09-14T03:54:31.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:31 smithi114 ceph-mon[103029]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1k op/s; 8949/322893 objects degraded (2.772%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:31.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:31 smithi160 ceph-mon[88407]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1k op/s; 8949/322893 objects degraded (2.772%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:32.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 111141 111125 89.0266 49.875 0.0178655 0.0112272 2024-09-14T03:54:32.548 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:32.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:32 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1971809895' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:32.549 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:32 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1971809895' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:32 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1971809895' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:33.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 111919 111903 88.5152 48.625 0.0201617 0.0112915 2024-09-14T03:54:33.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:33 smithi114 ceph-mon[106969]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 51 MiB/s wr, 809 op/s; 8738/325998 objects degraded (2.680%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:54:33.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:33 smithi114 ceph-mon[106969]: from='client.54307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:33 smithi114 ceph-mon[106969]: from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:33.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:33 smithi114 ceph-mon[103029]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 51 MiB/s wr, 809 op/s; 8738/325998 objects degraded (2.680%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:54:33.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:33 smithi114 ceph-mon[103029]: from='client.54307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:33.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:33 smithi114 ceph-mon[103029]: from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:33 smithi160 ceph-mon[88407]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 51 MiB/s wr, 809 op/s; 8738/325998 objects degraded (2.680%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:54:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:33 smithi160 ceph-mon[88407]: from='client.54307 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:33 smithi160 ceph-mon[88407]: from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:34.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:54:34.306996+0000 min lat: 0.00128604 max lat: 1.89669 avg lat: 0.0113448 2024-09-14T03:54:34.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:54:34.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 112805 112789 88.1009 55.375 0.0139666 0.0113448 2024-09-14T03:54:34.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8738/325998 objects degraded (2.680%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:34.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8738/325998 objects degraded (2.680%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8738/325998 objects degraded (2.680%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:35.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 113664 113648 87.676 53.6875 0.0184998 0.0113999 2024-09-14T03:54:35.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:35 smithi114 ceph-mon[106969]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s; 8738/335172 objects degraded (2.607%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:35 smithi114 ceph-mon[103029]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s; 8738/335172 objects degraded (2.607%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:35 smithi160 ceph-mon[88407]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s; 8738/335172 objects degraded (2.607%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:36.230 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:36.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 114431 114415 87.1914 47.9375 0.0276738 0.0114631 2024-09-14T03:54:36.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:36 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1307058794' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:36.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:36 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1307058794' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:36 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1307058794' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:37.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 115234 115218 86.7456 50.1875 0.0147955 0.0115225 2024-09-14T03:54:37.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:37 smithi114 ceph-mon[106969]: from='client.44372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:37.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:37 smithi114 ceph-mon[106969]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 48 MiB/s wr, 773 op/s; 8501/338226 objects degraded (2.513%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T03:54:37.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:37 smithi114 ceph-mon[106969]: from='client.54328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:37.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:37 smithi114 ceph-mon[103029]: from='client.44372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:37.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:37 smithi114 ceph-mon[103029]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 48 MiB/s wr, 773 op/s; 8501/338226 objects degraded (2.513%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T03:54:37.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:37 smithi114 ceph-mon[103029]: from='client.54328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:37 smithi160 ceph-mon[88407]: from='client.44372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:37 smithi160 ceph-mon[88407]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 48 MiB/s wr, 773 op/s; 8501/338226 objects degraded (2.513%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T03:54:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:37 smithi160 ceph-mon[88407]: from='client.54328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:38.307 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 116110 116094 86.3646 54.75 0.0087916 0.0115734 2024-09-14T03:54:38.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:38.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:39.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 116904 116888 85.9324 49.625 0.0186349 0.0116309 2024-09-14T03:54:39.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:39 smithi114 ceph-mon[106969]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s; 8501/340575 objects degraded (2.496%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T03:54:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:39 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8501/338226 objects degraded (2.513%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3489489785' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:39.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:54:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:54:39.458+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:39.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:39 smithi114 ceph-mon[103029]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s; 8501/340575 objects degraded (2.496%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T03:54:39.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:39 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8501/338226 objects degraded (2.513%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:39.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3489489785' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:39 smithi160 ceph-mon[88407]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s; 8501/340575 objects degraded (2.496%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T03:54:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:39 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8501/338226 objects degraded (2.513%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3489489785' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:39.876 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:40.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 117647 117631 85.4731 46.4375 0.0214404 0.0116935 2024-09-14T03:54:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:40 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:40 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:40 smithi114 ceph-mon[103029]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:40 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:40 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:40 smithi114 ceph-mon[106969]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:54:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:54:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:54:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:40 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:40 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:40 smithi160 ceph-mon[88407]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:41.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 118511 118495 85.1113 54 0.0246487 0.0117434 2024-09-14T03:54:41.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:41 smithi114 ceph-mon[106969]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 948 op/s; 8501/347634 objects degraded (2.445%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T03:54:41.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:41 smithi114 ceph-mon[106969]: from='client.54340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:41.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:54:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:54:41] "GET /metrics HTTP/1.1" 200 36395 "" "Prometheus/2.51.0" 2024-09-14T03:54:41.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:41 smithi114 ceph-mon[103029]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 948 op/s; 8501/347634 objects degraded (2.445%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T03:54:41.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:41 smithi114 ceph-mon[103029]: from='client.54340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:41 smithi160 ceph-mon[88407]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 948 op/s; 8501/347634 objects degraded (2.445%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T03:54:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:41 smithi160 ceph-mon[88407]: from='client.54340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:42.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 119353 119337 84.7421 52.625 0.0149612 0.0117949 2024-09-14T03:54:43.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 120120 120104 84.3286 47.9375 0.0314511 0.0118527 2024-09-14T03:54:43.503 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:43.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:43 smithi114 ceph-mon[103029]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 48 MiB/s wr, 768 op/s; 8234/350562 objects degraded (2.349%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:54:43.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:43 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3523978935' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:43.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:43 smithi114 ceph-mon[106969]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 48 MiB/s wr, 768 op/s; 8234/350562 objects degraded (2.349%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:54:43.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:43 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3523978935' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:43 smithi160 ceph-mon[88407]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 48 MiB/s wr, 768 op/s; 8234/350562 objects degraded (2.349%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:54:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:43 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3523978935' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:44.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0096 2024-09-14T03:54:44.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 120862 2024-09-14T03:54:44.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-14T03:54:44.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-14T03:54:44.308 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 83.923 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 44.457 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 246.375 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 0 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1342 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 711.32 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3942 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 0 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0119112 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0256064 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 1.89669 2024-09-14T03:54:44.309 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00128604 2024-09-14T03:54:44.315 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-14T03:54:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:44 smithi114 ceph-mon[103029]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:44 smithi114 ceph-mon[103029]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:44 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8234/350562 objects degraded (2.349%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:44 smithi114 ceph-mon[106969]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:44 smithi114 ceph-mon[106969]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:44 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8234/350562 objects degraded (2.349%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:44 smithi160 ceph-mon[88407]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:44 smithi160 ceph-mon[88407]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:44 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8234/350562 objects degraded (2.349%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:45.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:45 smithi114 ceph-mon[106969]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 58 MiB/s wr, 933 op/s; 8234/359613 objects degraded (2.290%); 2.6 MiB/s, 41 objects/s recovering 2024-09-14T03:54:45.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:45 smithi114 ceph-mon[103029]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 58 MiB/s wr, 933 op/s; 8234/359613 objects degraded (2.290%); 2.6 MiB/s, 41 objects/s recovering 2024-09-14T03:54:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:45 smithi160 ceph-mon[88407]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 58 MiB/s wr, 933 op/s; 8234/359613 objects degraded (2.290%); 2.6 MiB/s, 41 objects/s recovering 2024-09-14T03:54:46.791 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:46.791 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:46 smithi114 ceph-mon[103029]: osdmap e100: 8 total, 8 up, 8 in 2024-09-14T03:54:46.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/348569125' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:46.792 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:46 smithi114 ceph-mon[106969]: osdmap e100: 8 total, 8 up, 8 in 2024-09-14T03:54:46.792 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/348569125' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:46 smithi160 ceph-mon[88407]: osdmap e100: 8 total, 8 up, 8 in 2024-09-14T03:54:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/348569125' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:47.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:47 smithi114 ceph-mon[106969]: pgmap v204: 89 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 49 MiB/s wr, 809 op/s; 8011/360660 objects degraded (2.221%); 3.1 MiB/s, 48 objects/s recovering 2024-09-14T03:54:47.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:47 smithi114 ceph-mon[106969]: osdmap e101: 8 total, 8 up, 8 in 2024-09-14T03:54:47.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:47 smithi114 ceph-mon[106969]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:47 smithi114 ceph-mon[103029]: pgmap v204: 89 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 49 MiB/s wr, 809 op/s; 8011/360660 objects degraded (2.221%); 3.1 MiB/s, 48 objects/s recovering 2024-09-14T03:54:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:47 smithi114 ceph-mon[103029]: osdmap e101: 8 total, 8 up, 8 in 2024-09-14T03:54:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:47 smithi114 ceph-mon[103029]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:47 smithi160 ceph-mon[88407]: pgmap v204: 89 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 49 MiB/s wr, 809 op/s; 8011/360660 objects degraded (2.221%); 3.1 MiB/s, 48 objects/s recovering 2024-09-14T03:54:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:47 smithi160 ceph-mon[88407]: osdmap e101: 8 total, 8 up, 8 in 2024-09-14T03:54:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:47 smithi160 ceph-mon[88407]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:48.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:48 smithi114 ceph-mon[103029]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:48.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:48 smithi114 ceph-mon[106969]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:48 smithi160 ceph-mon[88407]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:49 smithi114 ceph-mon[103029]: pgmap v206: 89 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 37 MiB/s wr, 703 op/s; 8011/359502 objects degraded (2.228%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T03:54:49.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:49 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8011/359502 objects degraded (2.228%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:49 smithi114 ceph-mon[106969]: pgmap v206: 89 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 37 MiB/s wr, 703 op/s; 8011/359502 objects degraded (2.228%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T03:54:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:49 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8011/359502 objects degraded (2.228%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:49 smithi160 ceph-mon[88407]: pgmap v206: 89 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 37 MiB/s wr, 703 op/s; 8011/359502 objects degraded (2.228%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T03:54:49.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:49 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8011/359502 objects degraded (2.228%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:50.414 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:50.524 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:54:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:54:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:54:50.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:50 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2087436314' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:50.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:50 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2087436314' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:50 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2087436314' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:51.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:51 smithi114 ceph-mon[106969]: from='client.34557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:51.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:51 smithi114 ceph-mon[106969]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 32 MiB/s wr, 831 op/s; 7756/354807 objects degraded (2.186%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:54:51.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:51 smithi114 ceph-mon[106969]: from='client.54376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:51.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:54:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:54:51] "GET /metrics HTTP/1.1" 200 36395 "" "Prometheus/2.51.0" 2024-09-14T03:54:51.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:51 smithi114 ceph-mon[103029]: from='client.34557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:51.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:51 smithi114 ceph-mon[103029]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 32 MiB/s wr, 831 op/s; 7756/354807 objects degraded (2.186%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:54:51.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:51 smithi114 ceph-mon[103029]: from='client.54376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:51 smithi160 ceph-mon[88407]: from='client.34557 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:51 smithi160 ceph-mon[88407]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 32 MiB/s wr, 831 op/s; 7756/354807 objects degraded (2.186%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:54:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:51 smithi160 ceph-mon[88407]: from='client.54376 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:53.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:53 smithi114 ceph-mon[106969]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 7.9 MiB/s wr, 571 op/s; 7756/351996 objects degraded (2.203%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:54:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3333568937' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:53 smithi114 ceph-mon[103029]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 7.9 MiB/s wr, 571 op/s; 7756/351996 objects degraded (2.203%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:54:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3333568937' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:53 smithi160 ceph-mon[88407]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 127 B/s rd, 7.9 MiB/s wr, 571 op/s; 7756/351996 objects degraded (2.203%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:54:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:54:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3333568937' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:54.030 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:54.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:54:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:54:54.477+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:54 smithi114 ceph-mon[103029]: from='client.44438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:54 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7756/351996 objects degraded (2.203%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:54.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:54 smithi114 ceph-mon[106969]: from='client.44438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:54.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:54 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7756/351996 objects degraded (2.203%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:54.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:54 smithi160 ceph-mon[88407]: from='client.44438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:54 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7756/351996 objects degraded (2.203%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:54:55.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:55 smithi114 ceph-mon[106969]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 114 B/s rd, 7.1 MiB/s wr, 854 op/s; 7599/342849 objects degraded (2.216%); 4.4 MiB/s, 71 objects/s recovering 2024-09-14T03:54:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:55 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:55 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:55 smithi114 ceph-mon[106969]: from='client.54388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:55 smithi114 ceph-mon[103029]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 114 B/s rd, 7.1 MiB/s wr, 854 op/s; 7599/342849 objects degraded (2.216%); 4.4 MiB/s, 71 objects/s recovering 2024-09-14T03:54:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:55 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:55 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:55 smithi114 ceph-mon[103029]: from='client.54388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:55 smithi160 ceph-mon[88407]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 114 B/s rd, 7.1 MiB/s wr, 854 op/s; 7599/342849 objects degraded (2.216%); 4.4 MiB/s, 71 objects/s recovering 2024-09-14T03:54:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:55 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:54:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:55 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:54:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:55 smithi160 ceph-mon[88407]: from='client.54388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:57.630 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:54:57.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:57 smithi114 ceph-mon[106969]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 102 B/s rd, 2.2 MiB/s wr, 768 op/s; 7599/339771 objects degraded (2.237%); 2.6 MiB/s, 41 objects/s recovering 2024-09-14T03:54:57.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:57 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/557534738' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:57 smithi114 ceph-mon[103029]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 102 B/s rd, 2.2 MiB/s wr, 768 op/s; 7599/339771 objects degraded (2.237%); 2.6 MiB/s, 41 objects/s recovering 2024-09-14T03:54:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:57 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/557534738' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:57 smithi160 ceph-mon[88407]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 102 B/s rd, 2.2 MiB/s wr, 768 op/s; 7599/339771 objects degraded (2.237%); 2.6 MiB/s, 41 objects/s recovering 2024-09-14T03:54:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:57 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/557534738' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:54:58.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:58 smithi114 ceph-mon[106969]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:58.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:58 smithi114 ceph-mon[106969]: from='client.54403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:58.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:58 smithi114 ceph-mon[103029]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:58.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:58 smithi114 ceph-mon[103029]: from='client.54403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:58 smithi160 ceph-mon[88407]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:58 smithi160 ceph-mon[88407]: from='client.54403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:54:59.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:59 smithi114 ceph-mon[106969]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 1.9 MiB/s wr, 702 op/s; 7599/337701 objects degraded (2.250%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:59.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:54:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7599/337701 objects degraded (2.250%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:59.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:59 smithi114 ceph-mon[103029]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 1.9 MiB/s wr, 702 op/s; 7599/337701 objects degraded (2.250%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:59.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:54:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7599/337701 objects degraded (2.250%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:54:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:59 smithi160 ceph-mon[88407]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 1.9 MiB/s wr, 702 op/s; 7599/337701 objects degraded (2.250%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:54:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:54:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7599/337701 objects degraded (2.250%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:55:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:55:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:55:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:55:01.269 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:01.560 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:55:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:55:01] "GET /metrics HTTP/1.1" 200 36398 "" "Prometheus/2.51.0" 2024-09-14T03:55:01.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:01 smithi114 ceph-mon[106969]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 MiB/s wr, 823 op/s; 7423/331173 objects degraded (2.241%); 3.1 MiB/s, 48 objects/s recovering 2024-09-14T03:55:01.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:01 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/982913793' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:01.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:01 smithi114 ceph-mon[103029]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 MiB/s wr, 823 op/s; 7423/331173 objects degraded (2.241%); 3.1 MiB/s, 48 objects/s recovering 2024-09-14T03:55:01.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:01 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/982913793' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:01.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:01 smithi160 ceph-mon[88407]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 MiB/s wr, 823 op/s; 7423/331173 objects degraded (2.241%); 3.1 MiB/s, 48 objects/s recovering 2024-09-14T03:55:01.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:01 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/982913793' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:02 smithi114 ceph-mon[106969]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:02 smithi114 ceph-mon[106969]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:02 smithi114 ceph-mon[103029]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:02 smithi114 ceph-mon[103029]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:02 smithi160 ceph-mon[88407]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:02 smithi160 ceph-mon[88407]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:03 smithi114 ceph-mon[106969]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 7423/328398 objects degraded (2.260%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T03:55:03.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:03 smithi114 ceph-mon[103029]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 7423/328398 objects degraded (2.260%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T03:55:03.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:03 smithi160 ceph-mon[88407]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 7423/328398 objects degraded (2.260%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T03:55:04.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7423/328398 objects degraded (2.260%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:55:04.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:04 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1503671992' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:04.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7423/328398 objects degraded (2.260%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:55:04.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:04 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1503671992' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:04.903 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7423/328398 objects degraded (2.260%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:55:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:04 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1503671992' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:05.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:05 smithi114 ceph-mon[106969]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 7248/319560 objects degraded (2.268%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:55:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:05 smithi114 ceph-mon[106969]: from='client.44471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:05 smithi114 ceph-mon[103029]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 7248/319560 objects degraded (2.268%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:55:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:05 smithi114 ceph-mon[103029]: from='client.44471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:05 smithi160 ceph-mon[88407]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 7248/319560 objects degraded (2.268%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:55:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:05 smithi160 ceph-mon[88407]: from='client.44471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:06.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:06 smithi114 ceph-mon[106969]: from='client.44477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:06.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:06 smithi114 ceph-mon[103029]: from='client.44477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:06 smithi160 ceph-mon[88407]: from='client.44477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:07.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:07 smithi114 ceph-mon[106969]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 7248/316683 objects degraded (2.289%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:07 smithi114 ceph-mon[103029]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 7248/316683 objects degraded (2.289%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:07 smithi160 ceph-mon[88407]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 7248/316683 objects degraded (2.289%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:08.528 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:08.836 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:08.836 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2245267' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:08.837 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:08.837 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2245267' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2245267' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:09 smithi114 ceph-mon[106969]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 7248/314460 objects degraded (2.305%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:09 smithi114 ceph-mon[106969]: from='client.34629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:09 smithi114 ceph-mon[106969]: from='client.34635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7248/314460 objects degraded (2.305%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:55:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:09 smithi114 ceph-mon[103029]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 7248/314460 objects degraded (2.305%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:09 smithi114 ceph-mon[103029]: from='client.34629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:09 smithi114 ceph-mon[103029]: from='client.34635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7248/314460 objects degraded (2.305%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:55:09.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:09 smithi160 ceph-mon[88407]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 7248/314460 objects degraded (2.305%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:09 smithi160 ceph-mon[88407]: from='client.34629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:09 smithi160 ceph-mon[88407]: from='client.34635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7248/314460 objects degraded (2.305%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:55:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:10.501 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:55:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:55:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:55:11.801 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:11 smithi114 ceph-mon[106969]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 823 op/s; 7091/308034 objects degraded (2.302%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:55:11.801 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:11 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1619351866' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:11.802 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:55:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:55:11] "GET /metrics HTTP/1.1" 200 36397 "" "Prometheus/2.51.0" 2024-09-14T03:55:11.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:11 smithi114 ceph-mon[103029]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 823 op/s; 7091/308034 objects degraded (2.302%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:55:11.802 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:11 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1619351866' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:11.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:11 smithi160 ceph-mon[88407]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 823 op/s; 7091/308034 objects degraded (2.302%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:55:11.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:11 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1619351866' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:12.158 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:12.772 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:12 smithi160 ceph-mon[88407]: from='client.34647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:12.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:12 smithi114 ceph-mon[103029]: from='client.34647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:12.967 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:12 smithi114 ceph-mon[106969]: from='client.34647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:13 smithi160 ceph-mon[88407]: pgmap v218: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 7091/305226 objects degraded (2.323%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T03:55:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:13 smithi160 ceph-mon[88407]: from='client.44492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:13.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:13 smithi114 ceph-mon[103029]: pgmap v218: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 7091/305226 objects degraded (2.323%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T03:55:13.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:13 smithi114 ceph-mon[103029]: from='client.44492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:13.919 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:13 smithi114 ceph-mon[106969]: pgmap v218: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 7091/305226 objects degraded (2.323%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T03:55:13.919 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:13 smithi114 ceph-mon[106969]: from='client.44492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7091/305226 objects degraded (2.323%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:14 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:14 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:15.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7091/305226 objects degraded (2.323%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:15.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:14 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:15.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:14 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:15.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7091/305226 objects degraded (2.323%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:15.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:14 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:15.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:14 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:15.805 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:15.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:55:15.573+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:55:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: pgmap v219: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 6895/297060 objects degraded (2.321%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T03:55:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3245390705' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:55:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:55:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:16.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:16.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:55:16.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: pgmap v219: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 6895/297060 objects degraded (2.321%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T03:55:16.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:16.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:16.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3245390705' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:16.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:55:16.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:55:16.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:16.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:16.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:16.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:16.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:16.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:55:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: pgmap v219: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 6895/297060 objects degraded (2.321%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T03:55:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3245390705' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:16.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:55:16.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:55:16.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:16.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:16.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:16.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:16.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:16.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:55:17.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:16 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:55:17.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:16 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:55:17.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:16 smithi114 ceph-mon[106969]: from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:17.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:16 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:55:17.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:16 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:55:17.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:16 smithi114 ceph-mon[103029]: from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:16 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:55:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:16 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-14T03:55:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:16 smithi160 ceph-mon[88407]: from='client.54445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:18.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:17 smithi114 ceph-mon[106969]: from='client.54451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:18.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:17 smithi114 ceph-mon[106969]: pgmap v220: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 6895/294390 objects degraded (2.342%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:18.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:17 smithi114 ceph-mon[103029]: from='client.54451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:18.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:17 smithi114 ceph-mon[103029]: pgmap v220: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 6895/294390 objects degraded (2.342%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:17 smithi160 ceph-mon[88407]: from='client.54451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:17 smithi160 ceph-mon[88407]: pgmap v220: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 6895/294390 objects degraded (2.342%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:19.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2294162368' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:18 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2294162368' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:18 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2294162368' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:19.435 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:20.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:19 smithi114 ceph-mon[106969]: pgmap v221: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 678 op/s; 6895/292248 objects degraded (2.359%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6895/292248 objects degraded (2.359%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:20.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:19 smithi114 ceph-mon[103029]: pgmap v221: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 678 op/s; 6895/292248 objects degraded (2.359%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:20.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6895/292248 objects degraded (2.359%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:20.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:55:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:55:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:55:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:19 smithi160 ceph-mon[88407]: pgmap v221: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 678 op/s; 6895/292248 objects degraded (2.359%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:55:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6895/292248 objects degraded (2.359%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:21.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:21 smithi114 ceph-mon[106969]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:21.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:21 smithi114 ceph-mon[106969]: from='client.44504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:21 smithi114 ceph-mon[103029]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:21 smithi114 ceph-mon[103029]: from='client.44504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:21 smithi160 ceph-mon[88407]: from='client.54454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:21 smithi160 ceph-mon[88407]: from='client.44504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:55:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:55:21] "GET /metrics HTTP/1.1" 200 36397 "" "Prometheus/2.51.0" 2024-09-14T03:55:22.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:22 smithi114 ceph-mon[106969]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 793 op/s; 6704/285882 objects degraded (2.345%); 2.8 MiB/s, 45 objects/s recovering 2024-09-14T03:55:22.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:22 smithi114 ceph-mon[103029]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 793 op/s; 6704/285882 objects degraded (2.345%); 2.8 MiB/s, 45 objects/s recovering 2024-09-14T03:55:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:22 smithi160 ceph-mon[88407]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 793 op/s; 6704/285882 objects degraded (2.345%); 2.8 MiB/s, 45 objects/s recovering 2024-09-14T03:55:23.059 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:23.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:23 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/633213252' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:23.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/633213252' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:23 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/633213252' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:24.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:24 smithi114 ceph-mon[106969]: pgmap v223: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 6704/283131 objects degraded (2.368%); 2.0 MiB/s, 32 objects/s recovering 2024-09-14T03:55:24.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:24 smithi114 ceph-mon[106969]: from='client.54463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:24.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:24 smithi114 ceph-mon[103029]: pgmap v223: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 6704/283131 objects degraded (2.368%); 2.0 MiB/s, 32 objects/s recovering 2024-09-14T03:55:24.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:24 smithi114 ceph-mon[103029]: from='client.54463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:24 smithi160 ceph-mon[88407]: pgmap v223: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 6704/283131 objects degraded (2.368%); 2.0 MiB/s, 32 objects/s recovering 2024-09-14T03:55:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:24 smithi160 ceph-mon[88407]: from='client.54463 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:25.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:25 smithi114 ceph-mon[106969]: from='client.44519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:25.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:25 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6704/283131 objects degraded (2.368%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:25.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:25 smithi114 ceph-mon[103029]: from='client.44519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:25.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:25 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6704/283131 objects degraded (2.368%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:25.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:25 smithi160 ceph-mon[88407]: from='client.44519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:25.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:25 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6704/283131 objects degraded (2.368%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:26.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:26 smithi114 ceph-mon[106969]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 6493/274395 objects degraded (2.366%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T03:55:26.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:26 smithi114 ceph-mon[103029]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 6493/274395 objects degraded (2.366%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T03:55:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:26 smithi160 ceph-mon[88407]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 856 op/s; 6493/274395 objects degraded (2.366%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T03:55:26.675 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:27.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:27 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2435788612' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:27.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:27 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2435788612' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:27.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:27 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2435788612' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:28.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:28 smithi114 ceph-mon[103029]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 705 op/s; 6493/271641 objects degraded (2.390%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:55:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:28 smithi114 ceph-mon[103029]: from='client.34695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:28 smithi114 ceph-mon[103029]: osdmap e102: 8 total, 8 up, 8 in 2024-09-14T03:55:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:28 smithi114 ceph-mon[103029]: from='client.34701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:28.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:28 smithi114 ceph-mon[106969]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 705 op/s; 6493/271641 objects degraded (2.390%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:55:28.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:28 smithi114 ceph-mon[106969]: from='client.34695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:28.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:28 smithi114 ceph-mon[106969]: osdmap e102: 8 total, 8 up, 8 in 2024-09-14T03:55:28.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:28 smithi114 ceph-mon[106969]: from='client.34701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:28 smithi160 ceph-mon[88407]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 705 op/s; 6493/271641 objects degraded (2.390%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:55:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:28 smithi160 ceph-mon[88407]: from='client.34695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:28 smithi160 ceph-mon[88407]: osdmap e102: 8 total, 8 up, 8 in 2024-09-14T03:55:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:28 smithi160 ceph-mon[88407]: from='client.34701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:29.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:29 smithi114 ceph-mon[103029]: osdmap e103: 8 total, 8 up, 8 in 2024-09-14T03:55:29.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:29 smithi114 ceph-mon[106969]: osdmap e103: 8 total, 8 up, 8 in 2024-09-14T03:55:29.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:29 smithi160 ceph-mon[88407]: osdmap e103: 8 total, 8 up, 8 in 2024-09-14T03:55:30.290 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:30.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:30 smithi114 ceph-mon[106969]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 6493/269862 objects degraded (2.406%); 1.6 MiB/s, 26 objects/s recovering 2024-09-14T03:55:30.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:30 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6493/269862 objects degraded (2.406%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:30 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1118411308' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:30 smithi114 ceph-mon[103029]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 6493/269862 objects degraded (2.406%); 1.6 MiB/s, 26 objects/s recovering 2024-09-14T03:55:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:30 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6493/269862 objects degraded (2.406%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:30 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1118411308' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:30.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:55:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:55:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:55:30.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:30 smithi160 ceph-mon[88407]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 6493/269862 objects degraded (2.406%); 1.6 MiB/s, 26 objects/s recovering 2024-09-14T03:55:30.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:30 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6493/269862 objects degraded (2.406%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:30.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:30 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1118411308' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:31.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:55:31.368 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:55:31.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-14T03:55:31.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:55:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:55:31] "GET /metrics HTTP/1.1" 200 36395 "" "Prometheus/2.51.0" 2024-09-14T03:55:32.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:32 smithi114 ceph-mon[106969]: from='client.54484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:32.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:32 smithi114 ceph-mon[106969]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 780 op/s; 6215/264396 objects degraded (2.351%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T03:55:32.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:32 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:55:32.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:32 smithi114 ceph-mon[106969]: Upgrade: osd.2 is safe to restart 2024-09-14T03:55:32.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:32 smithi114 ceph-mon[106969]: from='client.44546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:32 smithi114 ceph-mon[103029]: from='client.54484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:32 smithi114 ceph-mon[103029]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 780 op/s; 6215/264396 objects degraded (2.351%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T03:55:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:32 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:55:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:32 smithi114 ceph-mon[103029]: Upgrade: osd.2 is safe to restart 2024-09-14T03:55:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:32 smithi114 ceph-mon[103029]: from='client.44546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:32 smithi160 ceph-mon[88407]: from='client.54484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:32 smithi160 ceph-mon[88407]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 780 op/s; 6215/264396 objects degraded (2.351%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T03:55:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:32 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-14T03:55:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:32 smithi160 ceph-mon[88407]: Upgrade: osd.2 is safe to restart 2024-09-14T03:55:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:32 smithi160 ceph-mon[88407]: from='client.44546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:33.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[106969]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 515 op/s; 6215/262014 objects degraded (2.372%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:55:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[106969]: Upgrade: Updating osd.2 2024-09-14T03:55:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-14T03:55:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:55:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[106969]: Deploying daemon osd.2 on smithi114 2024-09-14T03:55:33.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1421775717' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:33.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[103029]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 515 op/s; 6215/262014 objects degraded (2.372%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:55:33.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[103029]: Upgrade: Updating osd.2 2024-09-14T03:55:33.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:33.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-14T03:55:33.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:55:33.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[103029]: Deploying daemon osd.2 on smithi114 2024-09-14T03:55:33.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:33 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1421775717' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:33 smithi160 ceph-mon[88407]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 515 op/s; 6215/262014 objects degraded (2.372%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T03:55:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:33 smithi160 ceph-mon[88407]: Upgrade: Updating osd.2 2024-09-14T03:55:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-14T03:55:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:55:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:33 smithi160 ceph-mon[88407]: Deploying daemon osd.2 on smithi114 2024-09-14T03:55:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:33 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1421775717' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:33.939 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:34.799 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:34 smithi114 ceph-mon[106969]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:34.799 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6215/262014 objects degraded (2.372%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:34.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:34 smithi114 ceph-mon[103029]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:34.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6215/262014 objects degraded (2.372%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:34 smithi160 ceph-mon[88407]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6215/262014 objects degraded (2.372%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:35.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:35 smithi114 ceph-mon[103029]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 5964/253974 objects degraded (2.348%); 4.1 MiB/s, 66 objects/s recovering 2024-09-14T03:55:35.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:35 smithi114 ceph-mon[103029]: from='client.44558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:35.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:35 smithi114 ceph-mon[106969]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 5964/253974 objects degraded (2.348%); 4.1 MiB/s, 66 objects/s recovering 2024-09-14T03:55:35.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:35 smithi114 ceph-mon[106969]: from='client.44558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:35.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:35 smithi160 ceph-mon[88407]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 5964/253974 objects degraded (2.348%); 4.1 MiB/s, 66 objects/s recovering 2024-09-14T03:55:35.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:35 smithi160 ceph-mon[88407]: from='client.44558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:36.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:36 smithi114 systemd[1]: Stopping Ceph osd.2 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:55:37.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:36 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[57613]: 2024-09-14T03:55:36.872+0000 7f9806d11640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T03:55:37.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:36 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[57613]: 2024-09-14T03:55:36.872+0000 7f9806d11640 -1 osd.2 103 *** Got signal Terminated *** 2024-09-14T03:55:37.371 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:36 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[57613]: 2024-09-14T03:55:36.872+0000 7f9806d11640 -1 osd.2 103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T03:55:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:37.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:37 smithi114 ceph-mon[103029]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 5964/251340 objects degraded (2.373%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T03:55:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:37 smithi114 ceph-mon[103029]: osd.2 marked itself down and dead 2024-09-14T03:55:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:37 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3637176239' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:37 smithi114 ceph-mon[106969]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 5964/251340 objects degraded (2.373%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T03:55:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:37 smithi114 ceph-mon[106969]: osd.2 marked itself down and dead 2024-09-14T03:55:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:37 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3637176239' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:37 smithi160 ceph-mon[88407]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 5964/251340 objects degraded (2.373%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T03:55:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:37 smithi160 ceph-mon[88407]: osd.2 marked itself down and dead 2024-09-14T03:55:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:37 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3637176239' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:38.370 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:38 smithi114 podman[127387]: 2024-09-14 03:55:38.025625999 +0000 UTC m=+1.315043716 container died a76e6ee3bc5426f63388b2f9d936ec3816f98461a441adf9b100f6d0e07435c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-eb16b97, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-14T03:55:38.807 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[106969]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:55:38.807 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[106969]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:38.807 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[106969]: osdmap e104: 8 total, 7 up, 8 in 2024-09-14T03:55:38.807 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:38.807 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:38.807 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[106969]: from='client.44576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:38.807 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[103029]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:55:38.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[103029]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:38.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[103029]: osdmap e104: 8 total, 7 up, 8 in 2024-09-14T03:55:38.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:38.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:38.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:38 smithi114 ceph-mon[103029]: from='client.44576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:38.808 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:38 smithi114 podman[127387]: 2024-09-14 03:55:38.716168195 +0000 UTC m=+2.005585921 container cleanup a76e6ee3bc5426f63388b2f9d936ec3816f98461a441adf9b100f6d0e07435c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-eb16b97, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T03:55:38.808 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:38 smithi114 bash[127387]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2 2024-09-14T03:55:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:38 smithi160 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:55:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:38 smithi160 ceph-mon[88407]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:38 smithi160 ceph-mon[88407]: osdmap e104: 8 total, 7 up, 8 in 2024-09-14T03:55:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:38 smithi160 ceph-mon[88407]: from='client.44576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:39.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:38 smithi114 podman[127462]: 2024-09-14 03:55:38.987014501 +0000 UTC m=+0.955887533 container remove a76e6ee3bc5426f63388b2f9d936ec3816f98461a441adf9b100f6d0e07435c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T03:55:39.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:39 smithi114 ceph-mon[103029]: pgmap v234: 89 pgs: 8 peering, 3 stale+active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 5964/249750 objects degraded (2.388%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:55:39.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:39 smithi114 ceph-mon[103029]: osdmap e105: 8 total, 7 up, 8 in 2024-09-14T03:55:39.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:39 smithi114 ceph-mon[103029]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-14T03:55:39.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:39 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 5964/249750 objects degraded (2.388%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:39.851 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:39 smithi114 ceph-mon[106969]: pgmap v234: 89 pgs: 8 peering, 3 stale+active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 5964/249750 objects degraded (2.388%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:55:39.851 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:39 smithi114 ceph-mon[106969]: osdmap e105: 8 total, 7 up, 8 in 2024-09-14T03:55:39.852 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:39 smithi114 ceph-mon[106969]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-14T03:55:39.852 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:39 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 5964/249750 objects degraded (2.388%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:39.852 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:39 smithi114 podman[127538]: 2024-09-14 03:55:39.445089437 +0000 UTC m=+0.043931662 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:55:39.852 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:39 smithi114 podman[127538]: 2024-09-14 03:55:39.571446427 +0000 UTC m=+0.170288647 container create 1636c3360dbc13ee4a1a2dd43a5117a9ccb866e2b2b6430736fe4986b2d19338 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-14T03:55:39.852 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:39 smithi114 podman[127538]: 2024-09-14 03:55:39.849622699 +0000 UTC m=+0.448464936 container init 1636c3360dbc13ee4a1a2dd43a5117a9ccb866e2b2b6430736fe4986b2d19338 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=main-a278231, CEPH_POINT_RELEASE=) 2024-09-14T03:55:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:39 smithi160 ceph-mon[88407]: pgmap v234: 89 pgs: 8 peering, 3 stale+active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 5964/249750 objects degraded (2.388%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:55:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:39 smithi160 ceph-mon[88407]: osdmap e105: 8 total, 7 up, 8 in 2024-09-14T03:55:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:39 smithi160 ceph-mon[88407]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-14T03:55:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:39 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 5964/249750 objects degraded (2.388%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:40.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:39 smithi114 podman[127538]: 2024-09-14 03:55:39.85726802 +0000 UTC m=+0.456110248 container start 1636c3360dbc13ee4a1a2dd43a5117a9ccb866e2b2b6430736fe4986b2d19338 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, RELEASE=main-a278231, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:55:40.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:39 smithi114 podman[127538]: 2024-09-14 03:55:39.97450473 +0000 UTC m=+0.573346964 container attach 1636c3360dbc13ee4a1a2dd43a5117a9ccb866e2b2b6430736fe4986b2d19338 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=main-a278231, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:55:40.539 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:40 smithi114 podman[127597]: 2024-09-14 03:55:40.215249043 +0000 UTC m=+0.047324424 container died 1636c3360dbc13ee4a1a2dd43a5117a9ccb866e2b2b6430736fe4986b2d19338 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, RELEASE=main-a278231, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-14T03:55:40.554 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:55:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:55:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:55:40.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2608233290' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:40.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2608233290' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:40.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2608233290' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:40.964 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:41.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:40 smithi114 podman[127597]: 2024-09-14 03:55:40.955608572 +0000 UTC m=+0.787683949 container remove 1636c3360dbc13ee4a1a2dd43a5117a9ccb866e2b2b6430736fe4986b2d19338 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=main-a278231, ceph=True, org.label-schema.license=GPLv2) 2024-09-14T03:55:41.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:40 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-14T03:55:41.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:40 smithi114 systemd[1]: Stopped Ceph osd.2 for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:55:41.317 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:40 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.2.service: Consumed 18min 27.631s CPU time. 2024-09-14T03:55:41.318 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:41 smithi114 systemd[1]: Starting Ceph osd.2 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:55:41.569 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:55:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:55:41] "GET /metrics HTTP/1.1" 200 36391 "" "Prometheus/2.51.0" 2024-09-14T03:55:41.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:41 smithi114 ceph-mon[106969]: pgmap v236: 89 pgs: 3 active+undersized, 1 active+recovering+undersized+degraded+remapped, 17 peering, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 62 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 20765/244905 objects degraded (8.479%); 5.2 MiB/s, 82 objects/s recovering 2024-09-14T03:55:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:41 smithi114 ceph-mon[106969]: from='client.54508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:41.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:41 smithi114 ceph-mon[103029]: pgmap v236: 89 pgs: 3 active+undersized, 1 active+recovering+undersized+degraded+remapped, 17 peering, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 62 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 20765/244905 objects degraded (8.479%); 5.2 MiB/s, 82 objects/s recovering 2024-09-14T03:55:41.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:41 smithi114 ceph-mon[103029]: from='client.54508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:41.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:41 smithi114 podman[127742]: 2024-09-14 03:55:41.586592585 +0000 UTC m=+0.039330563 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:55:41.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:41 smithi114 podman[127742]: 2024-09-14 03:55:41.717666883 +0000 UTC m=+0.170404856 container create e7ca83ccbfb5e1a788416c65d07d3f42058db88301f8de31248a4c50c9224ee4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True) 2024-09-14T03:55:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:41 smithi160 ceph-mon[88407]: pgmap v236: 89 pgs: 3 active+undersized, 1 active+recovering+undersized+degraded+remapped, 17 peering, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 62 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 20765/244905 objects degraded (8.479%); 5.2 MiB/s, 82 objects/s recovering 2024-09-14T03:55:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:41 smithi160 ceph-mon[88407]: from='client.54508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:42.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 podman[127742]: 2024-09-14 03:55:42.015243469 +0000 UTC m=+0.467981434 container init e7ca83ccbfb5e1a788416c65d07d3f42058db88301f8de31248a4c50c9224ee4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-a278231, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2) 2024-09-14T03:55:42.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 podman[127742]: 2024-09-14 03:55:42.024320971 +0000 UTC m=+0.477058937 container start e7ca83ccbfb5e1a788416c65d07d3f42058db88301f8de31248a4c50c9224ee4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-a278231, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-14T03:55:42.236 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 podman[127742]: 2024-09-14 03:55:42.12691013 +0000 UTC m=+0.579648098 container attach e7ca83ccbfb5e1a788416c65d07d3f42058db88301f8de31248a4c50c9224ee4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=main-a278231, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-14T03:55:42.237 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate[127754]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:55:42.237 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 bash[127742]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:55:42.575 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate[127754]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:55:42.575 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 bash[127742]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:55:42.847 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-mon[103029]: from='client.34743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:42.847 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-mon[106969]: from='client.34743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:42 smithi160 ceph-mon[88407]: from='client.34743 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:43.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate[127754]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-14T03:55:43.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 bash[127742]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-14T03:55:43.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate[127754]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-14T03:55:43.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 bash[127742]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-14T03:55:43.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate[127754]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-14T03:55:43.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 bash[127742]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-14T03:55:43.121 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate[127754]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-14T03:55:43.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 bash[127742]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-14T03:55:43.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate[127754]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-14T03:55:43.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 bash[127742]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-14T03:55:43.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate[127754]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-14T03:55:43.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 bash[127742]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-14T03:55:43.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate[127754]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-14T03:55:43.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 bash[127742]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-14T03:55:43.122 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:42 smithi114 podman[127900]: 2024-09-14 03:55:42.998542869 +0000 UTC m=+0.030872767 container died e7ca83ccbfb5e1a788416c65d07d3f42058db88301f8de31248a4c50c9224ee4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a278231, io.buildah.version=1.37.2) 2024-09-14T03:55:43.579 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:43 smithi114 ceph-mon[103029]: pgmap v237: 89 pgs: 4 active+undersized, 1 active+recovering+undersized+degraded+remapped, 17 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 60 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 479 op/s; 20767/242463 objects degraded (8.565%); 3.2 MiB/s, 51 objects/s recovering 2024-09-14T03:55:43.867 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:43 smithi114 ceph-mon[106969]: pgmap v237: 89 pgs: 4 active+undersized, 1 active+recovering+undersized+degraded+remapped, 17 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 60 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 479 op/s; 20767/242463 objects degraded (8.565%); 3.2 MiB/s, 51 objects/s recovering 2024-09-14T03:55:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:43 smithi160 ceph-mon[88407]: pgmap v237: 89 pgs: 4 active+undersized, 1 active+recovering+undersized+degraded+remapped, 17 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+undersized+degraded, 60 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 479 op/s; 20767/242463 objects degraded (8.565%); 3.2 MiB/s, 51 objects/s recovering 2024-09-14T03:55:44.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:43 smithi114 podman[127900]: 2024-09-14 03:55:43.866757072 +0000 UTC m=+0.899086965 container remove e7ca83ccbfb5e1a788416c65d07d3f42058db88301f8de31248a4c50c9224ee4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=main-a278231, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T03:55:44.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:44 smithi114 podman[127977]: 2024-09-14 03:55:44.079299432 +0000 UTC m=+0.034327515 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:55:44.480 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:44 smithi114 podman[127977]: 2024-09-14 03:55:44.201271671 +0000 UTC m=+0.156299742 container create f3696f57b771b2ab55ca6defccd4d170c3ed7be670e84a6b3aed2cf789cb4797 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-a278231) 2024-09-14T03:55:44.620 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:44 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/815558465' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:44 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 20767/242463 objects degraded (8.565%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:44 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/815558465' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:44 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 20767/242463 objects degraded (8.565%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:44.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:44 smithi114 podman[127977]: 2024-09-14 03:55:44.47934408 +0000 UTC m=+0.434372167 container init f3696f57b771b2ab55ca6defccd4d170c3ed7be670e84a6b3aed2cf789cb4797 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-a278231, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-14T03:55:44.871 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:44 smithi114 podman[127977]: 2024-09-14 03:55:44.495153938 +0000 UTC m=+0.450182015 container start f3696f57b771b2ab55ca6defccd4d170c3ed7be670e84a6b3aed2cf789cb4797 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-a278231, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-14T03:55:44.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:44 smithi114 bash[127977]: f3696f57b771b2ab55ca6defccd4d170c3ed7be670e84a6b3aed2cf789cb4797 2024-09-14T03:55:44.872 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:44 smithi114 systemd[1]: Started Ceph osd.2 for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:55:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:44 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/815558465' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:44 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 20767/242463 objects degraded (8.565%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:45.693 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[103029]: pgmap v238: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 34047/232677 objects degraded (14.633%); 5.5 MiB/s, 88 objects/s recovering 2024-09-14T03:55:45.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[103029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-14T03:55:45.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[103029]: from='client.54523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:45.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:45.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:45.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:45.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[103029]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:45.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[106969]: pgmap v238: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 34047/232677 objects degraded (14.633%); 5.5 MiB/s, 88 objects/s recovering 2024-09-14T03:55:45.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[106969]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-14T03:55:45.695 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[106969]: from='client.54523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:45.695 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:45.695 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:45.695 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:45.695 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-mon[106969]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:45.695 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:45 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[128013]: 2024-09-14T03:55:45.421+0000 7f6b1a540640 -1 Falling back to public interface 2024-09-14T03:55:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:45 smithi160 ceph-mon[88407]: pgmap v238: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 34047/232677 objects degraded (14.633%); 5.5 MiB/s, 88 objects/s recovering 2024-09-14T03:55:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:45 smithi160 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-14T03:55:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:45 smithi160 ceph-mon[88407]: from='client.54523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:45 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:45 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:45 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:45 smithi160 ceph-mon[88407]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:47 smithi114 ceph-mon[103029]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 34047/229533 objects degraded (14.833%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T03:55:47.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:47 smithi114 ceph-mon[106969]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 34047/229533 objects degraded (14.833%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T03:55:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:47 smithi160 ceph-mon[88407]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 34047/229533 objects degraded (14.833%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T03:55:48.229 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:48 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3269531113' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:48 smithi160 ceph-mon[88407]: from='client.34773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:48.961 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:48 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3269531113' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:48.962 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:48 smithi114 ceph-mon[106969]: from='client.34773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:48.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:48 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3269531113' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:48.962 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:48 smithi114 ceph-mon[103029]: from='client.34773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:49 smithi160 ceph-mon[88407]: pgmap v240: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 673 op/s; 34047/229533 objects degraded (14.833%); 4.4 MiB/s, 70 objects/s recovering 2024-09-14T03:55:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:49 smithi160 ceph-mon[88407]: from='client.44609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:49 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 34047/229533 objects degraded (14.833%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:49 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:49 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:50.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:49 smithi114 ceph-mon[106969]: pgmap v240: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 673 op/s; 34047/229533 objects degraded (14.833%); 4.4 MiB/s, 70 objects/s recovering 2024-09-14T03:55:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:49 smithi114 ceph-mon[106969]: from='client.44609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:49 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 34047/229533 objects degraded (14.833%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:49 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:49 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:50.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:49 smithi114 ceph-mon[103029]: pgmap v240: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 673 op/s; 34047/229533 objects degraded (14.833%); 4.4 MiB/s, 70 objects/s recovering 2024-09-14T03:55:50.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:49 smithi114 ceph-mon[103029]: from='client.44609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:50.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:49 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 34047/229533 objects degraded (14.833%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:50.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:49 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:50.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:49 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:55:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:55:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:55:51.693 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:51 smithi114 ceph-mon[106969]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 737 op/s; 32217/220269 objects degraded (14.626%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T03:55:51.693 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:51.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:51.694 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:51 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2017315829' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:51.694 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:55:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:55:51] "GET /metrics HTTP/1.1" 200 36391 "" "Prometheus/2.51.0" 2024-09-14T03:55:51.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:51 smithi114 ceph-mon[103029]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 737 op/s; 32217/220269 objects degraded (14.626%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T03:55:51.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:51.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:51.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2017315829' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:51.817 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:51 smithi160 ceph-mon[88407]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 737 op/s; 32217/220269 objects degraded (14.626%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T03:55:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:51 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2017315829' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:52.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:52 smithi114 ceph-mon[106969]: osdmap e106: 8 total, 7 up, 8 in 2024-09-14T03:55:52.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:52 smithi114 ceph-mon[106969]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:52.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:52 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:52.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:52 smithi114 ceph-mon[103029]: osdmap e106: 8 total, 7 up, 8 in 2024-09-14T03:55:52.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:52 smithi114 ceph-mon[103029]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:52.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:52 smithi160 ceph-mon[88407]: osdmap e106: 8 total, 7 up, 8 in 2024-09-14T03:55:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:52 smithi160 ceph-mon[88407]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:52 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:55:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:53 smithi114 ceph-mon[106969]: from='client.34791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:53 smithi114 ceph-mon[106969]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 32217/217335 objects degraded (14.824%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:55:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:53 smithi114 ceph-mon[106969]: osdmap e107: 8 total, 7 up, 8 in 2024-09-14T03:55:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:53 smithi114 ceph-mon[103029]: from='client.34791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:53 smithi114 ceph-mon[103029]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 32217/217335 objects degraded (14.824%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:55:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:53 smithi114 ceph-mon[103029]: osdmap e107: 8 total, 7 up, 8 in 2024-09-14T03:55:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:53 smithi160 ceph-mon[88407]: from='client.34791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:53 smithi160 ceph-mon[88407]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 60 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 32217/217335 objects degraded (14.824%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T03:55:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:53 smithi160 ceph-mon[88407]: osdmap e107: 8 total, 7 up, 8 in 2024-09-14T03:55:54.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:54 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 32217/217335 objects degraded (14.824%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:55.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:54 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 32217/217335 objects degraded (14.824%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:55.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:54 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 32217/217335 objects degraded (14.824%), 13 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:55:55.362 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:55.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[128013]: 2024-09-14T03:55:55.339+0000 7f6b1a540640 -1 osd.2 103 log_to_monitors true 2024-09-14T03:55:56.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:55 smithi114 ceph-mon[106969]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 61 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 30691/210069 objects degraded (14.610%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T03:55:56.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:55 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4198184654' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:56.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:55 smithi114 ceph-mon[106969]: from='osd.2 [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-14T03:55:56.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:55 smithi114 ceph-mon[106969]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-14T03:55:56.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:55 smithi114 ceph-mon[103029]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 61 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 30691/210069 objects degraded (14.610%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T03:55:56.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:55 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4198184654' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:56.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:55 smithi114 ceph-mon[103029]: from='osd.2 [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-14T03:55:56.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:55 smithi114 ceph-mon[103029]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-14T03:55:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:55 smithi160 ceph-mon[88407]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 61 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 30691/210069 objects degraded (14.610%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T03:55:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:55 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4198184654' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:55 smithi160 ceph-mon[88407]: from='osd.2 [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-14T03:55:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:55 smithi160 ceph-mon[88407]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-14T03:55:56.581 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[128013]: 2024-09-14T03:55:56.220+0000 7f6b0e2e2640 -1 osd.2 103 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:55:56.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='client.54568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:56.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-14T03:55:56.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: osdmap e108: 8 total, 7 up, 8 in 2024-09-14T03:55:56.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='osd.2 [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:55:56.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:55:56.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='client.34803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:56.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='osd.2 ' entity='osd.2' 2024-09-14T03:55:56.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:56.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:56.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:55:56.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:55:56.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:56.873 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:55:56.794+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-14T03:55:56.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='client.54568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:56.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-14T03:55:56.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: osdmap e108: 8 total, 7 up, 8 in 2024-09-14T03:55:56.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='osd.2 [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:55:56.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:55:56.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='client.34803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:56.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='osd.2 ' entity='osd.2' 2024-09-14T03:55:56.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:56.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:56.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:55:56.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:55:56.874 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:56 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='client.54568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-14T03:55:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: osdmap e108: 8 total, 7 up, 8 in 2024-09-14T03:55:57.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='osd.2 [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:55:57.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:55:57.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='client.34803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:55:57.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='osd.2 ' entity='osd.2' 2024-09-14T03:55:57.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:57.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:57.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:55:57.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:55:57.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:56 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:55:58.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 61 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 30691/207714 objects degraded (14.776%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T03:55:58.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:58.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:58.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:58.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:58.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:55:58.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:55:58.098 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-14T03:55:58.098 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:55:58.098 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: osd.2 [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440] boot 2024-09-14T03:55:58.098 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: osdmap e109: 8 total, 8 up, 8 in 2024-09-14T03:55:58.098 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-14T03:55:58.098 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 61 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 30691/207714 objects degraded (14.776%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T03:55:58.098 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:58.098 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:58.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:58.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:58.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:55:58.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:55:58.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-14T03:55:58.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:55:58.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: osd.2 [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440] boot 2024-09-14T03:55:58.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: osdmap e109: 8 total, 8 up, 8 in 2024-09-14T03:55:58.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:57 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-14T03:55:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 61 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 30691/207714 objects degraded (14.776%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T03:55:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:55:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:55:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:55:58.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:55:58.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-09-14T03:55:58.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:55:58.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: osd.2 [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440] boot 2024-09-14T03:55:58.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: osdmap e109: 8 total, 8 up, 8 in 2024-09-14T03:55:58.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-14T03:55:58.543 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:55:59.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:58 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2333135511' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:59.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:58 smithi114 ceph-mon[103029]: osdmap e110: 8 total, 8 up, 8 in 2024-09-14T03:55:59.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:58 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2333135511' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:59.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:58 smithi114 ceph-mon[106969]: osdmap e110: 8 total, 8 up, 8 in 2024-09-14T03:55:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:58 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2333135511' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:55:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:58 smithi160 ceph-mon[88407]: osdmap e110: 8 total, 8 up, 8 in 2024-09-14T03:56:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:59 smithi160 ceph-mon[88407]: pgmap v250: 89 pgs: 4 peering, 1 active+recovering+undersized+degraded+remapped, 12 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 61 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 590 op/s; 30691/207078 objects degraded (14.821%); 3.6 MiB/s, 58 objects/s recovering 2024-09-14T03:56:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:59 smithi160 ceph-mon[88407]: from='client.44645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:59 smithi160 ceph-mon[88407]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 30691/207714 objects degraded (14.776%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:55:59 smithi160 ceph-mon[88407]: osdmap e111: 8 total, 8 up, 8 in 2024-09-14T03:56:00.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:59 smithi114 ceph-mon[103029]: pgmap v250: 89 pgs: 4 peering, 1 active+recovering+undersized+degraded+remapped, 12 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 61 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 590 op/s; 30691/207078 objects degraded (14.821%); 3.6 MiB/s, 58 objects/s recovering 2024-09-14T03:56:00.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:59 smithi114 ceph-mon[103029]: from='client.44645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:00.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:59 smithi114 ceph-mon[103029]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:00.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 30691/207714 objects degraded (14.776%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:00.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:55:59 smithi114 ceph-mon[103029]: osdmap e111: 8 total, 8 up, 8 in 2024-09-14T03:56:00.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:59 smithi114 ceph-mon[106969]: pgmap v250: 89 pgs: 4 peering, 1 active+recovering+undersized+degraded+remapped, 12 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 61 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 590 op/s; 30691/207078 objects degraded (14.821%); 3.6 MiB/s, 58 objects/s recovering 2024-09-14T03:56:00.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:59 smithi114 ceph-mon[106969]: from='client.44645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:00.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:59 smithi114 ceph-mon[106969]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:00.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 30691/207714 objects degraded (14.776%), 12 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:00.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:55:59 smithi114 ceph-mon[106969]: osdmap e111: 8 total, 8 up, 8 in 2024-09-14T03:56:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:56:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:56:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:56:01.696 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:56:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:56:01] "GET /metrics HTTP/1.1" 200 36392 "" "Prometheus/2.51.0" 2024-09-14T03:56:01.950 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:01.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:01 smithi114 ceph-mon[106969]: pgmap v252: 89 pgs: 3 activating+undersized+degraded+remapped, 2 remapped+peering, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 72 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 559 op/s; 11293/202068 objects degraded (5.589%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T03:56:01.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:01 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2890308061' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:01.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:01 smithi114 ceph-mon[103029]: pgmap v252: 89 pgs: 3 activating+undersized+degraded+remapped, 2 remapped+peering, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 72 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 559 op/s; 11293/202068 objects degraded (5.589%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T03:56:01.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:01 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2890308061' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:01 smithi160 ceph-mon[88407]: pgmap v252: 89 pgs: 3 activating+undersized+degraded+remapped, 2 remapped+peering, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 72 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 559 op/s; 11293/202068 objects degraded (5.589%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T03:56:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:01 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2890308061' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:03.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:02 smithi114 ceph-mon[106969]: from='client.54601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:03.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:02 smithi114 ceph-mon[103029]: from='client.54601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:02 smithi160 ceph-mon[88407]: from='client.54601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:04.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:03 smithi114 ceph-mon[106969]: pgmap v253: 89 pgs: 3 activating+undersized+degraded+remapped, 2 remapped+peering, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 451 op/s; 7869/199584 objects degraded (3.943%); 4.0 MiB/s, 64 objects/s recovering 2024-09-14T03:56:04.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:03 smithi114 ceph-mon[106969]: from='client.34827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:04.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:03 smithi114 ceph-mon[103029]: pgmap v253: 89 pgs: 3 activating+undersized+degraded+remapped, 2 remapped+peering, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 451 op/s; 7869/199584 objects degraded (3.943%); 4.0 MiB/s, 64 objects/s recovering 2024-09-14T03:56:04.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:03 smithi114 ceph-mon[103029]: from='client.34827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:03 smithi160 ceph-mon[88407]: pgmap v253: 89 pgs: 3 activating+undersized+degraded+remapped, 2 remapped+peering, 4 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 451 op/s; 7869/199584 objects degraded (3.943%); 4.0 MiB/s, 64 objects/s recovering 2024-09-14T03:56:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:03 smithi160 ceph-mon[88407]: from='client.34827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:05.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7869/199584 objects degraded (3.943%), 7 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:05.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7869/199584 objects degraded (3.943%), 7 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7869/199584 objects degraded (3.943%), 7 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:05.577 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:05 smithi114 ceph-mon[103029]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 8331/193164 objects degraded (4.313%); 7.0 MiB/s, 152 objects/s recovering 2024-09-14T03:56:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:05 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1813329853' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:05 smithi114 ceph-mon[106969]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 8331/193164 objects degraded (4.313%); 7.0 MiB/s, 152 objects/s recovering 2024-09-14T03:56:05.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:05 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1813329853' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:05 smithi160 ceph-mon[88407]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 8331/193164 objects degraded (4.313%); 7.0 MiB/s, 152 objects/s recovering 2024-09-14T03:56:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:05 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1813329853' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:07.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:06 smithi114 ceph-mon[103029]: from='client.34833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:07.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:06 smithi114 ceph-mon[103029]: from='client.44678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:07.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:06 smithi114 ceph-mon[106969]: from='client.34833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:07.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:06 smithi114 ceph-mon[106969]: from='client.44678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:07.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:06 smithi160 ceph-mon[88407]: from='client.34833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:07.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:06 smithi160 ceph-mon[88407]: from='client.44678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:07 smithi114 ceph-mon[103029]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 703 op/s; 8331/189783 objects degraded (4.390%); 6.2 MiB/s, 134 objects/s recovering 2024-09-14T03:56:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:56:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:07 smithi114 ceph-mon[106969]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 703 op/s; 8331/189783 objects degraded (4.390%); 6.2 MiB/s, 134 objects/s recovering 2024-09-14T03:56:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:56:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:07 smithi160 ceph-mon[88407]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 703 op/s; 8331/189783 objects degraded (4.390%); 6.2 MiB/s, 134 objects/s recovering 2024-09-14T03:56:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:56:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:09.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3028393821' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:09.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3028393821' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3028393821' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:09.217 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:09 smithi114 ceph-mon[103029]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 598 op/s; 8331/189123 objects degraded (4.405%); 5.1 MiB/s, 109 objects/s recovering 2024-09-14T03:56:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8331/189123 objects degraded (4.405%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:09 smithi114 ceph-mon[103029]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:09 smithi114 ceph-mon[106969]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 598 op/s; 8331/189123 objects degraded (4.405%); 5.1 MiB/s, 109 objects/s recovering 2024-09-14T03:56:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8331/189123 objects degraded (4.405%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:09 smithi114 ceph-mon[106969]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:09 smithi160 ceph-mon[88407]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 598 op/s; 8331/189123 objects degraded (4.405%); 5.1 MiB/s, 109 objects/s recovering 2024-09-14T03:56:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8331/189123 objects degraded (4.405%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:09 smithi160 ceph-mon[88407]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:56:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:56:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:56:11.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:10 smithi114 ceph-mon[103029]: from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:11.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:10 smithi114 ceph-mon[106969]: from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:10 smithi160 ceph-mon[88407]: from='client.54625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:11.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:56:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:56:11] "GET /metrics HTTP/1.1" 200 36391 "" "Prometheus/2.51.0" 2024-09-14T03:56:12.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:11 smithi114 ceph-mon[103029]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 8037/182340 objects degraded (4.408%); 4.0 MiB/s, 122 objects/s recovering 2024-09-14T03:56:12.121 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:56:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:56:11.813+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:12.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:11 smithi114 ceph-mon[106969]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 8037/182340 objects degraded (4.408%); 4.0 MiB/s, 122 objects/s recovering 2024-09-14T03:56:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:11 smithi160 ceph-mon[88407]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 626 op/s; 8037/182340 objects degraded (4.408%); 4.0 MiB/s, 122 objects/s recovering 2024-09-14T03:56:12.823 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:13.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:56:13.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:12 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:56:13.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:12 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:13.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:12 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2646741432' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:13.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:56:13.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:12 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:56:13.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:12 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:13.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:12 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2646741432' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:56:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:12 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:56:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:12 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:12 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2646741432' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:14.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:13 smithi114 ceph-mon[106969]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 8037/178839 objects degraded (4.494%); 3.7 MiB/s, 113 objects/s recovering 2024-09-14T03:56:14.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:13 smithi114 ceph-mon[106969]: from='client.44705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:14.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:13 smithi114 ceph-mon[106969]: osdmap e112: 8 total, 8 up, 8 in 2024-09-14T03:56:14.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:13 smithi114 ceph-mon[106969]: from='client.54637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:14.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:13 smithi114 ceph-mon[103029]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 8037/178839 objects degraded (4.494%); 3.7 MiB/s, 113 objects/s recovering 2024-09-14T03:56:14.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:13 smithi114 ceph-mon[103029]: from='client.44705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:14.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:13 smithi114 ceph-mon[103029]: osdmap e112: 8 total, 8 up, 8 in 2024-09-14T03:56:14.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:13 smithi114 ceph-mon[103029]: from='client.54637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:13 smithi160 ceph-mon[88407]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 8037/178839 objects degraded (4.494%); 3.7 MiB/s, 113 objects/s recovering 2024-09-14T03:56:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:13 smithi160 ceph-mon[88407]: from='client.44705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:13 smithi160 ceph-mon[88407]: osdmap e112: 8 total, 8 up, 8 in 2024-09-14T03:56:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:13 smithi160 ceph-mon[88407]: from='client.54637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:15.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:14 smithi114 ceph-mon[106969]: osdmap e113: 8 total, 8 up, 8 in 2024-09-14T03:56:15.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8037/178839 objects degraded (4.494%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:15.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:14 smithi114 ceph-mon[103029]: osdmap e113: 8 total, 8 up, 8 in 2024-09-14T03:56:15.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8037/178839 objects degraded (4.494%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:14 smithi160 ceph-mon[88407]: osdmap e113: 8 total, 8 up, 8 in 2024-09-14T03:56:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8037/178839 objects degraded (4.494%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:16.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:15 smithi114 ceph-mon[106969]: pgmap v261: 89 pgs: 1 peering, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 6876/172995 objects degraded (3.975%); 4.4 MiB/s, 149 objects/s recovering 2024-09-14T03:56:16.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1101827311' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:15 smithi114 ceph-mon[103029]: pgmap v261: 89 pgs: 1 peering, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 6876/172995 objects degraded (3.975%); 4.4 MiB/s, 149 objects/s recovering 2024-09-14T03:56:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1101827311' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:15 smithi160 ceph-mon[88407]: pgmap v261: 89 pgs: 1 peering, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 6876/172995 objects degraded (3.975%); 4.4 MiB/s, 149 objects/s recovering 2024-09-14T03:56:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1101827311' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:16.276 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:18.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:17 smithi114 ceph-mon[103029]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:18.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:17 smithi114 ceph-mon[103029]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 6665/169182 objects degraded (3.940%); 6.1 MiB/s, 209 objects/s recovering 2024-09-14T03:56:18.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:17 smithi114 ceph-mon[103029]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:17 smithi114 ceph-mon[106969]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:17 smithi114 ceph-mon[106969]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 6665/169182 objects degraded (3.940%); 6.1 MiB/s, 209 objects/s recovering 2024-09-14T03:56:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:17 smithi114 ceph-mon[106969]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:17 smithi160 ceph-mon[88407]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:17 smithi160 ceph-mon[88407]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 6665/169182 objects degraded (3.940%); 6.1 MiB/s, 209 objects/s recovering 2024-09-14T03:56:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:17 smithi160 ceph-mon[88407]: from='client.44720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:20.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:19 smithi114 ceph-mon[106969]: pgmap v263: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 570 op/s; 6665/168633 objects degraded (3.952%); 3.8 MiB/s, 128 objects/s recovering 2024-09-14T03:56:20.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6665/168633 objects degraded (3.952%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:20.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2487497938' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:19 smithi114 ceph-mon[103029]: pgmap v263: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 570 op/s; 6665/168633 objects degraded (3.952%); 3.8 MiB/s, 128 objects/s recovering 2024-09-14T03:56:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6665/168633 objects degraded (3.952%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:19 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2487497938' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:19 smithi160 ceph-mon[88407]: pgmap v263: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 570 op/s; 6665/168633 objects degraded (3.952%); 3.8 MiB/s, 128 objects/s recovering 2024-09-14T03:56:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6665/168633 objects degraded (3.952%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:19 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2487497938' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:56:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:56:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:56:21.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:20 smithi114 ceph-mon[106969]: from='client.54661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:21.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:20 smithi114 ceph-mon[103029]: from='client.54661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:21.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:20 smithi160 ceph-mon[88407]: from='client.54661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:21.843 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:56:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:56:21] "GET /metrics HTTP/1.1" 200 36391 "" "Prometheus/2.51.0" 2024-09-14T03:56:22.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:21 smithi114 ceph-mon[106969]: from='client.44726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:22.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:21 smithi114 ceph-mon[106969]: pgmap v264: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 6665/162882 objects degraded (4.092%); 3.8 MiB/s, 186 objects/s recovering 2024-09-14T03:56:22.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:21 smithi114 ceph-mon[106969]: osdmap e114: 8 total, 8 up, 8 in 2024-09-14T03:56:22.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:21 smithi114 ceph-mon[103029]: from='client.44726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:22.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:21 smithi114 ceph-mon[103029]: pgmap v264: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 6665/162882 objects degraded (4.092%); 3.8 MiB/s, 186 objects/s recovering 2024-09-14T03:56:22.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:21 smithi114 ceph-mon[103029]: osdmap e114: 8 total, 8 up, 8 in 2024-09-14T03:56:22.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:21 smithi160 ceph-mon[88407]: from='client.44726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:22.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:21 smithi160 ceph-mon[88407]: pgmap v264: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 6665/162882 objects degraded (4.092%); 3.8 MiB/s, 186 objects/s recovering 2024-09-14T03:56:22.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:21 smithi160 ceph-mon[88407]: osdmap e114: 8 total, 8 up, 8 in 2024-09-14T03:56:23.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:22 smithi114 ceph-mon[103029]: osdmap e115: 8 total, 8 up, 8 in 2024-09-14T03:56:23.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:22 smithi114 ceph-mon[106969]: osdmap e115: 8 total, 8 up, 8 in 2024-09-14T03:56:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:22 smithi160 ceph-mon[88407]: osdmap e115: 8 total, 8 up, 8 in 2024-09-14T03:56:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:23.521 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:24.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:23 smithi114 ceph-mon[103029]: pgmap v267: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 584 op/s; 6451/158958 objects degraded (4.058%); 3.3 MiB/s, 145 objects/s recovering 2024-09-14T03:56:24.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/156780364' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:24.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:23 smithi114 ceph-mon[106969]: pgmap v267: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 584 op/s; 6451/158958 objects degraded (4.058%); 3.3 MiB/s, 145 objects/s recovering 2024-09-14T03:56:24.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:23 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/156780364' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:23 smithi160 ceph-mon[88407]: pgmap v267: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 584 op/s; 6451/158958 objects degraded (4.058%); 3.3 MiB/s, 145 objects/s recovering 2024-09-14T03:56:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:23 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/156780364' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:24 smithi160 ceph-mon[88407]: from='client.44732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:24 smithi160 ceph-mon[88407]: from='client.44738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6451/158958 objects degraded (4.058%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:25.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:24 smithi114 ceph-mon[103029]: from='client.44732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:25.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:24 smithi114 ceph-mon[103029]: from='client.44738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:25.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6451/158958 objects degraded (4.058%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:25.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:24 smithi114 ceph-mon[106969]: from='client.44732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:25.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:24 smithi114 ceph-mon[106969]: from='client.44738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:25.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6451/158958 objects degraded (4.058%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:25 smithi160 ceph-mon[88407]: pgmap v268: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 6451/154854 objects degraded (4.166%); 1.7 MiB/s, 104 objects/s recovering 2024-09-14T03:56:26.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:25 smithi114 ceph-mon[103029]: pgmap v268: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 6451/154854 objects degraded (4.166%); 1.7 MiB/s, 104 objects/s recovering 2024-09-14T03:56:26.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:25 smithi114 ceph-mon[106969]: pgmap v268: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 6451/154854 objects degraded (4.166%); 1.7 MiB/s, 104 objects/s recovering 2024-09-14T03:56:27.106 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:27.107 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:26 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2160666677' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:27.107 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:26 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:56:27.107 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:56:26 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:56:26.834+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:27.108 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:26 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2160666677' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:27.108 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:26 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:56:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:26 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2160666677' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:26 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:56:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:27 smithi160 ceph-mon[88407]: pgmap v269: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 6451/152559 objects degraded (4.229%); 1.7 MiB/s, 104 objects/s recovering 2024-09-14T03:56:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:27 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:56:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:27 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:27 smithi160 ceph-mon[88407]: from='client.54676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:27 smithi114 ceph-mon[106969]: pgmap v269: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 6451/152559 objects degraded (4.229%); 1.7 MiB/s, 104 objects/s recovering 2024-09-14T03:56:28.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:27 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:56:28.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:27 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:28.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:27 smithi114 ceph-mon[106969]: from='client.54676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:27 smithi114 ceph-mon[103029]: pgmap v269: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 6451/152559 objects degraded (4.229%); 1.7 MiB/s, 104 objects/s recovering 2024-09-14T03:56:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:27 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:56:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:27 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:27 smithi114 ceph-mon[103029]: from='client.54676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:28 smithi160 ceph-mon[88407]: from='client.44747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:29.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:28 smithi114 ceph-mon[103029]: from='client.44747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:29.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:28 smithi114 ceph-mon[106969]: from='client.44747 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:30.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:29 smithi160 ceph-mon[88407]: pgmap v270: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 6193/148596 objects degraded (4.168%); 3.7 MiB/s, 77 objects/s recovering 2024-09-14T03:56:30.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6193/148596 objects degraded (4.168%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:29 smithi114 ceph-mon[103029]: pgmap v270: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 6193/148596 objects degraded (4.168%); 3.7 MiB/s, 77 objects/s recovering 2024-09-14T03:56:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6193/148596 objects degraded (4.168%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:29 smithi114 ceph-mon[106969]: pgmap v270: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 595 op/s; 6193/148596 objects degraded (4.168%); 3.7 MiB/s, 77 objects/s recovering 2024-09-14T03:56:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6193/148596 objects degraded (4.168%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:56:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:56:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:56:30.724 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:31.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:30 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/481861337' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:31.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:30 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/481861337' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:31.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:30 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/481861337' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:31.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:56:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:56:31] "GET /metrics HTTP/1.1" 200 36394 "" "Prometheus/2.51.0" 2024-09-14T03:56:32.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:31 smithi114 ceph-mon[103029]: pgmap v271: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 6193/142281 objects degraded (4.353%); 3.1 MiB/s, 64 objects/s recovering 2024-09-14T03:56:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:31 smithi114 ceph-mon[103029]: from='client.34905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:32.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:31 smithi114 ceph-mon[106969]: pgmap v271: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 6193/142281 objects degraded (4.353%); 3.1 MiB/s, 64 objects/s recovering 2024-09-14T03:56:32.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:31 smithi114 ceph-mon[106969]: from='client.34905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:31 smithi160 ceph-mon[88407]: pgmap v271: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 6193/142281 objects degraded (4.353%); 3.1 MiB/s, 64 objects/s recovering 2024-09-14T03:56:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:31 smithi160 ceph-mon[88407]: from='client.34905 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:33.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:32 smithi114 ceph-mon[103029]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:33.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:32 smithi114 ceph-mon[106969]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:33.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:32 smithi160 ceph-mon[88407]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:34.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:33 smithi114 ceph-mon[103029]: pgmap v272: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 6020/138036 objects degraded (4.361%); 2.5 MiB/s, 54 objects/s recovering 2024-09-14T03:56:34.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:33 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1465306015' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:34.355 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:33 smithi114 ceph-mon[106969]: pgmap v272: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 6020/138036 objects degraded (4.361%); 2.5 MiB/s, 54 objects/s recovering 2024-09-14T03:56:34.355 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:33 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1465306015' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:34.355 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:33 smithi160 ceph-mon[88407]: pgmap v272: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 6020/138036 objects degraded (4.361%); 2.5 MiB/s, 54 objects/s recovering 2024-09-14T03:56:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:33 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1465306015' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:35.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6020/138036 objects degraded (4.361%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6020/138036 objects degraded (4.361%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6020/138036 objects degraded (4.361%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:36.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:35 smithi114 ceph-mon[103029]: from='client.34917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:36.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:35 smithi114 ceph-mon[103029]: pgmap v273: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 6020/133023 objects degraded (4.526%); 2.2 MiB/s, 48 objects/s recovering 2024-09-14T03:56:36.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:35 smithi114 ceph-mon[103029]: from='client.44768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:35 smithi114 ceph-mon[106969]: from='client.34917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:35 smithi114 ceph-mon[106969]: pgmap v273: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 6020/133023 objects degraded (4.526%); 2.2 MiB/s, 48 objects/s recovering 2024-09-14T03:56:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:35 smithi114 ceph-mon[106969]: from='client.44768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:35 smithi160 ceph-mon[88407]: from='client.34917 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:35 smithi160 ceph-mon[88407]: pgmap v273: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 6020/133023 objects degraded (4.526%); 2.2 MiB/s, 48 objects/s recovering 2024-09-14T03:56:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:35 smithi160 ceph-mon[88407]: from='client.44768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:37.994 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:37 smithi114 ceph-mon[103029]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 6020/130140 objects degraded (4.626%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:56:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:37 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/100409296' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:37 smithi114 ceph-mon[106969]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 6020/130140 objects degraded (4.626%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:56:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:37 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/100409296' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:37 smithi160 ceph-mon[88407]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 6020/130140 objects degraded (4.626%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:56:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:37 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/100409296' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:39.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:38 smithi114 ceph-mon[103029]: from='client.44771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:39.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:38 smithi114 ceph-mon[106969]: from='client.44771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:39.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:38 smithi160 ceph-mon[88407]: from='client.44771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:40.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:39 smithi114 ceph-mon[106969]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 5823/125727 objects degraded (4.631%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:56:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:39 smithi114 ceph-mon[106969]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:39 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 5823/125727 objects degraded (4.631%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:39 smithi114 ceph-mon[103029]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 5823/125727 objects degraded (4.631%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:56:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:39 smithi114 ceph-mon[103029]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:39 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 5823/125727 objects degraded (4.631%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:56:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:56:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:56:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:39 smithi160 ceph-mon[88407]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 5823/125727 objects degraded (4.631%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:56:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:39 smithi160 ceph-mon[88407]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:39 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 5823/125727 objects degraded (4.631%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:41.621 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:41.858 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:56:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:56:41] "GET /metrics HTTP/1.1" 200 36386 "" "Prometheus/2.51.0" 2024-09-14T03:56:41.858 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:56:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:56:41.856+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:41 smithi114 ceph-mon[103029]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 809 op/s; 5823/119445 objects degraded (4.875%); 1.9 MiB/s, 30 objects/s recovering 2024-09-14T03:56:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:41 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2678308520' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:41 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:56:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:41 smithi114 ceph-mon[106969]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 809 op/s; 5823/119445 objects degraded (4.875%); 1.9 MiB/s, 30 objects/s recovering 2024-09-14T03:56:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:41 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2678308520' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:41 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:56:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:41 smithi160 ceph-mon[88407]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 809 op/s; 5823/119445 objects degraded (4.875%); 1.9 MiB/s, 30 objects/s recovering 2024-09-14T03:56:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:41 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2678308520' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:41 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:56:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:42 smithi114 ceph-mon[103029]: from='client.54721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:42 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:56:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:42 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:42 smithi114 ceph-mon[103029]: from='client.34947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:43 smithi114 ceph-mon[106969]: from='client.54721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:43 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:56:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:43 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:43 smithi114 ceph-mon[106969]: from='client.34947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:42 smithi160 ceph-mon[88407]: from='client.54721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:42 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:56:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:42 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:56:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:42 smithi160 ceph-mon[88407]: from='client.34947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:44 smithi114 ceph-mon[103029]: pgmap v277: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 5617/115254 objects degraded (4.874%); 3.0 MiB/s, 47 objects/s recovering 2024-09-14T03:56:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:44 smithi114 ceph-mon[106969]: pgmap v277: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 5617/115254 objects degraded (4.874%); 3.0 MiB/s, 47 objects/s recovering 2024-09-14T03:56:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:44 smithi160 ceph-mon[88407]: pgmap v277: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 5617/115254 objects degraded (4.874%); 3.0 MiB/s, 47 objects/s recovering 2024-09-14T03:56:45.228 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:45.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:45 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 5617/115254 objects degraded (4.874%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:45.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:45 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3898004910' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:45.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:45 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 5617/115254 objects degraded (4.874%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:45.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:45 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3898004910' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:45.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:45 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 5617/115254 objects degraded (4.874%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:45.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:45 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3898004910' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:46 smithi114 ceph-mon[103029]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 5617/110043 objects degraded (5.104%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:56:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:46 smithi114 ceph-mon[103029]: from='client.44801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:46.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:46 smithi114 ceph-mon[106969]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 5617/110043 objects degraded (5.104%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:56:46.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:46 smithi114 ceph-mon[106969]: from='client.44801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:46 smithi160 ceph-mon[88407]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 5617/110043 objects degraded (5.104%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:56:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:46 smithi160 ceph-mon[88407]: from='client.44801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:47.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:47 smithi114 ceph-mon[103029]: from='client.44807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:47 smithi114 ceph-mon[106969]: from='client.44807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:47 smithi160 ceph-mon[88407]: from='client.44807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:48.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:48 smithi114 ceph-mon[103029]: pgmap v279: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 5617/107235 objects degraded (5.238%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:56:48.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:48 smithi114 ceph-mon[106969]: pgmap v279: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 5617/107235 objects degraded (5.238%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:56:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:48 smithi160 ceph-mon[88407]: pgmap v279: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 5617/107235 objects degraded (5.238%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:56:48.853 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:49.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:49 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/603428963' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:49.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:49 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/603428963' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:49 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/603428963' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:50.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:50 smithi114 ceph-mon[106969]: pgmap v280: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 5391/102978 objects degraded (5.235%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:56:50.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:50 smithi114 ceph-mon[106969]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:50.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:50 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 5391/102978 objects degraded (5.235%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:50 smithi114 ceph-mon[103029]: pgmap v280: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 5391/102978 objects degraded (5.235%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:56:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:50 smithi114 ceph-mon[103029]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:50 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 5391/102978 objects degraded (5.235%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:56:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:56:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:56:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:50 smithi160 ceph-mon[88407]: pgmap v280: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 5391/102978 objects degraded (5.235%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T03:56:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:50 smithi160 ceph-mon[88407]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:50.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:50 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 5391/102978 objects degraded (5.235%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:56:51.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:51 smithi114 ceph-mon[106969]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:51 smithi114 ceph-mon[103029]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:51 smithi160 ceph-mon[88407]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:51.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:56:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:56:51] "GET /metrics HTTP/1.1" 200 36386 "" "Prometheus/2.51.0" 2024-09-14T03:56:52.109 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:52.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:52 smithi114 ceph-mon[103029]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 5391/96687 objects degraded (5.576%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:56:52.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:52 smithi114 ceph-mon[103029]: osdmap e116: 8 total, 8 up, 8 in 2024-09-14T03:56:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:52 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2554807892' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:52.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:52 smithi114 ceph-mon[106969]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 5391/96687 objects degraded (5.576%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:56:52.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:52 smithi114 ceph-mon[106969]: osdmap e116: 8 total, 8 up, 8 in 2024-09-14T03:56:52.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:52 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2554807892' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:52 smithi160 ceph-mon[88407]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.0 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 5391/96687 objects degraded (5.576%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:56:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:52 smithi160 ceph-mon[88407]: osdmap e116: 8 total, 8 up, 8 in 2024-09-14T03:56:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:52 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2554807892' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:53.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:53 smithi114 ceph-mon[103029]: osdmap e117: 8 total, 8 up, 8 in 2024-09-14T03:56:53.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:53 smithi114 ceph-mon[103029]: from='client.34974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:53.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:53 smithi114 ceph-mon[106969]: osdmap e117: 8 total, 8 up, 8 in 2024-09-14T03:56:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:53 smithi114 ceph-mon[106969]: from='client.34974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:53 smithi160 ceph-mon[88407]: osdmap e117: 8 total, 8 up, 8 in 2024-09-14T03:56:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:53 smithi160 ceph-mon[88407]: from='client.34974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:56:54.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:54 smithi114 ceph-mon[103029]: pgmap v284: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 4333/92397 objects degraded (4.690%); 3.1 MiB/s, 64 objects/s recovering 2024-09-14T03:56:54.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:54 smithi114 ceph-mon[103029]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:54 smithi114 ceph-mon[106969]: pgmap v284: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 4333/92397 objects degraded (4.690%); 3.1 MiB/s, 64 objects/s recovering 2024-09-14T03:56:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:54 smithi114 ceph-mon[106969]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:54 smithi160 ceph-mon[88407]: pgmap v284: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 4333/92397 objects degraded (4.690%); 3.1 MiB/s, 64 objects/s recovering 2024-09-14T03:56:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:54 smithi160 ceph-mon[88407]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:55.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:55 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4333/92397 objects degraded (4.690%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T03:56:55.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:55 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4333/92397 objects degraded (4.690%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T03:56:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:55 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4333/92397 objects degraded (4.690%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T03:56:55.708 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:56.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:56 smithi114 ceph-mon[106969]: pgmap v285: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 4069/88314 objects degraded (4.607%); 3.8 MiB/s, 97 objects/s recovering 2024-09-14T03:56:56.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:56 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1885808765' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:56.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:56 smithi114 ceph-mon[103029]: pgmap v285: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 4069/88314 objects degraded (4.607%); 3.8 MiB/s, 97 objects/s recovering 2024-09-14T03:56:56.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:56 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1885808765' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:56 smithi160 ceph-mon[88407]: pgmap v285: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 4069/88314 objects degraded (4.607%); 3.8 MiB/s, 97 objects/s recovering 2024-09-14T03:56:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:56 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1885808765' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:57.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:57 smithi114 ceph-mon[103029]: from='client.44840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:57.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:57 smithi114 ceph-mon[103029]: from='client.54769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:57.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:57 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:56:57.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:57 smithi114 ceph-mon[106969]: from='client.44840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:57.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:57 smithi114 ceph-mon[106969]: from='client.54769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:57.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:57 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:56:57.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:57 smithi160 ceph-mon[88407]: from='client.44840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:57.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:57 smithi160 ceph-mon[88407]: from='client.54769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:56:57.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:57 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:56:58.362 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:58 smithi114 ceph-mon[103029]: pgmap v286: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 4069/85878 objects degraded (4.738%); 2.0 MiB/s, 69 objects/s recovering 2024-09-14T03:56:58.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:58 smithi114 ceph-mon[106969]: pgmap v286: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 4069/85878 objects degraded (4.738%); 2.0 MiB/s, 69 objects/s recovering 2024-09-14T03:56:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:58 smithi160 ceph-mon[88407]: pgmap v286: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 712 op/s; 4069/85878 objects degraded (4.738%); 2.0 MiB/s, 69 objects/s recovering 2024-09-14T03:56:59.371 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:56:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:59 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/485876311' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:56:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4069/85878 objects degraded (4.738%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T03:56:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:59 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/485876311' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:56:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4069/85878 objects degraded (4.738%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T03:56:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:59 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/485876311' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:56:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:56:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4069/85878 objects degraded (4.738%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T03:57:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:00 smithi160 ceph-mon[88407]: pgmap v287: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 4069/82113 objects degraded (4.955%); 2.0 MiB/s, 107 objects/s recovering 2024-09-14T03:57:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:57:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:57:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:57:00.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:00 smithi114 ceph-mon[106969]: pgmap v287: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 4069/82113 objects degraded (4.955%); 2.0 MiB/s, 107 objects/s recovering 2024-09-14T03:57:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:00 smithi114 ceph-mon[103029]: pgmap v287: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.7 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 606 op/s; 4069/82113 objects degraded (4.955%); 2.0 MiB/s, 107 objects/s recovering 2024-09-14T03:57:01.138 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:01 smithi160 ceph-mon[88407]: from='client.54781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:01.138 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:01 smithi160 ceph-mon[88407]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:01.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:01 smithi114 ceph-mon[106969]: from='client.54781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:01.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:01 smithi114 ceph-mon[106969]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:01.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:01 smithi114 ceph-mon[103029]: from='client.54781 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:01.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:01 smithi114 ceph-mon[103029]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:01.800 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:57:01] "GET /metrics HTTP/1.1" 200 36383 "" "Prometheus/2.51.0" 2024-09-14T03:57:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:02 smithi160 ceph-mon[88407]: pgmap v288: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 3817/76422 objects degraded (4.995%); 1.7 MiB/s, 118 objects/s recovering 2024-09-14T03:57:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:02 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:02 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:02.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:02 smithi114 ceph-mon[106969]: pgmap v288: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 3817/76422 objects degraded (4.995%); 1.7 MiB/s, 118 objects/s recovering 2024-09-14T03:57:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:02 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:02 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:02 smithi114 ceph-mon[103029]: pgmap v288: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.6 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 3817/76422 objects degraded (4.995%); 1.7 MiB/s, 118 objects/s recovering 2024-09-14T03:57:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:02 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:02 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:03.078 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/682322800' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:57:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:57:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:57:03.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:57:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:57:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:57:03.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:03.372 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:57:03.026+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:57:03.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:03.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:03.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/682322800' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:03.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:57:03.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:57:03.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:03.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:57:03.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:57:03.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:57:03.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:57:03.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/682322800' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:57:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:57:03.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:03.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:57:03.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:57:03.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:57:03.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:57:03.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:04.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:04 smithi114 ceph-mon[106969]: pgmap v289: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 3817/72666 objects degraded (5.253%); 517 KiB/s, 98 objects/s recovering 2024-09-14T03:57:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:04 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:04 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:57:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:04 smithi114 ceph-mon[106969]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3817/72666 objects degraded (5.253%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:04.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:04 smithi114 ceph-mon[103029]: pgmap v289: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 3817/72666 objects degraded (5.253%); 517 KiB/s, 98 objects/s recovering 2024-09-14T03:57:04.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:04 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:04.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:04 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:57:04.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:04 smithi114 ceph-mon[103029]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:04.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3817/72666 objects degraded (5.253%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:04 smithi160 ceph-mon[88407]: pgmap v289: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.5 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 3817/72666 objects degraded (5.253%); 517 KiB/s, 98 objects/s recovering 2024-09-14T03:57:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:04 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:04 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:57:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:04 smithi160 ceph-mon[88407]: from='client.35004 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3817/72666 objects degraded (5.253%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:05.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:05 smithi114 ceph-mon[103029]: from='client.44870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:05.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:05 smithi114 ceph-mon[106969]: from='client.44870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:05 smithi160 ceph-mon[88407]: from='client.44870 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:06.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:06 smithi114 ceph-mon[106969]: pgmap v290: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 3556/68478 objects degraded (5.193%); 448 KiB/s, 107 objects/s recovering 2024-09-14T03:57:06.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2438786751' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:06 smithi114 ceph-mon[103029]: pgmap v290: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 3556/68478 objects degraded (5.193%); 448 KiB/s, 107 objects/s recovering 2024-09-14T03:57:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2438786751' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:06 smithi160 ceph-mon[88407]: pgmap v290: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 664 op/s; 3556/68478 objects degraded (5.193%); 448 KiB/s, 107 objects/s recovering 2024-09-14T03:57:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2438786751' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:06.721 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:07 smithi114 ceph-mon[103029]: pgmap v291: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 3556/65700 objects degraded (5.412%); 0 B/s, 85 objects/s recovering 2024-09-14T03:57:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:07 smithi114 ceph-mon[103029]: from='client.44882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:07 smithi114 ceph-mon[103029]: from='client.54811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:07 smithi114 ceph-mon[106969]: pgmap v291: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 3556/65700 objects degraded (5.412%); 0 B/s, 85 objects/s recovering 2024-09-14T03:57:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:07 smithi114 ceph-mon[106969]: from='client.44882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:07.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:07 smithi114 ceph-mon[106969]: from='client.54811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:07 smithi160 ceph-mon[88407]: pgmap v291: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 3556/65700 objects degraded (5.412%); 0 B/s, 85 objects/s recovering 2024-09-14T03:57:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:07 smithi160 ceph-mon[88407]: from='client.44882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:07 smithi160 ceph-mon[88407]: from='client.54811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:08.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:09.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:09 smithi114 ceph-mon[103029]: osdmap e118: 8 total, 8 up, 8 in 2024-09-14T03:57:09.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:09 smithi114 ceph-mon[103029]: pgmap v293: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.3 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 3556/61623 objects degraded (5.771%); 0 B/s, 93 objects/s recovering 2024-09-14T03:57:09.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3556/65700 objects degraded (5.412%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:09.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:09 smithi114 ceph-mon[106969]: osdmap e118: 8 total, 8 up, 8 in 2024-09-14T03:57:09.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:09 smithi114 ceph-mon[106969]: pgmap v293: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.3 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 3556/61623 objects degraded (5.771%); 0 B/s, 93 objects/s recovering 2024-09-14T03:57:09.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3556/65700 objects degraded (5.412%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:09.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:09 smithi160 ceph-mon[88407]: osdmap e118: 8 total, 8 up, 8 in 2024-09-14T03:57:09.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:09 smithi160 ceph-mon[88407]: pgmap v293: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.3 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 3556/61623 objects degraded (5.771%); 0 B/s, 93 objects/s recovering 2024-09-14T03:57:09.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3556/65700 objects degraded (5.412%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:10.297 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:10.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:10 smithi114 ceph-mon[103029]: osdmap e119: 8 total, 8 up, 8 in 2024-09-14T03:57:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:10 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/800864357' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:10.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:10 smithi114 ceph-mon[106969]: osdmap e119: 8 total, 8 up, 8 in 2024-09-14T03:57:10.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:10 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/800864357' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:57:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:57:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:57:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:10 smithi160 ceph-mon[88407]: osdmap e119: 8 total, 8 up, 8 in 2024-09-14T03:57:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:10 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/800864357' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:11.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:11 smithi114 ceph-mon[103029]: from='client.44900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:11.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:11 smithi114 ceph-mon[103029]: pgmap v295: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 3278/55533 objects degraded (5.903%); 0 B/s, 95 objects/s recovering 2024-09-14T03:57:11.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:11 smithi114 ceph-mon[103029]: from='client.35025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:11.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:57:11] "GET /metrics HTTP/1.1" 200 36364 "" "Prometheus/2.51.0" 2024-09-14T03:57:11.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:11 smithi114 ceph-mon[106969]: from='client.44900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:11.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:11 smithi114 ceph-mon[106969]: pgmap v295: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 3278/55533 objects degraded (5.903%); 0 B/s, 95 objects/s recovering 2024-09-14T03:57:11.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:11 smithi114 ceph-mon[106969]: from='client.35025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:11.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:11 smithi160 ceph-mon[88407]: from='client.44900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:11.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:11 smithi160 ceph-mon[88407]: pgmap v295: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 3278/55533 objects degraded (5.903%); 0 B/s, 95 objects/s recovering 2024-09-14T03:57:11.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:11 smithi160 ceph-mon[88407]: from='client.35025 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:13 smithi114 ceph-mon[103029]: pgmap v296: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 3278/52101 objects degraded (6.292%); 0 B/s, 66 objects/s recovering 2024-09-14T03:57:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:13 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/592132330' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:13 smithi114 ceph-mon[106969]: pgmap v296: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 3278/52101 objects degraded (6.292%); 0 B/s, 66 objects/s recovering 2024-09-14T03:57:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:13 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/592132330' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:13 smithi160 ceph-mon[88407]: pgmap v296: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 3278/52101 objects degraded (6.292%); 0 B/s, 66 objects/s recovering 2024-09-14T03:57:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:13 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/592132330' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:13.922 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:14.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:14 smithi114 ceph-mon[103029]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:14.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3278/52101 objects degraded (6.292%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:14.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:14 smithi114 ceph-mon[106969]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:14.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3278/52101 objects degraded (6.292%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:14 smithi160 ceph-mon[88407]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3278/52101 objects degraded (6.292%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:15.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:15 smithi114 ceph-mon[106969]: from='client.35031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:15 smithi114 ceph-mon[106969]: pgmap v297: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 994 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 2809/47781 objects degraded (5.879%); 0 B/s, 125 objects/s recovering 2024-09-14T03:57:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:15 smithi114 ceph-mon[106969]: osdmap e120: 8 total, 8 up, 8 in 2024-09-14T03:57:15.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:15 smithi114 ceph-mon[103029]: from='client.35031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:15.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:15 smithi114 ceph-mon[103029]: pgmap v297: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 994 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 2809/47781 objects degraded (5.879%); 0 B/s, 125 objects/s recovering 2024-09-14T03:57:15.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:15 smithi114 ceph-mon[103029]: osdmap e120: 8 total, 8 up, 8 in 2024-09-14T03:57:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:15 smithi160 ceph-mon[88407]: from='client.35031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:15 smithi160 ceph-mon[88407]: pgmap v297: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 994 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 2809/47781 objects degraded (5.879%); 0 B/s, 125 objects/s recovering 2024-09-14T03:57:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:15 smithi160 ceph-mon[88407]: osdmap e120: 8 total, 8 up, 8 in 2024-09-14T03:57:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:16 smithi114 ceph-mon[106969]: osdmap e121: 8 total, 8 up, 8 in 2024-09-14T03:57:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:16 smithi114 ceph-mon[103029]: osdmap e121: 8 total, 8 up, 8 in 2024-09-14T03:57:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:16 smithi160 ceph-mon[88407]: osdmap e121: 8 total, 8 up, 8 in 2024-09-14T03:57:17.551 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:17.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:17 smithi114 ceph-mon[106969]: pgmap v300: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 925 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 1897/44445 objects degraded (4.268%); 0 B/s, 70 objects/s recovering 2024-09-14T03:57:17.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:17 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1773408401' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:17.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:17 smithi114 ceph-mon[103029]: pgmap v300: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 925 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 1897/44445 objects degraded (4.268%); 0 B/s, 70 objects/s recovering 2024-09-14T03:57:17.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:17 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1773408401' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:17.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:17 smithi160 ceph-mon[88407]: pgmap v300: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 925 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 602 op/s; 1897/44445 objects degraded (4.268%); 0 B/s, 70 objects/s recovering 2024-09-14T03:57:17.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:17 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1773408401' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:18.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:57:18.048+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:57:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-mon[106969]: from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:18.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:57:18.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-mon[106969]: from='client.44924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:18.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-mon[103029]: from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:18.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:18.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:18.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:57:18.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:18 smithi114 ceph-mon[103029]: from='client.44924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:18 smithi160 ceph-mon[88407]: from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:18 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:18 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-09-14T03:57:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:18 smithi160 ceph-mon[88407]: from='client.44924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:19.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:19 smithi114 ceph-mon[106969]: pgmap v301: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 829 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 654 op/s; 1897/39831 objects degraded (4.763%); 0 B/s, 63 objects/s recovering 2024-09-14T03:57:19.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1897/39831 objects degraded (4.763%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:19.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:19 smithi114 ceph-mon[103029]: pgmap v301: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 829 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 654 op/s; 1897/39831 objects degraded (4.763%); 0 B/s, 63 objects/s recovering 2024-09-14T03:57:19.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1897/39831 objects degraded (4.763%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:19 smithi160 ceph-mon[88407]: pgmap v301: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 829 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 654 op/s; 1897/39831 objects degraded (4.763%); 0 B/s, 63 objects/s recovering 2024-09-14T03:57:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1897/39831 objects degraded (4.763%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:57:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:57:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:57:21.187 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:21.592 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:57:21] "GET /metrics HTTP/1.1" 200 36364 "" "Prometheus/2.51.0" 2024-09-14T03:57:21.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:21 smithi114 ceph-mon[106969]: pgmap v302: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 756 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 655 op/s; 1841/36357 objects degraded (5.064%); 0 B/s, 129 objects/s recovering 2024-09-14T03:57:21.906 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2944680224' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:21.906 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:21 smithi114 ceph-mon[106969]: from='client.44930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:21.906 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:21 smithi114 ceph-mon[103029]: pgmap v302: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 756 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 655 op/s; 1841/36357 objects degraded (5.064%); 0 B/s, 129 objects/s recovering 2024-09-14T03:57:21.907 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2944680224' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:21.907 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:21 smithi114 ceph-mon[103029]: from='client.44930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:21.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:21 smithi160 ceph-mon[88407]: pgmap v302: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 756 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 655 op/s; 1841/36357 objects degraded (5.064%); 0 B/s, 129 objects/s recovering 2024-09-14T03:57:21.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2944680224' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:21.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:21 smithi160 ceph-mon[88407]: from='client.44930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:22.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:22 smithi114 ceph-mon[106969]: from='client.35055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:22 smithi114 ceph-mon[103029]: from='client.35055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:22 smithi160 ceph-mon[88407]: from='client.35055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:23.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:23 smithi114 ceph-mon[106969]: pgmap v303: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 666 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 1841/32001 objects degraded (5.753%); 0 B/s, 70 objects/s recovering 2024-09-14T03:57:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:23 smithi114 ceph-mon[103029]: pgmap v303: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 666 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 1841/32001 objects degraded (5.753%); 0 B/s, 70 objects/s recovering 2024-09-14T03:57:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:23 smithi160 ceph-mon[88407]: pgmap v303: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 666 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 1841/32001 objects degraded (5.753%); 0 B/s, 70 objects/s recovering 2024-09-14T03:57:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:24.747 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:24.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1841/32001 objects degraded (5.753%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:24.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:24 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1923388970' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:24.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1841/32001 objects degraded (5.753%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:24.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:24 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1923388970' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1841/32001 objects degraded (5.753%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:24 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1923388970' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:25.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:25 smithi114 ceph-mon[106969]: pgmap v304: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 556 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 1841/26754 objects degraded (6.881%); 0 B/s, 56 objects/s recovering 2024-09-14T03:57:25.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:25 smithi114 ceph-mon[106969]: from='client.54877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:25.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:25 smithi114 ceph-mon[103029]: pgmap v304: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 556 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 1841/26754 objects degraded (6.881%); 0 B/s, 56 objects/s recovering 2024-09-14T03:57:25.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:25 smithi114 ceph-mon[103029]: from='client.54877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:25 smithi160 ceph-mon[88407]: pgmap v304: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 556 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 1841/26754 objects degraded (6.881%); 0 B/s, 56 objects/s recovering 2024-09-14T03:57:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:25 smithi160 ceph-mon[88407]: from='client.54877 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:26 smithi160 ceph-mon[88407]: from='client.54883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:26 smithi114 ceph-mon[106969]: from='client.54883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:26 smithi114 ceph-mon[103029]: from='client.54883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:27.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:27 smithi160 ceph-mon[88407]: pgmap v305: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 481 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 1841/23148 objects degraded (7.953%); 0 B/s, 72 objects/s recovering 2024-09-14T03:57:28.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:27 smithi114 ceph-mon[106969]: pgmap v305: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 481 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 1841/23148 objects degraded (7.953%); 0 B/s, 72 objects/s recovering 2024-09-14T03:57:28.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:27 smithi114 ceph-mon[103029]: pgmap v305: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 481 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 686 op/s; 1841/23148 objects degraded (7.953%); 0 B/s, 72 objects/s recovering 2024-09-14T03:57:28.385 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:28.639 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 03:57:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=cleanup t=2024-09-14T03:57:28.192990407Z level=info msg="Completed cleanup jobs" duration=12.714188ms 2024-09-14T03:57:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:28 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2166126590' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:29.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:28 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2166126590' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:29.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:28 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2166126590' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:29 smithi160 ceph-mon[88407]: from='client.44954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:29 smithi160 ceph-mon[88407]: pgmap v306: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 406 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 1841/19524 objects degraded (9.429%); 0 B/s, 65 objects/s recovering 2024-09-14T03:57:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:29 smithi160 ceph-mon[88407]: osdmap e122: 8 total, 8 up, 8 in 2024-09-14T03:57:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:29 smithi160 ceph-mon[88407]: from='client.44960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1841/19524 objects degraded (9.429%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:30.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:29 smithi114 ceph-mon[106969]: from='client.44954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:29 smithi114 ceph-mon[106969]: pgmap v306: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 406 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 1841/19524 objects degraded (9.429%); 0 B/s, 65 objects/s recovering 2024-09-14T03:57:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:29 smithi114 ceph-mon[106969]: osdmap e122: 8 total, 8 up, 8 in 2024-09-14T03:57:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:29 smithi114 ceph-mon[106969]: from='client.44960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1841/19524 objects degraded (9.429%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:30.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:29 smithi114 ceph-mon[103029]: from='client.44954 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:30.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:29 smithi114 ceph-mon[103029]: pgmap v306: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 406 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 1841/19524 objects degraded (9.429%); 0 B/s, 65 objects/s recovering 2024-09-14T03:57:30.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:29 smithi114 ceph-mon[103029]: osdmap e122: 8 total, 8 up, 8 in 2024-09-14T03:57:30.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:29 smithi114 ceph-mon[103029]: from='client.44960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:30.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1841/19524 objects degraded (9.429%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:30.648 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:57:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:57:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:57:30.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:30 smithi160 ceph-mon[88407]: osdmap e123: 8 total, 8 up, 8 in 2024-09-14T03:57:31.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:30 smithi114 ceph-mon[106969]: osdmap e123: 8 total, 8 up, 8 in 2024-09-14T03:57:31.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:30 smithi114 ceph-mon[103029]: osdmap e123: 8 total, 8 up, 8 in 2024-09-14T03:57:31.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:31 smithi114 ceph-mon[106969]: pgmap v309: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active, 86 active+clean; 279 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 925/13431 objects degraded (6.887%); 0 B/s, 77 objects/s recovering 2024-09-14T03:57:31.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:31 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3532505318' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:31.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:57:31] "GET /metrics HTTP/1.1" 200 36363 "" "Prometheus/2.51.0" 2024-09-14T03:57:31.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:31 smithi114 ceph-mon[103029]: pgmap v309: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active, 86 active+clean; 279 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 925/13431 objects degraded (6.887%); 0 B/s, 77 objects/s recovering 2024-09-14T03:57:31.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:31 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3532505318' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:31.995 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:32.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:31 smithi160 ceph-mon[88407]: pgmap v309: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active, 86 active+clean; 279 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 925/13431 objects degraded (6.887%); 0 B/s, 77 objects/s recovering 2024-09-14T03:57:32.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:31 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3532505318' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:33.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:32 smithi114 ceph-mon[106969]: from='client.54895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:33.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:32 smithi114 ceph-mon[103029]: from='client.54895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:32 smithi160 ceph-mon[88407]: from='client.54895 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:33.370 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:57:33.065+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-14T03:57:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-mon[106969]: pgmap v310: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active, 86 active+clean; 211 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 925/10197 objects degraded (9.071%); 0 B/s, 77 objects/s recovering 2024-09-14T03:57:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-mon[106969]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-14T03:57:34.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-mon[103029]: pgmap v310: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active, 86 active+clean; 211 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 925/10197 objects degraded (9.071%); 0 B/s, 77 objects/s recovering 2024-09-14T03:57:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-mon[103029]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:34.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:33 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-14T03:57:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:33 smithi160 ceph-mon[88407]: pgmap v310: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active, 86 active+clean; 211 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 925/10197 objects degraded (9.071%); 0 B/s, 77 objects/s recovering 2024-09-14T03:57:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:33 smithi160 ceph-mon[88407]: from='client.35088 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:33 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:33 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-14T03:57:34.640 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 120862 objects 2024-09-14T03:57:34.640 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :170.326 2024-09-14T03:57:35.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 925/10197 objects degraded (9.071%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:35.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 925/10197 objects degraded (9.071%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 925/10197 objects degraded (9.071%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:35.216 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:36.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:35 smithi114 ceph-mon[106969]: pgmap v311: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active, 86 active+clean; 126 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 925/6078 objects degraded (15.219%); 0 B/s, 49 objects/s recovering 2024-09-14T03:57:36.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:35 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1904932370' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:36.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:35 smithi114 ceph-mon[106969]: from='client.44981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:36.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:35 smithi114 ceph-mon[103029]: pgmap v311: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active, 86 active+clean; 126 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 925/6078 objects degraded (15.219%); 0 B/s, 49 objects/s recovering 2024-09-14T03:57:36.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:35 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1904932370' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:36.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:35 smithi114 ceph-mon[103029]: from='client.44981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:35 smithi160 ceph-mon[88407]: pgmap v311: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active, 86 active+clean; 126 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 925/6078 objects degraded (15.219%); 0 B/s, 49 objects/s recovering 2024-09-14T03:57:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:35 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1904932370' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:35 smithi160 ceph-mon[88407]: from='client.44981 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:36.602 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-14T03:57:36.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-14T03:57:36.904 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:57:37.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:36 smithi114 ceph-mon[106969]: from='client.44987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:37.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:36 smithi114 ceph-mon[106969]: osdmap e124: 8 total, 8 up, 8 in 2024-09-14T03:57:37.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:36 smithi114 ceph-mon[103029]: from='client.44987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:37.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:36 smithi114 ceph-mon[103029]: osdmap e124: 8 total, 8 up, 8 in 2024-09-14T03:57:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:36 smithi160 ceph-mon[88407]: from='client.44987 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:36 smithi160 ceph-mon[88407]: osdmap e124: 8 total, 8 up, 8 in 2024-09-14T03:57:38.091 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:38.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:37 smithi114 ceph-mon[103029]: pgmap v313: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 59 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 525 op/s; 925/2886 objects degraded (32.051%); 0 B/s, 87 objects/s recovering 2024-09-14T03:57:38.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:37 smithi114 ceph-mon[103029]: osdmap e125: 8 total, 8 up, 8 in 2024-09-14T03:57:38.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:38.092 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:37 smithi114 ceph-mon[106969]: pgmap v313: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 59 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 525 op/s; 925/2886 objects degraded (32.051%); 0 B/s, 87 objects/s recovering 2024-09-14T03:57:38.092 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:37 smithi114 ceph-mon[106969]: osdmap e125: 8 total, 8 up, 8 in 2024-09-14T03:57:38.092 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:37 smithi160 ceph-mon[88407]: pgmap v313: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 59 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 525 op/s; 925/2886 objects degraded (32.051%); 0 B/s, 87 objects/s recovering 2024-09-14T03:57:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:37 smithi160 ceph-mon[88407]: osdmap e125: 8 total, 8 up, 8 in 2024-09-14T03:57:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:38.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:38 smithi114 ceph-mon[106969]: osdmap e126: 8 total, 8 up, 8 in 2024-09-14T03:57:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/571146128' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:38 smithi114 ceph-mon[106969]: from='client.44999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:38 smithi114 ceph-mon[103029]: osdmap e126: 8 total, 8 up, 8 in 2024-09-14T03:57:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/571146128' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:38 smithi114 ceph-mon[103029]: from='client.44999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:38 smithi160 ceph-mon[88407]: osdmap e126: 8 total, 8 up, 8 in 2024-09-14T03:57:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/571146128' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:38 smithi160 ceph-mon[88407]: from='client.44999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:39.706 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_2' removed 2024-09-14T03:57:40.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[106969]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:40.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[106969]: pgmap v316: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 10 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 536 op/s; 925/534 objects degraded (173.221%); 0 B/s, 47 objects/s recovering 2024-09-14T03:57:40.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[106969]: osdmap e127: 8 total, 8 up, 8 in 2024-09-14T03:57:40.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/180190918' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T03:57:40.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T03:57:40.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 925/534 objects degraded (173.221%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:40.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[103029]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:40.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[103029]: pgmap v316: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 10 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 536 op/s; 925/534 objects degraded (173.221%); 0 B/s, 47 objects/s recovering 2024-09-14T03:57:40.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[103029]: osdmap e127: 8 total, 8 up, 8 in 2024-09-14T03:57:40.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/180190918' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T03:57:40.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T03:57:40.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:39 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 925/534 objects degraded (173.221%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:39 smithi160 ceph-mon[88407]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:39 smithi160 ceph-mon[88407]: pgmap v316: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 10 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 536 op/s; 925/534 objects degraded (173.221%); 0 B/s, 47 objects/s recovering 2024-09-14T03:57:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:39 smithi160 ceph-mon[88407]: osdmap e127: 8 total, 8 up, 8 in 2024-09-14T03:57:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/180190918' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T03:57:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:39 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T03:57:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:39 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 925/534 objects degraded (173.221%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T03:57:40.378 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-14T03:57:40.379 INFO:tasks.radosbench:Beginning radosbench... 2024-09-14T03:57:40.379 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-14T03:57:40.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-14T03:57:40.646 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:57:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:57:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:57:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:57:40.974 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:40.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:40 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T03:57:40.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:40 smithi114 ceph-mon[103029]: osdmap e128: 8 total, 8 up, 8 in 2024-09-14T03:57:40.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3583190489' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:40.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:40 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T03:57:40.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:40 smithi114 ceph-mon[106969]: osdmap e128: 8 total, 8 up, 8 in 2024-09-14T03:57:40.975 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3583190489' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:40 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T03:57:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:40 smithi160 ceph-mon[88407]: osdmap e128: 8 total, 8 up, 8 in 2024-09-14T03:57:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3583190489' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:41.713 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:57:41] "GET /metrics HTTP/1.1" 200 36367 "" "Prometheus/2.51.0" 2024-09-14T03:57:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-mon[103029]: pgmap v319: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 925/534 objects degraded (173.221%), 1 pg degraded, 2 pgs undersized) 2024-09-14T03:57:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T03:57:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-mon[103029]: from='client.45023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-mon[103029]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-mon[106969]: pgmap v319: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 925/534 objects degraded (173.221%), 1 pg degraded, 2 pgs undersized) 2024-09-14T03:57:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T03:57:42.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-mon[106969]: from='client.45023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:42.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:41 smithi114 ceph-mon[106969]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:41 smithi160 ceph-mon[88407]: pgmap v319: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:41 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 925/534 objects degraded (173.221%), 1 pg degraded, 2 pgs undersized) 2024-09-14T03:57:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:41 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T03:57:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:41 smithi160 ceph-mon[88407]: from='client.45023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:41 smithi160 ceph-mon[88407]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:43.727 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_3' created 2024-09-14T03:57:43.831 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:44.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:43 smithi114 ceph-mon[103029]: pgmap v320: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:44.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:43 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3741966480' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-14T03:57:44.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:43 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-14T03:57:44.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:43 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2797390208' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:44.054 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:43 smithi114 ceph-mon[106969]: pgmap v320: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:44.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:43 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3741966480' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-14T03:57:44.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:43 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-14T03:57:44.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:43 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2797390208' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:44.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:43 smithi160 ceph-mon[88407]: pgmap v320: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:44.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:43 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3741966480' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-14T03:57:44.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:43 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-14T03:57:44.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:43 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2797390208' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:44.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-14T03:57:44.827 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:57:44.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:44 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-14T03:57:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:44 smithi114 ceph-mon[103029]: osdmap e129: 8 total, 8 up, 8 in 2024-09-14T03:57:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:44 smithi114 ceph-mon[103029]: from='client.35166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:44 smithi114 ceph-mon[103029]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:44 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-14T03:57:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:44 smithi114 ceph-mon[106969]: osdmap e129: 8 total, 8 up, 8 in 2024-09-14T03:57:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:44 smithi114 ceph-mon[106969]: from='client.35166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:44 smithi114 ceph-mon[106969]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:44 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-14T03:57:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:44 smithi160 ceph-mon[88407]: osdmap e129: 8 total, 8 up, 8 in 2024-09-14T03:57:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:44 smithi160 ceph-mon[88407]: from='client.35166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:44 smithi160 ceph-mon[88407]: from='client.35172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:46.012 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:45 smithi114 ceph-mon[103029]: pgmap v322: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:46.012 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:45 smithi114 ceph-mon[103029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:57:46.012 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:45 smithi114 ceph-mon[103029]: osdmap e130: 8 total, 8 up, 8 in 2024-09-14T03:57:46.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:45 smithi114 ceph-mon[106969]: pgmap v322: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:46.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:45 smithi114 ceph-mon[106969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:57:46.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:45 smithi114 ceph-mon[106969]: osdmap e130: 8 total, 8 up, 8 in 2024-09-14T03:57:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:45 smithi160 ceph-mon[88407]: pgmap v322: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:45 smithi160 ceph-mon[88407]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T03:57:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:45 smithi160 ceph-mon[88407]: osdmap e130: 8 total, 8 up, 8 in 2024-09-14T03:57:46.697 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:46.739 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-14T03:57:47.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:46 smithi114 ceph-mon[106969]: osdmap e131: 8 total, 8 up, 8 in 2024-09-14T03:57:47.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1351507907' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:47.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1670690454' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:57:47.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:46 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:57:47.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:46 smithi114 ceph-mon[103029]: osdmap e131: 8 total, 8 up, 8 in 2024-09-14T03:57:47.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1351507907' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:47.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1670690454' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:57:47.062 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:46 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:57:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:46 smithi160 ceph-mon[88407]: osdmap e131: 8 total, 8 up, 8 in 2024-09-14T03:57:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1351507907' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1670690454' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:57:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:46 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T03:57:48.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[103029]: pgmap v325: 89 pgs: 3 creating+peering, 13 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:48.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[103029]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:48.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:57:48.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[103029]: osdmap e132: 8 total, 8 up, 8 in 2024-09-14T03:57:48.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[103029]: from='client.45077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:48.086 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:48.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[106969]: pgmap v325: 89 pgs: 3 creating+peering, 13 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:48.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[106969]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:48.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:57:48.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[106969]: osdmap e132: 8 total, 8 up, 8 in 2024-09-14T03:57:48.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[106969]: from='client.45077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:48.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:47 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:47 smithi160 ceph-mon[88407]: pgmap v325: 89 pgs: 3 creating+peering, 13 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:47 smithi160 ceph-mon[88407]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:47 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T03:57:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:47 smithi160 ceph-mon[88407]: osdmap e132: 8 total, 8 up, 8 in 2024-09-14T03:57:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:47 smithi160 ceph-mon[88407]: from='client.45077 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:48.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:47 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:48.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:48 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:57:48.082+0000 7fd419828640 -1 mgr.server reply reply (11) Resource temporarily unavailable 13 pgs have unknown state; cannot draw any conclusions 2024-09-14T03:57:48.371 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:48 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:57:48.082+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy 13 pgs have unknown state; cannot draw any conclusionsunsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-14T03:57:48.741 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-14T03:57:48.744 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-14T03:57:48.744 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-14T03:57:48.795 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-14T03:57:48.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-14T03:57:48.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_135745 2024-09-14T03:57:48.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:57:48.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-14T03:57:49.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:48 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:49.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:48 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:49.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:48 smithi114 ceph-mon[103029]: Upgrade: 13 pgs have unknown state; cannot draw any conclusions 2024-09-14T03:57:49.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:48 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:49.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:48 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:49.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:48 smithi114 ceph-mon[106969]: Upgrade: 13 pgs have unknown state; cannot draw any conclusions 2024-09-14T03:57:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:48 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:57:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:48 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:57:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:48 smithi160 ceph-mon[88407]: Upgrade: 13 pgs have unknown state; cannot draw any conclusions 2024-09-14T03:57:49.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 15 3810 3795 237.175 237.188 0.00306769 0.00419948 2024-09-14T03:57:50.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:49 smithi114 ceph-mon[106969]: pgmap v327: 89 pgs: 3 creating+peering, 4 unknown, 82 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:49 smithi114 ceph-mon[106969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:57:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:49 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T03:57:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:49 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2224131228' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:49 smithi114 ceph-mon[103029]: pgmap v327: 89 pgs: 3 creating+peering, 4 unknown, 82 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:49 smithi114 ceph-mon[103029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:57:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:49 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T03:57:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:49 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2224131228' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:50.133 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:49 smithi160 ceph-mon[88407]: pgmap v327: 89 pgs: 3 creating+peering, 4 unknown, 82 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-14T03:57:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:49 smithi160 ceph-mon[88407]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T03:57:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:49 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T03:57:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:49 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2224131228' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:57:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:57:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:57:50.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 7334 7318 228.671 220.188 0.00579037 0.00436504 2024-09-14T03:57:51.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:50 smithi114 ceph-mon[103029]: from='client.35217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:51.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:50 smithi114 ceph-mon[106969]: from='client.35217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:51.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:50 smithi160 ceph-mon[88407]: from='client.35217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:51.759 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:57:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:57:51] "GET /metrics HTTP/1.1" 200 36367 "" "Prometheus/2.51.0" 2024-09-14T03:57:51.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 10989 10973 228.586 228.438 0.00541445 0.00436671 2024-09-14T03:57:52.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:51 smithi114 ceph-mon[103029]: pgmap v328: 89 pgs: 89 active+clean; 39 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.4 MiB/s wr, 103 op/s 2024-09-14T03:57:52.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:51 smithi114 ceph-mon[103029]: from='client.45104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:52.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:51 smithi114 ceph-mon[106969]: pgmap v328: 89 pgs: 89 active+clean; 39 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.4 MiB/s wr, 103 op/s 2024-09-14T03:57:52.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:51 smithi114 ceph-mon[106969]: from='client.45104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:52.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:51 smithi160 ceph-mon[88407]: pgmap v328: 89 pgs: 89 active+clean; 39 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.4 MiB/s wr, 103 op/s 2024-09-14T03:57:52.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:51 smithi160 ceph-mon[88407]: from='client.45104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:52.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 14372 14356 224.293 211.438 0.00651385 0.00445128 2024-09-14T03:57:53.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 17227 17211 215.119 178.438 0.00527787 0.00464111 2024-09-14T03:57:53.850 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:53.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:53 smithi114 ceph-mon[106969]: pgmap v329: 89 pgs: 89 active+clean; 171 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s wr, 351 op/s 2024-09-14T03:57:53.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:53.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3911214788' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:53 smithi114 ceph-mon[103029]: pgmap v329: 89 pgs: 89 active+clean; 171 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s wr, 351 op/s 2024-09-14T03:57:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3911214788' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:53 smithi160 ceph-mon[88407]: pgmap v329: 89 pgs: 89 active+clean; 171 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s wr, 351 op/s 2024-09-14T03:57:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:57:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3911214788' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:54.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 20035 20019 208.513 175.5 0.00209413 0.00478933 2024-09-14T03:57:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:54 smithi160 ceph-mon[88407]: from='client.55009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:55.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:54 smithi114 ceph-mon[106969]: from='client.55009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:55.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:54 smithi114 ceph-mon[103029]: from='client.55009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:55.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 15 22974 22959 204.973 183.75 0.00357722 0.00487413 2024-09-14T03:57:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:55 smithi160 ceph-mon[88407]: from='client.45122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:55 smithi160 ceph-mon[88407]: pgmap v330: 89 pgs: 89 active+clean; 791 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-14T03:57:56.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:55 smithi114 ceph-mon[106969]: from='client.45122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:56.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:55 smithi114 ceph-mon[106969]: pgmap v330: 89 pgs: 89 active+clean; 791 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-14T03:57:56.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:55 smithi114 ceph-mon[103029]: from='client.45122 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:56.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:55 smithi114 ceph-mon[103029]: pgmap v330: 89 pgs: 89 active+clean; 791 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-14T03:57:56.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 25638 25622 200.154 166.438 0.00561458 0.00499038 2024-09-14T03:57:57.522 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:57:57.797 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 27940 27924 193.899 143.875 0.0137123 0.00515231 2024-09-14T03:57:57.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:57 smithi114 ceph-mon[106969]: pgmap v331: 89 pgs: 89 active+clean; 987 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-14T03:57:57.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:57 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2200979258' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:57 smithi114 ceph-mon[103029]: pgmap v331: 89 pgs: 89 active+clean; 987 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-14T03:57:57.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:57 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2200979258' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:57 smithi160 ceph-mon[88407]: pgmap v331: 89 pgs: 89 active+clean; 987 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-14T03:57:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:57 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2200979258' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:57:58.796 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 15 30247 30232 188.933 144.25 0.00563157 0.00528743 2024-09-14T03:57:59.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:58 smithi114 ceph-mon[106969]: from='client.55024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:59.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:58 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:59.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:58 smithi114 ceph-mon[106969]: from='client.45137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:59.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:58 smithi114 ceph-mon[103029]: from='client.55024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:59.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:58 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:59.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:58 smithi114 ceph-mon[103029]: from='client.45137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:58 smithi160 ceph-mon[88407]: from='client.55024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:58 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:57:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:58 smithi160 ceph-mon[88407]: from='client.45137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:57:59.797 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 32716 32700 185.779 154.25 0.005645 0.00537687 2024-09-14T03:58:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:57:59 smithi160 ceph-mon[88407]: pgmap v332: 89 pgs: 89 active+clean; 1.4 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-14T03:58:00.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:57:59 smithi114 ceph-mon[103029]: pgmap v332: 89 pgs: 89 active+clean; 1.4 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-14T03:58:00.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:57:59 smithi114 ceph-mon[106969]: pgmap v332: 89 pgs: 89 active+clean; 1.4 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-14T03:58:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:58:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:58:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:58:00.797 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 34625 34609 180.239 119.312 0.0122894 0.0055428 2024-09-14T03:58:01.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2114309521' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:01.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2114309521' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2114309521' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:01.175 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:01.753 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:58:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:58:01] "GET /metrics HTTP/1.1" 200 36353 "" "Prometheus/2.51.0" 2024-09-14T03:58:01.797 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 36404 36388 174.927 111.188 0.00477146 0.00571105 2024-09-14T03:58:02.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:01 smithi114 ceph-mon[106969]: pgmap v333: 89 pgs: 89 active+clean; 1.8 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-09-14T03:58:02.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:01 smithi114 ceph-mon[106969]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:01 smithi114 ceph-mon[103029]: pgmap v333: 89 pgs: 89 active+clean; 1.8 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-09-14T03:58:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:01 smithi114 ceph-mon[103029]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:01 smithi160 ceph-mon[88407]: pgmap v333: 89 pgs: 89 active+clean; 1.8 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-09-14T03:58:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:01 smithi160 ceph-mon[88407]: from='client.35256 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:02.797 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 38633 38617 172.382 139.312 0.00460211 0.00579629 2024-09-14T03:58:03.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:02 smithi114 ceph-mon[103029]: from='client.45149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:02 smithi114 ceph-mon[106969]: from='client.45149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:02 smithi160 ceph-mon[88407]: from='client.45149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:03.797 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 40580 40564 169.002 121.688 0.00760454 0.00591014 2024-09-14T03:58:04.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[106969]: pgmap v334: 89 pgs: 89 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 160 MiB/s wr, 2.55k op/s 2024-09-14T03:58:04.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:04.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[103029]: pgmap v334: 89 pgs: 89 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 160 MiB/s wr, 2.55k op/s 2024-09-14T03:58:04.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:04.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:03 smithi160 ceph-mon[88407]: pgmap v334: 89 pgs: 89 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 160 MiB/s wr, 2.55k op/s 2024-09-14T03:58:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:04.796 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:04.797 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 15 42182 42167 164.7 100.188 0.00762536 0.00606611 2024-09-14T03:58:05.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:04 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1608025638' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:05.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:04 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1608025638' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:04 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1608025638' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:05.797 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 43775 43759 160.864 99.5 0.011604 0.0062104 2024-09-14T03:58:06.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:05 smithi114 ceph-mon[106969]: pgmap v335: 89 pgs: 89 active+clean; 2.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 183 MiB/s wr, 2.92k op/s 2024-09-14T03:58:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:05 smithi114 ceph-mon[106969]: from='client.55051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:05 smithi114 ceph-mon[103029]: pgmap v335: 89 pgs: 89 active+clean; 2.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 183 MiB/s wr, 2.92k op/s 2024-09-14T03:58:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:05 smithi114 ceph-mon[103029]: from='client.55051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:05 smithi160 ceph-mon[88407]: pgmap v335: 89 pgs: 89 active+clean; 2.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 183 MiB/s wr, 2.92k op/s 2024-09-14T03:58:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:05 smithi160 ceph-mon[88407]: from='client.55051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:06.798 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 45407 45391 157.585 102 0.012283 0.00634001 2024-09-14T03:58:07.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:06 smithi114 ceph-mon[103029]: from='client.45161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:07.076 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:06 smithi114 ceph-mon[106969]: from='client.45161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:07.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:06 smithi160 ceph-mon[88407]: from='client.45161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:07.798 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 46866 46850 154.09 91.1875 0.00638591 0.00648356 2024-09-14T03:58:07.892 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:07 smithi160 ceph-mon[88407]: pgmap v336: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-09-14T03:58:07.892 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:07.892 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:07 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1288477456' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:07 smithi114 ceph-mon[103029]: pgmap v336: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-09-14T03:58:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:07 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1288477456' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:07 smithi114 ceph-mon[106969]: pgmap v336: 89 pgs: 89 active+clean; 2.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 141 MiB/s wr, 2.25k op/s 2024-09-14T03:58:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:07 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1288477456' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:08.338 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:08.800 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:58:08.799964+0000 min lat: 0.0011608 max lat: 0.156675 avg lat: 0.00660236 2024-09-14T03:58:08.800 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:58:08.800 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 48447 48431 151.313 98.8125 0.00772454 0.00660236 2024-09-14T03:58:09.800 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 50206 50190 149.343 109.938 0.00919009 0.00669137 2024-09-14T03:58:10.480 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:10 smithi114 ceph-mon[106969]: from='client.55069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:10.481 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:10 smithi114 ceph-mon[106969]: pgmap v337: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 148 MiB/s wr, 2.38k op/s 2024-09-14T03:58:10.481 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:10 smithi114 ceph-mon[106969]: from='client.35295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:10.481 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:10 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:10.481 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:10 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:10.481 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:10 smithi114 ceph-mon[103029]: from='client.55069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:10.481 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:10 smithi114 ceph-mon[103029]: pgmap v337: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 148 MiB/s wr, 2.38k op/s 2024-09-14T03:58:10.481 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:10 smithi114 ceph-mon[103029]: from='client.35295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:10.482 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:10 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:10.482 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:10 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:58:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:58:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:58:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:10 smithi160 ceph-mon[88407]: from='client.55069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:10 smithi160 ceph-mon[88407]: pgmap v337: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 148 MiB/s wr, 2.38k op/s 2024-09-14T03:58:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:10 smithi160 ceph-mon[88407]: from='client.35295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:10 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:10 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:10.800 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 51527 51511 146.307 82.5625 0.00229313 0.00681258 2024-09-14T03:58:11.783 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:11.783 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:58:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:58:11] "GET /metrics HTTP/1.1" 200 36362 "" "Prometheus/2.51.0" 2024-09-14T03:58:11.800 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 52695 52679 143.12 73 0.0898221 0.00697998 2024-09-14T03:58:12.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:12 smithi114 ceph-mon[103029]: pgmap v338: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-14T03:58:12.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:12 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/375415956' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:12 smithi114 ceph-mon[106969]: pgmap v338: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-14T03:58:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:12 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/375415956' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:12 smithi160 ceph-mon[88407]: pgmap v338: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-14T03:58:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:12 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/375415956' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:12.802 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 54276 54260 141.265 98.8125 0.0161181 0.0070727 2024-09-14T03:58:13.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:13 smithi114 ceph-mon[103029]: from='client.55081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:13 smithi114 ceph-mon[103029]: from='client.55084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:13.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:13 smithi114 ceph-mon[106969]: from='client.55081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:13.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:13 smithi114 ceph-mon[106969]: from='client.55084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:13 smithi160 ceph-mon[88407]: from='client.55081 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:13 smithi160 ceph-mon[88407]: from='client.55084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:13.804 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 56017 56001 139.956 108.812 0.00642125 0.00714057 2024-09-14T03:58:14.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:14 smithi114 ceph-mon[103029]: pgmap v339: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-14T03:58:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:14 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:14 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:14 smithi114 ceph-mon[106969]: pgmap v339: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-14T03:58:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:14 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:14 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:14 smithi160 ceph-mon[88407]: pgmap v339: 89 pgs: 89 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-14T03:58:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:14 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:14 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:14.804 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 57370 57354 137.826 84.5625 0.0117342 0.00725009 2024-09-14T03:58:15.355 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:15.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:58:15.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:58:15.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:15.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:15.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:15.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:15.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:15.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:58:15.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/566856175' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:15.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:58:15.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:58:15.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:15.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:15.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:15.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:15.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:15.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:58:15.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/566856175' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:58:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:58:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:15.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:15.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:15.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-14T03:58:15.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/566856175' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:15.804 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 58843 58827 136.131 92.0625 0.0107165 0.00734059 2024-09-14T03:58:16.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:16 smithi114 ceph-mon[103029]: pgmap v340: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-14T03:58:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:16 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:58:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:16 smithi114 ceph-mon[103029]: Upgrade: osd.3 is safe to restart 2024-09-14T03:58:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:16 smithi114 ceph-mon[106969]: pgmap v340: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-14T03:58:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:16 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:58:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:16 smithi114 ceph-mon[106969]: Upgrade: osd.3 is safe to restart 2024-09-14T03:58:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:16 smithi160 ceph-mon[88407]: pgmap v340: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-14T03:58:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:16 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-14T03:58:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:16 smithi160 ceph-mon[88407]: Upgrade: osd.3 is safe to restart 2024-09-14T03:58:16.804 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 60323 60307 134.573 92.5 0.0111487 0.0074262 2024-09-14T03:58:17.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:17 smithi114 ceph-mon[103029]: from='client.55090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:17.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:17 smithi114 ceph-mon[103029]: from='client.35322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:17.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:17 smithi114 ceph-mon[106969]: from='client.55090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:17.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:17 smithi114 ceph-mon[106969]: from='client.35322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:17 smithi160 ceph-mon[88407]: from='client.55090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:17 smithi160 ceph-mon[88407]: from='client.35322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:17.804 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 61455 61439 132.372 70.75 0.00723696 0.00754978 2024-09-14T03:58:18.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:18 smithi114 ceph-mon[103029]: pgmap v341: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-14T03:58:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-14T03:58:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:58:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:18 smithi114 ceph-mon[106969]: pgmap v341: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-14T03:58:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-14T03:58:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:58:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:18 smithi160 ceph-mon[88407]: pgmap v341: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-14T03:58:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-14T03:58:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:58:18.804 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 62836 62820 130.837 86.3125 0.00895253 0.00763752 2024-09-14T03:58:18.997 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:19.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:19 smithi114 ceph-mon[106969]: Upgrade: Updating osd.3 2024-09-14T03:58:19.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:19 smithi114 ceph-mon[106969]: Deploying daemon osd.3 on smithi114 2024-09-14T03:58:19.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/534168886' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:19.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:19 smithi114 ceph-mon[103029]: Upgrade: Updating osd.3 2024-09-14T03:58:19.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:19 smithi114 ceph-mon[103029]: Deploying daemon osd.3 on smithi114 2024-09-14T03:58:19.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:19 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/534168886' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:19 smithi160 ceph-mon[88407]: Upgrade: Updating osd.3 2024-09-14T03:58:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:19 smithi160 ceph-mon[88407]: Deploying daemon osd.3 on smithi114 2024-09-14T03:58:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:19 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/534168886' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:19.804 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 64290 64274 129.548 90.875 0.0107002 0.00771416 2024-09-14T03:58:20.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:20 smithi114 ceph-mon[103029]: pgmap v342: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-14T03:58:20.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:20 smithi114 ceph-mon[103029]: from='client.45212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:20.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:20 smithi114 ceph-mon[106969]: pgmap v342: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-14T03:58:20.513 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:20 smithi114 ceph-mon[106969]: from='client.45212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:20.662 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:58:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:58:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:58:20.662 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:20 smithi160 ceph-mon[88407]: pgmap v342: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-14T03:58:20.663 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:20 smithi160 ceph-mon[88407]: from='client.45212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:20.805 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 65704 65688 128.261 88.375 0.00482145 0.00779122 2024-09-14T03:58:21.367 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:21 smithi114 ceph-mon[106969]: from='client.35334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:21.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:21 smithi114 ceph-mon[103029]: from='client.35334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:21.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:58:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:58:21] "GET /metrics HTTP/1.1" 200 36362 "" "Prometheus/2.51.0" 2024-09-14T03:58:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:21 smithi160 ceph-mon[88407]: from='client.35334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:21.805 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 67181 67165 127.172 92.3125 0.0109907 0.00785844 2024-09-14T03:58:22.347 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:22 smithi114 ceph-mon[106969]: pgmap v343: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-14T03:58:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:22 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2548807490' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:22 smithi114 ceph-mon[103029]: pgmap v343: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-14T03:58:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:22 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2548807490' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:22 smithi160 ceph-mon[88407]: pgmap v343: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-14T03:58:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:22 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2548807490' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:22.805 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 67954 67938 124.852 48.3125 0.00306348 0.00796987 2024-09-14T03:58:23.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:23 smithi114 ceph-mon[106969]: from='client.35343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:23.788 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:23 smithi114 ceph-mon[106969]: pgmap v344: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-14T03:58:23.788 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:23.788 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:23 smithi114 ceph-mon[106969]: from='client.55111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:23.788 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:23 smithi114 ceph-mon[103029]: from='client.35343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:23.788 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:23 smithi114 ceph-mon[103029]: pgmap v344: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-14T03:58:23.788 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:23.789 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:23 smithi114 ceph-mon[103029]: from='client.55111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:23.789 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:23 smithi160 ceph-mon[88407]: from='client.35343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:23.789 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:23 smithi160 ceph-mon[88407]: pgmap v344: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-14T03:58:23.789 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:23.789 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:23 smithi160 ceph-mon[88407]: from='client.55111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:23.805 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 69394 69378 123.857 90 0.00829793 0.0080691 2024-09-14T03:58:24.120 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:23 smithi114 systemd[1]: Stopping Ceph osd.3 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:58:24.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:24 smithi114 ceph-mon[103029]: osd.3 marked itself down and dead 2024-09-14T03:58:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:24 smithi114 ceph-mon[106969]: osd.3 marked itself down and dead 2024-09-14T03:58:24.621 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[64118]: 2024-09-14T03:58:24.241+0000 7f2e0a57b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T03:58:24.621 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[64118]: 2024-09-14T03:58:24.241+0000 7f2e0a57b640 -1 osd.3 132 *** Got signal Terminated *** 2024-09-14T03:58:24.622 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:24 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[64118]: 2024-09-14T03:58:24.241+0000 7f2e0a57b640 -1 osd.3 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T03:58:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:24 smithi160 ceph-mon[88407]: osd.3 marked itself down and dead 2024-09-14T03:58:24.805 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 70118 70102 121.673 45.25 0.00304335 0.00810552 2024-09-14T03:58:25.302 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:24 smithi114 podman[137451]: 2024-09-14 03:58:24.95353261 +0000 UTC m=+0.837773925 container died b43e985ecf6cdcfd7959267bc8a4824836d46ee9b013eb2cdba9ff49ce2550f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T03:58:25.383 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:25.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:25 smithi114 ceph-mon[106969]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:58:25.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:25 smithi114 ceph-mon[106969]: osdmap e133: 8 total, 7 up, 8 in 2024-09-14T03:58:25.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:25 smithi114 ceph-mon[106969]: pgmap v346: 89 pgs: 16 stale+active+clean, 73 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-14T03:58:25.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2807971686' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:25 smithi114 ceph-mon[103029]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:58:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:25 smithi114 ceph-mon[103029]: osdmap e133: 8 total, 7 up, 8 in 2024-09-14T03:58:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:25 smithi114 ceph-mon[103029]: pgmap v346: 89 pgs: 16 stale+active+clean, 73 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-14T03:58:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2807971686' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:25.622 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:25 smithi114 podman[137451]: 2024-09-14 03:58:25.379787189 +0000 UTC m=+1.264028502 container cleanup b43e985ecf6cdcfd7959267bc8a4824836d46ee9b013eb2cdba9ff49ce2550f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, RELEASE=quincy-eb16b97, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-14T03:58:25.622 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:25 smithi114 bash[137451]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3 2024-09-14T03:58:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:25 smithi160 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-14T03:58:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:25 smithi160 ceph-mon[88407]: osdmap e133: 8 total, 7 up, 8 in 2024-09-14T03:58:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:25 smithi160 ceph-mon[88407]: pgmap v346: 89 pgs: 16 stale+active+clean, 73 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-14T03:58:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2807971686' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:25.805 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 15 70849 70834 119.622 45.75 0.02005 0.0083537 2024-09-14T03:58:26.120 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:25 smithi114 podman[137491]: 2024-09-14 03:58:25.623763606 +0000 UTC m=+0.667457957 container remove b43e985ecf6cdcfd7959267bc8a4824836d46ee9b013eb2cdba9ff49ce2550f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=quincy-eb16b97, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-09-14T03:58:26.579 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:26 smithi114 ceph-mon[103029]: osdmap e134: 8 total, 7 up, 8 in 2024-09-14T03:58:26.579 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:26 smithi114 ceph-mon[103029]: from='client.45248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:26.579 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:26 smithi114 ceph-mon[103029]: from='client.45254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:26.579 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:26 smithi114 ceph-mon[106969]: osdmap e134: 8 total, 7 up, 8 in 2024-09-14T03:58:26.579 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:26 smithi114 ceph-mon[106969]: from='client.45248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:26.579 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:26 smithi114 ceph-mon[106969]: from='client.45254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:26.580 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:26 smithi114 podman[137620]: 2024-09-14 03:58:26.160111455 +0000 UTC m=+0.027964882 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:58:26.580 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:26 smithi114 podman[137620]: 2024-09-14 03:58:26.283319735 +0000 UTC m=+0.151173148 container create 20cb102fdfae45b7542635dd53e5b8706708a50797e3e1c93e9a2e1ebbc8decb (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=main-a278231, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:58:26.580 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:26 smithi114 podman[137620]: 2024-09-14 03:58:26.578299292 +0000 UTC m=+0.446152699 container init 20cb102fdfae45b7542635dd53e5b8706708a50797e3e1c93e9a2e1ebbc8decb (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-a278231, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T03:58:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:26 smithi160 ceph-mon[88407]: osdmap e134: 8 total, 7 up, 8 in 2024-09-14T03:58:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:26 smithi160 ceph-mon[88407]: from='client.45248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:26 smithi160 ceph-mon[88407]: from='client.45254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:26.805 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 72511 72495 119.205 103.812 0.00613873 0.00838391 2024-09-14T03:58:26.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:26 smithi114 podman[137620]: 2024-09-14 03:58:26.5898779 +0000 UTC m=+0.457731314 container start 20cb102fdfae45b7542635dd53e5b8706708a50797e3e1c93e9a2e1ebbc8decb (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a278231, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:58:26.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:26 smithi114 podman[137620]: 2024-09-14 03:58:26.686696113 +0000 UTC m=+0.554549574 container attach 20cb102fdfae45b7542635dd53e5b8706708a50797e3e1c93e9a2e1ebbc8decb (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=main-a278231, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-14T03:58:27.260 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:26 smithi114 podman[137659]: 2024-09-14 03:58:26.954460245 +0000 UTC m=+0.037417920 container died 20cb102fdfae45b7542635dd53e5b8706708a50797e3e1c93e9a2e1ebbc8decb (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a278231, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-14T03:58:27.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:27 smithi114 ceph-mon[106969]: pgmap v348: 89 pgs: 3 active+undersized, 14 stale+active+clean, 1 active+undersized+degraded, 71 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 4494/204456 objects degraded (2.198%) 2024-09-14T03:58:27.601 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:27 smithi114 ceph-mon[103029]: pgmap v348: 89 pgs: 3 active+undersized, 14 stale+active+clean, 1 active+undersized+degraded, 71 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 4494/204456 objects degraded (2.198%) 2024-09-14T03:58:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:27 smithi160 ceph-mon[88407]: pgmap v348: 89 pgs: 3 active+undersized, 14 stale+active+clean, 1 active+undersized+degraded, 71 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 4494/204456 objects degraded (2.198%) 2024-09-14T03:58:27.805 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 74166 74150 118.801 103.438 0.0118449 0.00841243 2024-09-14T03:58:27.870 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:27 smithi114 podman[137659]: 2024-09-14 03:58:27.600306375 +0000 UTC m=+0.683264038 container remove 20cb102fdfae45b7542635dd53e5b8706708a50797e3e1c93e9a2e1ebbc8decb (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-a278231, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T03:58:27.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:27 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.3.service: Deactivated successfully. 2024-09-14T03:58:27.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:27 smithi114 systemd[1]: Stopped Ceph osd.3 for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:58:27.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:27 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.3.service: Consumed 23min 4.246s CPU time. 2024-09-14T03:58:28.245 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:27 smithi114 systemd[1]: Starting Ceph osd.3 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T03:58:28.539 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:28 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 4494/204456 objects degraded (2.198%), 1 pg degraded (PG_DEGRADED) 2024-09-14T03:58:28.540 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:28 smithi114 podman[137780]: 2024-09-14 03:58:28.149550596 +0000 UTC m=+0.031652933 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:58:28.540 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:28 smithi114 podman[137780]: 2024-09-14 03:58:28.268647635 +0000 UTC m=+0.150749969 container create e647f91c81cdfcced4a6ad00456a909b58642955ff5c0860bb2b99c3eee8e2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-a278231, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-14T03:58:28.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:28 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 4494/204456 objects degraded (2.198%), 1 pg degraded (PG_DEGRADED) 2024-09-14T03:58:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:28 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 4494/204456 objects degraded (2.198%), 1 pg degraded (PG_DEGRADED) 2024-09-14T03:58:28.805 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:58:28.805242+0000 min lat: 0.0011608 max lat: 1.04749 avg lat: 0.00845779 2024-09-14T03:58:28.805 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:58:28.806 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 75656 75640 118.159 93.125 0.00843822 0.00845779 2024-09-14T03:58:28.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:28 smithi114 podman[137780]: 2024-09-14 03:58:28.538807632 +0000 UTC m=+0.420909970 container init e647f91c81cdfcced4a6ad00456a909b58642955ff5c0860bb2b99c3eee8e2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-14T03:58:28.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:28 smithi114 podman[137780]: 2024-09-14 03:58:28.549427229 +0000 UTC m=+0.431529561 container start e647f91c81cdfcced4a6ad00456a909b58642955ff5c0860bb2b99c3eee8e2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a278231, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-14T03:58:28.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:28 smithi114 podman[137780]: 2024-09-14 03:58:28.647049939 +0000 UTC m=+0.529152276 container attach e647f91c81cdfcced4a6ad00456a909b58642955ff5c0860bb2b99c3eee8e2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=main-a278231, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T03:58:28.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:28 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate[137793]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:58:28.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:28 smithi114 bash[137780]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:58:28.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:28 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate[137793]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:58:28.871 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:28 smithi114 bash[137780]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-14T03:58:28.983 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:29.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4051376442' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:29.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-mon[106969]: pgmap v349: 89 pgs: 17 active+undersized, 7 stale+active+clean, 4 active+undersized+degraded, 61 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 13848/214575 objects degraded (6.454%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:58:29.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-mon[106969]: from='client.35373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:29.622 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate[137793]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-14T03:58:29.622 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 bash[137780]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-14T03:58:29.622 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate[137793]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-14T03:58:29.622 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 bash[137780]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-14T03:58:29.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4051376442' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:29.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-mon[103029]: pgmap v349: 89 pgs: 17 active+undersized, 7 stale+active+clean, 4 active+undersized+degraded, 61 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 13848/214575 objects degraded (6.454%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:58:29.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-mon[103029]: from='client.35373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:29 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4051376442' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:29 smithi160 ceph-mon[88407]: pgmap v349: 89 pgs: 17 active+undersized, 7 stale+active+clean, 4 active+undersized+degraded, 61 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 13848/214575 objects degraded (6.454%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:58:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:29 smithi160 ceph-mon[88407]: from='client.35373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:29.806 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 77189 77173 117.614 95.8125 0.00689111 0.00849769 2024-09-14T03:58:30.121 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate[137793]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-14T03:58:30.121 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 bash[137780]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-14T03:58:30.121 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate[137793]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-14T03:58:30.121 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 bash[137780]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-14T03:58:30.121 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate[137793]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-14T03:58:30.121 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 bash[137780]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-14T03:58:30.121 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate[137793]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-14T03:58:30.122 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 bash[137780]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-14T03:58:30.122 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate[137793]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-14T03:58:30.122 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 bash[137780]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-14T03:58:30.122 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:29 smithi114 podman[137780]: 2024-09-14 03:58:29.738995036 +0000 UTC m=+1.621097370 container died e647f91c81cdfcced4a6ad00456a909b58642955ff5c0860bb2b99c3eee8e2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=main-a278231, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-14T03:58:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:30 smithi160 ceph-mon[88407]: from='client.45269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:30.559 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:58:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:58:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:58:30.595 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:30 smithi114 ceph-mon[103029]: from='client.45269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:30.596 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:30 smithi114 ceph-mon[106969]: from='client.45269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:30.806 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 78773 78757 117.17 99 0.00628912 0.0085285 2024-09-14T03:58:30.870 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:30 smithi114 podman[138005]: 2024-09-14 03:58:30.594762607 +0000 UTC m=+0.847678398 container remove e647f91c81cdfcced4a6ad00456a909b58642955ff5c0860bb2b99c3eee8e2b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-a278231, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-09-14T03:58:31.147 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:30 smithi114 podman[138060]: 2024-09-14 03:58:30.798802669 +0000 UTC m=+0.038577263 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:58:31.147 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:30 smithi114 podman[138060]: 2024-09-14 03:58:30.979382468 +0000 UTC m=+0.219157048 container create 10460276d62fcf22147d36b06f9014b1b03d3c3af22378e54aa071f19a9d0de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-a278231, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-14T03:58:31.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:31 smithi114 ceph-mon[103029]: pgmap v350: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s; 27940/223116 objects degraded (12.523%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:58:31.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:31 smithi114 ceph-mon[106969]: pgmap v350: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s; 27940/223116 objects degraded (12.523%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:58:31.435 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:31 smithi114 podman[138060]: 2024-09-14 03:58:31.26571575 +0000 UTC m=+0.505490336 container init 10460276d62fcf22147d36b06f9014b1b03d3c3af22378e54aa071f19a9d0de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a278231, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-14T03:58:31.435 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:31 smithi114 podman[138060]: 2024-09-14 03:58:31.275225386 +0000 UTC m=+0.514999966 container start 10460276d62fcf22147d36b06f9014b1b03d3c3af22378e54aa071f19a9d0de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=main-a278231, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-14T03:58:31.436 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:31 smithi114 ceph-osd[138095]: -- 172.21.15.114:0/3331507093 <== mon.2 v2:172.21.15.114:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564e5f53fa40 con 0x564e6030c000 2024-09-14T03:58:31.436 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:31 smithi114 bash[138060]: 10460276d62fcf22147d36b06f9014b1b03d3c3af22378e54aa071f19a9d0de4 2024-09-14T03:58:31.436 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:31 smithi114 systemd[1]: Started Ceph osd.3 for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T03:58:31.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:31 smithi160 ceph-mon[88407]: pgmap v350: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s; 27940/223116 objects degraded (12.523%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:58:31.809 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 80189 80173 116.495 88.5 0.0131862 0.00857877 2024-09-14T03:58:31.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:58:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:58:31] "GET /metrics HTTP/1.1" 200 36366 "" "Prometheus/2.51.0" 2024-09-14T03:58:32.371 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[138073]: 2024-09-14T03:58:31.939+0000 7efe87a51640 -1 Falling back to public interface 2024-09-14T03:58:32.578 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:32.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:32.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:32 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/424547231' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:32.809 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 81394 81378 115.559 75.3125 0.0130002 0.00864846 2024-09-14T03:58:32.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:32.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:32.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:32.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:32 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/424547231' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:32.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:32.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:32.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:32.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:32 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/424547231' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:33 smithi160 ceph-mon[88407]: pgmap v351: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 28984/229110 objects degraded (12.651%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:58:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:33 smithi160 ceph-mon[88407]: from='client.45284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:33 smithi160 ceph-mon[88407]: from='client.35391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:33.761 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:33 smithi114 ceph-mon[106969]: pgmap v351: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 28984/229110 objects degraded (12.651%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:58:33.761 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:33 smithi114 ceph-mon[106969]: from='client.45284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:33.761 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:33 smithi114 ceph-mon[106969]: from='client.35391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:33.761 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:33 smithi114 ceph-mon[103029]: pgmap v351: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 28984/229110 objects degraded (12.651%); 16 KiB/s, 0 objects/s recovering 2024-09-14T03:58:33.761 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:33 smithi114 ceph-mon[103029]: from='client.45284 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:33.761 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:33 smithi114 ceph-mon[103029]: from='client.35391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:33.809 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 82740 82724 114.86 84.125 0.0152391 0.00870103 2024-09-14T03:58:34.766 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 28984/229110 objects degraded (12.651%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:34.766 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 28984/229110 objects degraded (12.651%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:34.809 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 15 84186 84171 114.329 90.4375 0.015539 0.0087416 2024-09-14T03:58:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 28984/229110 objects degraded (12.651%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:35.809 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 85492 85476 113.632 81.5625 0.00755161 0.00879508 2024-09-14T03:58:35.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:35 smithi114 ceph-mon[106969]: pgmap v352: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 30676/244932 objects degraded (12.524%); 13 KiB/s, 0 objects/s recovering 2024-09-14T03:58:35.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:35 smithi114 ceph-mon[103029]: pgmap v352: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 30676/244932 objects degraded (12.524%); 13 KiB/s, 0 objects/s recovering 2024-09-14T03:58:35.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:35 smithi160 ceph-mon[88407]: pgmap v352: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 30676/244932 objects degraded (12.524%); 13 KiB/s, 0 objects/s recovering 2024-09-14T03:58:36.187 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:36.809 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 86660 86644 112.785 73 0.0171601 0.00886183 2024-09-14T03:58:36.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:36 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/387968815' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:36.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:36 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:36.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:36 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:36.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:36 smithi114 ceph-mon[106969]: from='client.35403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:36 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/387968815' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:36 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:36 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:36 smithi114 ceph-mon[103029]: from='client.35403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:36.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:36 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/387968815' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:36.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:36 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:36.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:36 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:36.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:36 smithi160 ceph-mon[88407]: from='client.35403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:37.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 88024 88008 112.223 85.25 0.0123304 0.00890606 2024-09-14T03:58:38.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[106969]: pgmap v353: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 31103/246213 objects degraded (12.633%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:58:38.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[106969]: from='client.45302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:38.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:38.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:38.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:38.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:38.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[103029]: pgmap v353: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 31103/246213 objects degraded (12.633%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:58:38.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[103029]: from='client.45302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:38.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:38.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:38.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:38.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:38 smithi160 ceph-mon[88407]: pgmap v353: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 31103/246213 objects degraded (12.633%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:58:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:38 smithi160 ceph-mon[88407]: from='client.45302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:38.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 89415 89399 111.718 86.9375 0.0128808 0.00894626 2024-09-14T03:58:39.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:39 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 31951/257337 objects degraded (12.416%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:39.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:39 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 31951/257337 objects degraded (12.416%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:39 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 31951/257337 objects degraded (12.416%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:39.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 15 90593 90578 110.972 73.6875 0.00887484 0.00900586 2024-09-14T03:58:39.811 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:40 smithi160 ceph-mon[88407]: pgmap v354: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 31951/257337 objects degraded (12.416%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:58:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/833108857' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:40.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:58:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:58:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:58:40.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:40 smithi114 ceph-mon[106969]: pgmap v354: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 31951/257337 objects degraded (12.416%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:58:40.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/833108857' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:40 smithi114 ceph-mon[103029]: pgmap v354: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 31951/257337 objects degraded (12.416%); 11 KiB/s, 0 objects/s recovering 2024-09-14T03:58:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/833108857' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:40.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 91831 91815 110.325 77.3125 0.00771128 0.00905898 2024-09-14T03:58:41.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:41 smithi160 ceph-mon[88407]: from='client.35418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:41.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:41 smithi114 ceph-mon[106969]: from='client.35418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:41.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:41 smithi114 ceph-mon[103029]: from='client.35418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:41.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 92974 92958 109.591 71.4375 0.00879929 0.00911951 2024-09-14T03:58:41.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:58:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:58:41] "GET /metrics HTTP/1.1" 200 36381 "" "Prometheus/2.51.0" 2024-09-14T03:58:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:42 smithi160 ceph-mon[88407]: from='client.45320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:42 smithi160 ceph-mon[88407]: pgmap v355: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s; 33212/264861 objects degraded (12.539%) 2024-09-14T03:58:42.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:42 smithi114 ceph-mon[103029]: from='client.45320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:42.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:42 smithi114 ceph-mon[103029]: pgmap v355: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s; 33212/264861 objects degraded (12.539%) 2024-09-14T03:58:42.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:42 smithi114 ceph-mon[106969]: from='client.45320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:42.421 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:42 smithi114 ceph-mon[106969]: pgmap v355: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s; 33212/264861 objects degraded (12.539%) 2024-09-14T03:58:42.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 94115 94099 108.882 71.3125 0.013221 0.00917938 2024-09-14T03:58:43.324 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:43 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2968866877' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:43.324 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:43 smithi114 ceph-mon[106969]: from='osd.3 [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-14T03:58:43.324 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:43 smithi114 ceph-mon[106969]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-14T03:58:43.324 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:43 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[138073]: 2024-09-14T03:58:43.005+0000 7efe87a51640 -1 osd.3 132 log_to_monitors true 2024-09-14T03:58:43.324 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:43 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2968866877' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:43.325 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:43 smithi114 ceph-mon[103029]: from='osd.3 [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-14T03:58:43.325 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:43 smithi114 ceph-mon[103029]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-14T03:58:43.332 INFO:teuthology.orchestra.run.smithi114.stdout:true 2024-09-14T03:58:43.465 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:43 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2968866877' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:43.465 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:43 smithi160 ceph-mon[88407]: from='osd.3 [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-14T03:58:43.465 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:43 smithi160 ceph-mon[88407]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-14T03:58:43.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 94983 94967 107.889 54.25 0.0152964 0.00926373 2024-09-14T03:58:44.120 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 03:58:43 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[138073]: 2024-09-14T03:58:43.766+0000 7efe7b7f3640 -1 osd.3 132 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T03:58:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[106969]: pgmap v356: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s; 34054/269688 objects degraded (12.627%) 2024-09-14T03:58:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[106969]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-14T03:58:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[106969]: osdmap e135: 8 total, 7 up, 8 in 2024-09-14T03:58:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[106969]: from='osd.3 [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:58:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[106969]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:58:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[106969]: from='osd.3 ' entity='osd.3' 2024-09-14T03:58:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 34054/269688 objects degraded (12.627%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[103029]: pgmap v356: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s; 34054/269688 objects degraded (12.627%) 2024-09-14T03:58:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[103029]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-14T03:58:44.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[103029]: osdmap e135: 8 total, 7 up, 8 in 2024-09-14T03:58:44.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[103029]: from='osd.3 [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:58:44.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[103029]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:58:44.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[103029]: from='osd.3 ' entity='osd.3' 2024-09-14T03:58:44.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:44 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 34054/269688 objects degraded (12.627%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:44 smithi160 ceph-mon[88407]: pgmap v356: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s; 34054/269688 objects degraded (12.627%) 2024-09-14T03:58:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:44 smithi160 ceph-mon[88407]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-14T03:58:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:44 smithi160 ceph-mon[88407]: osdmap e135: 8 total, 7 up, 8 in 2024-09-14T03:58:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:44 smithi160 ceph-mon[88407]: from='osd.3 [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:58:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:44 smithi160 ceph-mon[88407]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]} : dispatch 2024-09-14T03:58:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:44 smithi160 ceph-mon[88407]: from='osd.3 ' entity='osd.3' 2024-09-14T03:58:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:44 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 34054/269688 objects degraded (12.627%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:44.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 96305 96289 107.438 82.625 0.0138884 0.00929934 2024-09-14T03:58:45.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[106969]: from='client.35433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:45.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[106969]: from='client.35439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:45.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[106969]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:58:45.538 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[106969]: osd.3 [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847] boot 2024-09-14T03:58:45.538 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[106969]: osdmap e136: 8 total, 8 up, 8 in 2024-09-14T03:58:45.538 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-14T03:58:45.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[103029]: from='client.35433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:45.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[103029]: from='client.35439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:45.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[103029]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:58:45.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[103029]: osd.3 [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847] boot 2024-09-14T03:58:45.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[103029]: osdmap e136: 8 total, 8 up, 8 in 2024-09-14T03:58:45.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-14T03:58:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:45 smithi160 ceph-mon[88407]: from='client.35433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:45 smithi160 ceph-mon[88407]: from='client.35439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:45 smithi160 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-14T03:58:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:45 smithi160 ceph-mon[88407]: osd.3 [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847] boot 2024-09-14T03:58:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:45 smithi160 ceph-mon[88407]: osdmap e136: 8 total, 8 up, 8 in 2024-09-14T03:58:45.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:45 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-14T03:58:45.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 96314 96298 105.563 0.5625 0.00335784 0.00929903 2024-09-14T03:58:45.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:58:45 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: 2024-09-14T03:58:45.668+0000 7fd419828640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-14T03:58:46.041 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev)": 4 2024-09-14T03:58:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: pgmap v358: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 35525/282849 objects degraded (12.560%) 2024-09-14T03:58:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:58:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:58:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: osdmap e137: 8 total, 8 up, 8 in 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3477756982' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: pgmap v358: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 35525/282849 objects degraded (12.560%) 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:46.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:46.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:46.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:46.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-14T03:58:46.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: osdmap e137: 8 total, 8 up, 8 in 2024-09-14T03:58:46.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3477756982' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: pgmap v358: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 35525/282849 objects degraded (12.560%) 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: osdmap e137: 8 total, 8 up, 8 in 2024-09-14T03:58:46.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3477756982' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-14T03:58:46.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 96314 96298 103.743 0 - 0.00929903 2024-09-14T03:58:47.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch upgrade pause' 2024-09-14T03:58:47.448 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:47 smithi114 ceph-mon[106969]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-14T03:58:47.449 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:47 smithi114 ceph-mon[106969]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-14T03:58:47.449 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:47 smithi114 ceph-mon[106969]: osdmap e138: 8 total, 8 up, 8 in 2024-09-14T03:58:47.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:47 smithi114 ceph-mon[103029]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-14T03:58:47.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:47 smithi114 ceph-mon[103029]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-14T03:58:47.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:47 smithi114 ceph-mon[103029]: osdmap e138: 8 total, 8 up, 8 in 2024-09-14T03:58:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:47 smithi160 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-14T03:58:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:47 smithi160 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-14T03:58:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:47 smithi160 ceph-mon[88407]: osdmap e138: 8 total, 8 up, 8 in 2024-09-14T03:58:47.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 97090 97074 102.807 24.25 0.00929917 0.00972126 2024-09-14T03:58:47.859 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:58:48.331 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:48 smithi114 ceph-mon[103029]: pgmap v361: 89 pgs: 26 active+undersized, 9 active+undersized+degraded, 54 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 62 MiB/s wr, 998 op/s; 35525/282849 objects degraded (12.560%) 2024-09-14T03:58:48.332 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:48 smithi114 ceph-mon[106969]: pgmap v361: 89 pgs: 26 active+undersized, 9 active+undersized+degraded, 54 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 62 MiB/s wr, 998 op/s; 35525/282849 objects degraded (12.560%) 2024-09-14T03:58:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:48 smithi160 ceph-mon[88407]: pgmap v361: 89 pgs: 26 active+undersized, 9 active+undersized+degraded, 54 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 62 MiB/s wr, 998 op/s; 35525/282849 objects degraded (12.560%) 2024-09-14T03:58:48.810 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:58:48.810390+0000 min lat: 0.0011608 max lat: 2.32524 avg lat: 0.00977755 2024-09-14T03:58:48.811 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:58:48.811 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 98166 98150 102.214 67.25 0.0112891 0.00977755 2024-09-14T03:58:49.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:49 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35525/282849 objects degraded (12.560%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:49.469 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:49 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35525/282849 objects degraded (12.560%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:49.665 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:49 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35525/282849 objects degraded (12.560%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:49.811 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 99224 99208 101.623 66.125 0.00937001 0.00983469 2024-09-14T03:58:50.345 INFO:teuthology.orchestra.run.smithi114.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:58:50.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:50 smithi114 ceph-mon[106969]: pgmap v363: 89 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized, 3 active+undersized+degraded, 75 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 18405/289365 objects degraded (6.360%) 2024-09-14T03:58:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:50 smithi114 ceph-mon[103029]: pgmap v363: 89 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized, 3 active+undersized+degraded, 75 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 18405/289365 objects degraded (6.360%) 2024-09-14T03:58:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:50 smithi160 ceph-mon[88407]: pgmap v363: 89 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized, 3 active+undersized+degraded, 75 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 18405/289365 objects degraded (6.360%) 2024-09-14T03:58:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:58:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:58:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:58:50.811 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 100210 100194 100.978 61.625 0.0102522 0.0098979 2024-09-14T03:58:51.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch ps' 2024-09-14T03:58:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:51 smithi114 ceph-mon[103029]: from='client.35451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:51 smithi114 ceph-mon[103029]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:58:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:51 smithi114 ceph-mon[103029]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 33 MiB/s wr, 533 op/s; 9844/292455 objects degraded (3.366%); 629 KiB/s, 9 objects/s recovering 2024-09-14T03:58:51.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:58:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:58:51] "GET /metrics HTTP/1.1" 200 36381 "" "Prometheus/2.51.0" 2024-09-14T03:58:51.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:51 smithi114 ceph-mon[106969]: from='client.35451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:51.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:51 smithi114 ceph-mon[106969]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:58:51.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:51.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:51 smithi114 ceph-mon[106969]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 33 MiB/s wr, 533 op/s; 9844/292455 objects degraded (3.366%); 629 KiB/s, 9 objects/s recovering 2024-09-14T03:58:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:51 smithi160 ceph-mon[88407]: from='client.35451 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:51 smithi160 ceph-mon[88407]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T03:58:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:51 smithi160 ceph-mon[88407]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 33 MiB/s wr, 533 op/s; 9844/292455 objects degraded (3.366%); 629 KiB/s, 9 objects/s recovering 2024-09-14T03:58:51.739 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:58:51.813 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 101195 101179 100.348 61.5625 0.0122773 0.00996049 2024-09-14T03:58:52.813 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 102187 102171 99.7493 62 0.0190245 0.0100199 2024-09-14T03:58:53.813 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 103217 103201 99.2051 64.375 0.0320635 0.0100747 2024-09-14T03:58:53.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:53 smithi114 ceph-mon[106969]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 33 MiB/s wr, 534 op/s; 9840/295242 objects degraded (3.333%); 521 KiB/s, 8 objects/s recovering 2024-09-14T03:58:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:53 smithi114 ceph-mon[103029]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 33 MiB/s wr, 534 op/s; 9840/295242 objects degraded (3.333%); 521 KiB/s, 8 objects/s recovering 2024-09-14T03:58:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:53 smithi160 ceph-mon[88407]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 33 MiB/s wr, 534 op/s; 9840/295242 objects degraded (3.333%); 521 KiB/s, 8 objects/s recovering 2024-09-14T03:58:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:58:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:58:54.482 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-14T03:58:54.482 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.a smithi114 *:9093,9094 running (12m) 18s ago 16m 19.1M - 0.27.0 11f11916f8cd eaf39a5cb581 2024-09-14T03:58:54.482 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.a smithi160 *:3000 running (11m) 45s ago 16m 86.2M - 9.4.7 954c08fa6188 aa46fcb7cedc 2024-09-14T03:58:54.482 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi114.ebmrat smithi114 running (15m) 18s ago 15m 20.1M - 17.2.7-1629-geb16b97b ab51d1bebd12 00f4a6ea534e 2024-09-14T03:58:54.482 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi160.ywyjae smithi160 running (15m) 45s ago 15m 16.2M - 17.2.7-1629-geb16b97b ab51d1bebd12 a87b5ff7e907 2024-09-14T03:58:54.482 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x smithi160 *:8443,9283,8765 running (10m) 45s ago 20m 458M - 19.3.0-4907-ga2782315 8801d2529e43 574eaed954d9 2024-09-14T03:58:54.482 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y smithi114 *:8443,9283,8765 running (11m) 18s ago 22m 520M - 19.3.0-4907-ga2782315 8801d2529e43 4f6e65854aef 2024-09-14T03:58:54.482 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a smithi114 running (10m) 18s ago 22m 63.7M 2048M 19.3.0-4907-ga2782315 8801d2529e43 2ca35e38ecf5 2024-09-14T03:58:54.482 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b smithi160 running (8m) 45s ago 20m 47.8M 2048M 19.3.0-4907-ga2782315 8801d2529e43 c470c18a013d 2024-09-14T03:58:54.482 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c smithi114 running (8m) 18s ago 20m 48.6M 2048M 19.3.0-4907-ga2782315 8801d2529e43 0d84986478c2 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.a smithi114 *:9100 running (11m) 18s ago 16m 10.7M - 1.7.0 72c9c2088986 ff1fe8404b33 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.b smithi160 *:9100 running (11m) 45s ago 16m 10.9M - 1.7.0 72c9c2088986 289de68494ec 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (6m) 18s ago 20m 679M 2480M 19.3.0-4907-ga2782315 8801d2529e43 bde17c62ee62 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (4m) 18s ago 19m 420M 2480M 19.3.0-4907-ga2782315 8801d2529e43 724ed5764620 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (3m) 18s ago 19m 314M 2480M 19.3.0-4907-ga2782315 8801d2529e43 f3696f57b771 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi114 running (23s) 18s ago 19m 12.2M 2480M 19.3.0-4907-ga2782315 8801d2529e43 10460276d62f 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi160 running (18m) 45s ago 18m 580M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 e89693bb3b76 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi160 running (18m) 45s ago 18m 984M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 fb664f7f461a 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 smithi160 running (18m) 45s ago 18m 597M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 0403446dedd1 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 smithi160 running (17m) 45s ago 17m 624M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 39f0a617d104 2024-09-14T03:58:54.483 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.a smithi160 *:9095 running (10m) 45s ago 17m 56.2M - 2.51.0 1d3b7f56885b 1b09e4c45bc1 2024-09-14T03:58:54.813 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 104154 104138 98.5893 58.5625 0.00879342 0.0101381 2024-09-14T03:58:54.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:54 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9840/295242 objects degraded (3.333%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:54.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:54 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9840/295242 objects degraded (3.333%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:54 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9840/295242 objects degraded (3.333%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:55.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph orch ps' 2024-09-14T03:58:55.670 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:58:55.813 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 105129 105113 98.0274 60.9375 0.0124738 0.0101962 2024-09-14T03:58:56.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:56 smithi114 ceph-mon[106969]: from='client.35454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:56.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:56 smithi114 ceph-mon[106969]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 59 MiB/s wr, 948 op/s; 9612/307695 objects degraded (3.124%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:58:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:56 smithi114 ceph-mon[103029]: from='client.35454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:56 smithi114 ceph-mon[103029]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 59 MiB/s wr, 948 op/s; 9612/307695 objects degraded (3.124%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:58:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:56 smithi160 ceph-mon[88407]: from='client.35454 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:58:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:56 smithi160 ceph-mon[88407]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 59 MiB/s wr, 948 op/s; 9612/307695 objects degraded (3.124%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T03:58:56.813 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 106138 106122 97.5132 63.0625 0.0263253 0.0102499 2024-09-14T03:58:57.813 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 107132 107116 97.0003 62.125 0.0135391 0.0103043 2024-09-14T03:58:58.410 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-14T03:58:58.410 INFO:teuthology.orchestra.run.smithi114.stdout:alertmanager.a smithi114 *:9093,9094 running (12m) 22s ago 16m 19.1M - 0.27.0 11f11916f8cd eaf39a5cb581 2024-09-14T03:58:58.410 INFO:teuthology.orchestra.run.smithi114.stdout:grafana.a smithi160 *:3000 running (11m) 49s ago 16m 86.2M - 9.4.7 954c08fa6188 aa46fcb7cedc 2024-09-14T03:58:58.410 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi114.ebmrat smithi114 running (15m) 22s ago 15m 20.1M - 17.2.7-1629-geb16b97b ab51d1bebd12 00f4a6ea534e 2024-09-14T03:58:58.410 INFO:teuthology.orchestra.run.smithi114.stdout:mds.foo.smithi160.ywyjae smithi160 running (15m) 49s ago 15m 16.2M - 17.2.7-1629-geb16b97b ab51d1bebd12 a87b5ff7e907 2024-09-14T03:58:58.410 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x smithi160 *:8443,9283,8765 running (10m) 49s ago 20m 458M - 19.3.0-4907-ga2782315 8801d2529e43 574eaed954d9 2024-09-14T03:58:58.410 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y smithi114 *:8443,9283,8765 running (11m) 22s ago 22m 520M - 19.3.0-4907-ga2782315 8801d2529e43 4f6e65854aef 2024-09-14T03:58:58.410 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a smithi114 running (10m) 22s ago 22m 63.7M 2048M 19.3.0-4907-ga2782315 8801d2529e43 2ca35e38ecf5 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b smithi160 running (8m) 49s ago 21m 47.8M 2048M 19.3.0-4907-ga2782315 8801d2529e43 c470c18a013d 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c smithi114 running (8m) 22s ago 20m 48.6M 2048M 19.3.0-4907-ga2782315 8801d2529e43 0d84986478c2 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.a smithi114 *:9100 running (11m) 22s ago 16m 10.7M - 1.7.0 72c9c2088986 ff1fe8404b33 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:node-exporter.b smithi160 *:9100 running (11m) 49s ago 16m 10.9M - 1.7.0 72c9c2088986 289de68494ec 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (6m) 22s ago 20m 679M 2480M 19.3.0-4907-ga2782315 8801d2529e43 bde17c62ee62 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi114 running (5m) 22s ago 20m 420M 2480M 19.3.0-4907-ga2782315 8801d2529e43 724ed5764620 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi114 running (3m) 22s ago 19m 314M 2480M 19.3.0-4907-ga2782315 8801d2529e43 f3696f57b771 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi114 running (27s) 22s ago 19m 12.2M 2480M 19.3.0-4907-ga2782315 8801d2529e43 10460276d62f 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi160 running (18m) 49s ago 18m 580M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 e89693bb3b76 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi160 running (18m) 49s ago 18m 984M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 fb664f7f461a 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 smithi160 running (18m) 49s ago 18m 597M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 0403446dedd1 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 smithi160 running (17m) 49s ago 17m 624M 2480M 17.2.7-1629-geb16b97b ab51d1bebd12 39f0a617d104 2024-09-14T03:58:58.411 INFO:teuthology.orchestra.run.smithi114.stdout:prometheus.a smithi160 *:9095 running (10m) 49s ago 17m 56.2M - 2.51.0 1d3b7f56885b 1b09e4c45bc1 2024-09-14T03:58:58.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:58 smithi114 ceph-mon[106969]: pgmap v367: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s; 9612/307695 objects degraded (3.124%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:58:58.412 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:58 smithi114 ceph-mon[103029]: pgmap v367: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s; 9612/307695 objects degraded (3.124%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:58:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:58 smithi160 ceph-mon[88407]: pgmap v367: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s; 9612/307695 objects degraded (3.124%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T03:58:58.813 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 107998 107982 96.3878 54.125 0.0204951 0.0103693 2024-09-14T03:58:59.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -e sha1=a2782315255a3e7d5f6cdc9eb35785187ff6ddf8 -- bash -c 'ceph versions' 2024-09-14T03:58:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:58:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9612/307695 objects degraded (3.124%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:58:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9612/307695 objects degraded (3.124%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:58:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9612/307695 objects degraded (3.124%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:58:59.761 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T03:58:59.814 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 108870 108854 95.7975 54.5 0.0161964 0.0104336 2024-09-14T03:59:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:00 smithi160 ceph-mon[88407]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:59:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:00 smithi160 ceph-mon[88407]: pgmap v368: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 62 MiB/s wr, 987 op/s; 9612/317631 objects degraded (3.026%); 1.5 MiB/s, 24 objects/s recovering 2024-09-14T03:59:00.559 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:59:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:59:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:59:00.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:00 smithi114 ceph-mon[106969]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:59:00.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:00 smithi114 ceph-mon[106969]: pgmap v368: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 62 MiB/s wr, 987 op/s; 9612/317631 objects degraded (3.026%); 1.5 MiB/s, 24 objects/s recovering 2024-09-14T03:59:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:00 smithi114 ceph-mon[103029]: from='client.35460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-14T03:59:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:00 smithi114 ceph-mon[103029]: pgmap v368: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 62 MiB/s wr, 987 op/s; 9612/317631 objects degraded (3.026%); 1.5 MiB/s, 24 objects/s recovering 2024-09-14T03:59:00.814 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 109794 109778 95.2691 57.75 0.0198638 0.0104916 2024-09-14T03:59:01.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:59:01.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:59:01.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:59:01.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:59:01.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:59:01.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:59:01.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:59:01.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:59:01.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:59:01.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:59:01.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:59:01.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:59:01.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:59:01.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:59:01.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:59:01.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:59:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:59:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:59:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:59:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:59:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T03:59:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T03:59:01.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T03:59:01.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T03:59:01.769 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:59:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:59:01] "GET /metrics HTTP/1.1" 200 36376 "" "Prometheus/2.51.0" 2024-09-14T03:59:01.814 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 110694 110678 94.7346 56.25 0.0112161 0.0105503 2024-09-14T03:59:02.418 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T03:59:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "mon": { 2024-09-14T03:59:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev)": 3 2024-09-14T03:59:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T03:59:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr": { 2024-09-14T03:59:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev)": 2 2024-09-14T03:59:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T03:59:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "osd": { 2024-09-14T03:59:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)": 4, 2024-09-14T03:59:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev)": 4 2024-09-14T03:59:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T03:59:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-09-14T03:59:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)": 2 2024-09-14T03:59:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T03:59:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "overall": { 2024-09-14T03:59:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 17.2.7-1629-geb16b97b (eb16b97b08b1a378bbb6cbc2dadc21f07d8ac10b) quincy (stable)": 6, 2024-09-14T03:59:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph version 19.3.0-4907-ga2782315 (a2782315255a3e7d5f6cdc9eb35785187ff6ddf8) squid (dev)": 9 2024-09-14T03:59:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T03:59:02.419 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T03:59:02.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:02 smithi114 ceph-mon[103029]: pgmap v369: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s; 9369/322314 objects degraded (2.907%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T03:59:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:02 smithi114 ceph-mon[106969]: pgmap v369: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s; 9369/322314 objects degraded (2.907%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T03:59:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:02 smithi160 ceph-mon[88407]: pgmap v369: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 914 op/s; 9369/322314 objects degraded (2.907%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T03:59:02.814 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 111587 111571 94.2086 55.8125 0.0131782 0.0106088 2024-09-14T03:59:03.457 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:03 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/591547394' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:59:03.458 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:03 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/591547394' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:59:03.500 DEBUG:teuthology.parallel:result is None 2024-09-14T03:59:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:03 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/591547394' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-14T03:59:03.814 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 112557 112541 93.7609 60.625 0.0134822 0.0106606 2024-09-14T03:59:04.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:04 smithi114 ceph-mon[106969]: pgmap v370: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 900 op/s; 9369/324897 objects degraded (2.884%); 2.5 MiB/s, 39 objects/s recovering 2024-09-14T03:59:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9369/324897 objects degraded (2.884%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:04.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:04 smithi114 ceph-mon[103029]: pgmap v370: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 900 op/s; 9369/324897 objects degraded (2.884%); 2.5 MiB/s, 39 objects/s recovering 2024-09-14T03:59:04.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9369/324897 objects degraded (2.884%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:04 smithi160 ceph-mon[88407]: pgmap v370: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 900 op/s; 9369/324897 objects degraded (2.884%); 2.5 MiB/s, 39 objects/s recovering 2024-09-14T03:59:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9369/324897 objects degraded (2.884%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:04.814 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 113534 113518 93.3306 61.0625 0.0353239 0.0107093 2024-09-14T03:59:05.814 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 114504 114488 92.9059 60.625 0.0122946 0.0107581 2024-09-14T03:59:06.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:06 smithi114 ceph-mon[106969]: pgmap v371: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 9106/335751 objects degraded (2.712%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T03:59:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:06 smithi114 ceph-mon[103029]: pgmap v371: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 9106/335751 objects degraded (2.712%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T03:59:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:06 smithi160 ceph-mon[88407]: pgmap v371: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 9106/335751 objects degraded (2.712%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T03:59:06.814 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 115496 115480 92.5096 62 0.00954316 0.0108041 2024-09-14T03:59:07.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 116462 116446 92.1028 60.375 0.00772508 0.0108521 2024-09-14T03:59:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:08 smithi114 ceph-mon[103029]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 9106/335751 objects degraded (2.712%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:59:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:08 smithi114 ceph-mon[106969]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 9106/335751 objects degraded (2.712%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:59:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:08 smithi160 ceph-mon[88407]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 9106/335751 objects degraded (2.712%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:59:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:08.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T03:59:08.814656+0000 min lat: 0.0011608 max lat: 2.32524 avg lat: 0.0109091 2024-09-14T03:59:08.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T03:59:08.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 117324 117308 91.625 53.875 0.01931 0.0109091 2024-09-14T03:59:09.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9106/335751 objects degraded (2.712%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:09.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9106/335751 objects degraded (2.712%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:09.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9106/335751 objects degraded (2.712%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:09.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 118277 118261 91.2291 59.5625 0.00620821 0.0109562 2024-09-14T03:59:10.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:10 smithi114 ceph-mon[106969]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 9106/345954 objects degraded (2.632%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:59:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:10 smithi114 ceph-mon[103029]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 9106/345954 objects degraded (2.632%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:59:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:10 smithi160 ceph-mon[88407]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 9106/345954 objects degraded (2.632%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T03:59:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:59:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:59:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:59:10.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 119218 119202 90.8338 58.8125 0.00849229 0.0110037 2024-09-14T03:59:11.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 120145 120129 90.4375 57.9375 0.00894887 0.0110523 2024-09-14T03:59:11.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:59:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:59:11] "GET /metrics HTTP/1.1" 200 36386 "" "Prometheus/2.51.0" 2024-09-14T03:59:12.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:12 smithi114 ceph-mon[106969]: pgmap v374: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s; 8798/350394 objects degraded (2.511%); 4.2 MiB/s, 67 objects/s recovering 2024-09-14T03:59:12.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:12 smithi114 ceph-mon[103029]: pgmap v374: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s; 8798/350394 objects degraded (2.511%); 4.2 MiB/s, 67 objects/s recovering 2024-09-14T03:59:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:12 smithi160 ceph-mon[88407]: pgmap v374: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s; 8798/350394 objects degraded (2.511%); 4.2 MiB/s, 67 objects/s recovering 2024-09-14T03:59:12.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 121077 121061 90.0543 58.25 0.014226 0.0110995 2024-09-14T03:59:13.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 122001 121985 89.6742 57.75 0.0117271 0.0111462 2024-09-14T03:59:14.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:14 smithi114 ceph-mon[106969]: pgmap v375: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 8798/353025 objects degraded (2.492%); 3.0 MiB/s, 47 objects/s recovering 2024-09-14T03:59:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8798/353025 objects degraded (2.492%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:14 smithi114 ceph-mon[103029]: pgmap v375: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 8798/353025 objects degraded (2.492%); 3.0 MiB/s, 47 objects/s recovering 2024-09-14T03:59:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8798/353025 objects degraded (2.492%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:14 smithi160 ceph-mon[88407]: pgmap v375: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 8798/353025 objects degraded (2.492%); 3.0 MiB/s, 47 objects/s recovering 2024-09-14T03:59:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8798/353025 objects degraded (2.492%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:14.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 122949 122933 89.3204 59.25 0.0225966 0.0111905 2024-09-14T03:59:15.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 15 123893 123878 88.9726 59.0625 0.030218 0.011234 2024-09-14T03:59:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:16 smithi114 ceph-mon[106969]: pgmap v376: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 8488/364251 objects degraded (2.330%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T03:59:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:16 smithi114 ceph-mon[103029]: pgmap v376: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 8488/364251 objects degraded (2.330%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T03:59:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:16 smithi160 ceph-mon[88407]: pgmap v376: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 8488/364251 objects degraded (2.330%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T03:59:16.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 124828 124812 88.6249 58.375 0.016292 0.0112782 2024-09-14T03:59:17.815 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 125516 125500 88.1123 43 0.00384196 0.0112977 2024-09-14T03:59:18.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:18 smithi114 ceph-mon[103029]: pgmap v377: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 49 MiB/s wr, 791 op/s; 8488/364251 objects degraded (2.330%); 3.2 MiB/s, 51 objects/s recovering 2024-09-14T03:59:18.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:18 smithi114 ceph-mon[103029]: osdmap e139: 8 total, 8 up, 8 in 2024-09-14T03:59:18.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:18 smithi114 ceph-mon[106969]: pgmap v377: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 49 MiB/s wr, 791 op/s; 8488/364251 objects degraded (2.330%); 3.2 MiB/s, 51 objects/s recovering 2024-09-14T03:59:18.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:18 smithi114 ceph-mon[106969]: osdmap e139: 8 total, 8 up, 8 in 2024-09-14T03:59:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:18 smithi160 ceph-mon[88407]: pgmap v377: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 49 MiB/s wr, 791 op/s; 8488/364251 objects degraded (2.330%); 3.2 MiB/s, 51 objects/s recovering 2024-09-14T03:59:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:18 smithi160 ceph-mon[88407]: osdmap e139: 8 total, 8 up, 8 in 2024-09-14T03:59:18.816 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0195 2024-09-14T03:59:18.816 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 125969 2024-09-14T03:59:18.816 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-14T03:59:18.816 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-14T03:59:18.816 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 87.4596 2024-09-14T03:59:18.816 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 44.8895 2024-09-14T03:59:18.816 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 237.188 2024-09-14T03:59:18.817 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 0 2024-09-14T03:59:18.817 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1399 2024-09-14T03:59:18.817 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 718.232 2024-09-14T03:59:18.817 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3795 2024-09-14T03:59:18.817 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 0 2024-09-14T03:59:18.817 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0114285 2024-09-14T03:59:18.817 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0313026 2024-09-14T03:59:18.818 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 2.32524 2024-09-14T03:59:18.818 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.0011608 2024-09-14T03:59:18.821 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-14T03:59:19.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:19 smithi114 ceph-mon[103029]: osdmap e140: 8 total, 8 up, 8 in 2024-09-14T03:59:19.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8488/364251 objects degraded (2.330%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:19.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:19 smithi114 ceph-mon[106969]: osdmap e140: 8 total, 8 up, 8 in 2024-09-14T03:59:19.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8488/364251 objects degraded (2.330%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:19 smithi160 ceph-mon[88407]: osdmap e140: 8 total, 8 up, 8 in 2024-09-14T03:59:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8488/364251 objects degraded (2.330%), 6 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:20.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:20 smithi114 ceph-mon[103029]: pgmap v380: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s; 8106/373512 objects degraded (2.170%); 5.4 MiB/s, 86 objects/s recovering 2024-09-14T03:59:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:20 smithi114 ceph-mon[106969]: pgmap v380: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s; 8106/373512 objects degraded (2.170%); 5.4 MiB/s, 86 objects/s recovering 2024-09-14T03:59:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:20 smithi160 ceph-mon[88407]: pgmap v380: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s; 8106/373512 objects degraded (2.170%); 5.4 MiB/s, 86 objects/s recovering 2024-09-14T03:59:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:59:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:59:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:59:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:59:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:59:21] "GET /metrics HTTP/1.1" 200 36386 "" "Prometheus/2.51.0" 2024-09-14T03:59:22.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:22 smithi114 ceph-mon[103029]: pgmap v381: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 982 op/s; 8106/376614 objects degraded (2.152%); 5.4 MiB/s, 86 objects/s recovering 2024-09-14T03:59:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:22 smithi114 ceph-mon[106969]: pgmap v381: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 982 op/s; 8106/376614 objects degraded (2.152%); 5.4 MiB/s, 86 objects/s recovering 2024-09-14T03:59:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:22 smithi160 ceph-mon[88407]: pgmap v381: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 982 op/s; 8106/376614 objects degraded (2.152%); 5.4 MiB/s, 86 objects/s recovering 2024-09-14T03:59:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:23.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:24.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:24 smithi114 ceph-mon[103029]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 34 MiB/s wr, 610 op/s; 8106/375465 objects degraded (2.159%); 3.0 MiB/s, 47 objects/s recovering 2024-09-14T03:59:24.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8106/375465 objects degraded (2.159%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:24 smithi114 ceph-mon[106969]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 34 MiB/s wr, 610 op/s; 8106/375465 objects degraded (2.159%); 3.0 MiB/s, 47 objects/s recovering 2024-09-14T03:59:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8106/375465 objects degraded (2.159%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:24 smithi160 ceph-mon[88407]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 34 MiB/s wr, 610 op/s; 8106/375465 objects degraded (2.159%); 3.0 MiB/s, 47 objects/s recovering 2024-09-14T03:59:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8106/375465 objects degraded (2.159%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:25 smithi114 ceph-mon[103029]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 36 MiB/s wr, 1.09k op/s; 7904/365652 objects degraded (2.162%); 4.6 MiB/s, 72 objects/s recovering 2024-09-14T03:59:25.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:25 smithi114 ceph-mon[106969]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 36 MiB/s wr, 1.09k op/s; 7904/365652 objects degraded (2.162%); 4.6 MiB/s, 72 objects/s recovering 2024-09-14T03:59:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:25 smithi160 ceph-mon[88407]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 127 B/s rd, 36 MiB/s wr, 1.09k op/s; 7904/365652 objects degraded (2.162%); 4.6 MiB/s, 72 objects/s recovering 2024-09-14T03:59:27.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:27 smithi114 ceph-mon[106969]: pgmap v384: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 110 B/s rd, 25 MiB/s wr, 845 op/s; 7904/365652 objects degraded (2.162%); 3.9 MiB/s, 63 objects/s recovering 2024-09-14T03:59:27.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:27 smithi114 ceph-mon[103029]: pgmap v384: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 110 B/s rd, 25 MiB/s wr, 845 op/s; 7904/365652 objects degraded (2.162%); 3.9 MiB/s, 63 objects/s recovering 2024-09-14T03:59:27.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:27 smithi160 ceph-mon[88407]: pgmap v384: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 110 B/s rd, 25 MiB/s wr, 845 op/s; 7904/365652 objects degraded (2.162%); 3.9 MiB/s, 63 objects/s recovering 2024-09-14T03:59:29.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:29 smithi114 ceph-mon[106969]: pgmap v385: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 99 B/s rd, 23 MiB/s wr, 1.11k op/s; 7674/355008 objects degraded (2.162%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T03:59:29.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7674/355008 objects degraded (2.162%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:29.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:29 smithi114 ceph-mon[103029]: pgmap v385: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 99 B/s rd, 23 MiB/s wr, 1.11k op/s; 7674/355008 objects degraded (2.162%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T03:59:29.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7674/355008 objects degraded (2.162%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:29 smithi160 ceph-mon[88407]: pgmap v385: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 99 B/s rd, 23 MiB/s wr, 1.11k op/s; 7674/355008 objects degraded (2.162%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T03:59:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7674/355008 objects degraded (2.162%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:59:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:59:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:59:31.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:31 smithi114 ceph-mon[103029]: pgmap v386: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 7.8 MiB/s wr, 859 op/s; 7674/351528 objects degraded (2.183%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:59:31.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:59:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:59:31] "GET /metrics HTTP/1.1" 200 36382 "" "Prometheus/2.51.0" 2024-09-14T03:59:31.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:31 smithi114 ceph-mon[106969]: pgmap v386: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 7.8 MiB/s wr, 859 op/s; 7674/351528 objects degraded (2.183%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:59:31.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:31 smithi160 ceph-mon[88407]: pgmap v386: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 7.8 MiB/s wr, 859 op/s; 7674/351528 objects degraded (2.183%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:59:33.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:33 smithi114 ceph-mon[103029]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 2.4 MiB/s wr, 845 op/s; 7674/348927 objects degraded (2.199%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:59:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:33 smithi114 ceph-mon[106969]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 2.4 MiB/s wr, 845 op/s; 7674/348927 objects degraded (2.199%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:59:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:33 smithi160 ceph-mon[88407]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 2.4 MiB/s wr, 845 op/s; 7674/348927 objects degraded (2.199%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T03:59:34.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7674/348927 objects degraded (2.199%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:34.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7674/348927 objects degraded (2.199%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7674/348927 objects degraded (2.199%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:35.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:35 smithi114 ceph-mon[103029]: pgmap v388: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 1.4 MiB/s wr, 1.08k op/s; 7460/338205 objects degraded (2.206%); 3.4 MiB/s, 53 objects/s recovering 2024-09-14T03:59:35.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:35 smithi114 ceph-mon[106969]: pgmap v388: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 1.4 MiB/s wr, 1.08k op/s; 7460/338205 objects degraded (2.206%); 3.4 MiB/s, 53 objects/s recovering 2024-09-14T03:59:35.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:35 smithi160 ceph-mon[88407]: pgmap v388: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 1.4 MiB/s wr, 1.08k op/s; 7460/338205 objects degraded (2.206%); 3.4 MiB/s, 53 objects/s recovering 2024-09-14T03:59:37.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:37 smithi114 ceph-mon[106969]: pgmap v389: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 7460/338205 objects degraded (2.206%); 2.3 MiB/s, 36 objects/s recovering 2024-09-14T03:59:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:37 smithi114 ceph-mon[103029]: pgmap v389: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 7460/338205 objects degraded (2.206%); 2.3 MiB/s, 36 objects/s recovering 2024-09-14T03:59:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:37 smithi160 ceph-mon[88407]: pgmap v389: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 7460/338205 objects degraded (2.206%); 2.3 MiB/s, 36 objects/s recovering 2024-09-14T03:59:38.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:39.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:39 smithi114 ceph-mon[103029]: pgmap v390: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 7209/327918 objects degraded (2.198%); 3.6 MiB/s, 57 objects/s recovering 2024-09-14T03:59:39.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:39 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7209/327918 objects degraded (2.198%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:39 smithi114 ceph-mon[106969]: pgmap v390: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 7209/327918 objects degraded (2.198%); 3.6 MiB/s, 57 objects/s recovering 2024-09-14T03:59:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:39 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7209/327918 objects degraded (2.198%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:39 smithi160 ceph-mon[88407]: pgmap v390: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 7209/327918 objects degraded (2.198%); 3.6 MiB/s, 57 objects/s recovering 2024-09-14T03:59:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:39 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7209/327918 objects degraded (2.198%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:59:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:59:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:59:41.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:41 smithi114 ceph-mon[103029]: pgmap v391: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 7209/324495 objects degraded (2.222%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T03:59:41.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:59:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:59:41] "GET /metrics HTTP/1.1" 200 36383 "" "Prometheus/2.51.0" 2024-09-14T03:59:41.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:41 smithi114 ceph-mon[106969]: pgmap v391: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 7209/324495 objects degraded (2.222%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T03:59:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:41 smithi160 ceph-mon[88407]: pgmap v391: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 7209/324495 objects degraded (2.222%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T03:59:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:43 smithi160 ceph-mon[88407]: pgmap v392: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 7209/321963 objects degraded (2.239%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T03:59:44.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:43 smithi114 ceph-mon[106969]: pgmap v392: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 7209/321963 objects degraded (2.239%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T03:59:44.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:43 smithi114 ceph-mon[103029]: pgmap v392: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 7209/321963 objects degraded (2.239%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T03:59:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:44 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7209/321963 objects degraded (2.239%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:45.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:44 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7209/321963 objects degraded (2.239%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:45.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:44 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7209/321963 objects degraded (2.239%), 5 pgs degraded (PG_DEGRADED) 2024-09-14T03:59:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:45 smithi160 ceph-mon[88407]: pgmap v393: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 6882/311328 objects degraded (2.211%); 4.1 MiB/s, 65 objects/s recovering 2024-09-14T03:59:46.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:45 smithi114 ceph-mon[103029]: pgmap v393: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 6882/311328 objects degraded (2.211%); 4.1 MiB/s, 65 objects/s recovering 2024-09-14T03:59:46.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:45 smithi114 ceph-mon[106969]: pgmap v393: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 6882/311328 objects degraded (2.211%); 4.1 MiB/s, 65 objects/s recovering 2024-09-14T03:59:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:47 smithi160 ceph-mon[88407]: pgmap v394: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 6882/311328 objects degraded (2.211%); 3.0 MiB/s, 48 objects/s recovering 2024-09-14T03:59:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:47 smithi160 ceph-mon[88407]: osdmap e141: 8 total, 8 up, 8 in 2024-09-14T03:59:48.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:47 smithi114 ceph-mon[103029]: pgmap v394: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 6882/311328 objects degraded (2.211%); 3.0 MiB/s, 48 objects/s recovering 2024-09-14T03:59:48.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:47 smithi114 ceph-mon[103029]: osdmap e141: 8 total, 8 up, 8 in 2024-09-14T03:59:48.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:47 smithi114 ceph-mon[106969]: pgmap v394: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 6882/311328 objects degraded (2.211%); 3.0 MiB/s, 48 objects/s recovering 2024-09-14T03:59:48.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:47 smithi114 ceph-mon[106969]: osdmap e141: 8 total, 8 up, 8 in 2024-09-14T03:59:49.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:48 smithi114 ceph-mon[103029]: osdmap e142: 8 total, 8 up, 8 in 2024-09-14T03:59:49.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:48 smithi114 ceph-mon[106969]: osdmap e142: 8 total, 8 up, 8 in 2024-09-14T03:59:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:48 smithi160 ceph-mon[88407]: osdmap e142: 8 total, 8 up, 8 in 2024-09-14T03:59:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:49 smithi114 ceph-mon[103029]: pgmap v397: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 920 op/s; 6619/302406 objects degraded (2.189%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T03:59:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:49 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6619/302406 objects degraded (2.189%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:59:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:49 smithi114 ceph-mon[106969]: pgmap v397: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 920 op/s; 6619/302406 objects degraded (2.189%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T03:59:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:49 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6619/302406 objects degraded (2.189%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:59:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:49 smithi160 ceph-mon[88407]: pgmap v397: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 920 op/s; 6619/302406 objects degraded (2.189%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T03:59:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:49 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6619/302406 objects degraded (2.189%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:59:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 03:59:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T03:59:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T03:59:51.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:51 smithi114 ceph-mon[103029]: pgmap v398: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 926 op/s; 6619/299715 objects degraded (2.208%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T03:59:51.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 03:59:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:03:59:51] "GET /metrics HTTP/1.1" 200 36383 "" "Prometheus/2.51.0" 2024-09-14T03:59:51.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:51 smithi114 ceph-mon[106969]: pgmap v398: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 926 op/s; 6619/299715 objects degraded (2.208%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T03:59:52.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:51 smithi160 ceph-mon[88407]: pgmap v398: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 926 op/s; 6619/299715 objects degraded (2.208%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T03:59:53.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:53.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:52 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:52 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T03:59:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:53 smithi114 ceph-mon[103029]: pgmap v399: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 6322/297639 objects degraded (2.124%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T03:59:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:53 smithi114 ceph-mon[106969]: pgmap v399: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 6322/297639 objects degraded (2.124%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T03:59:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:53 smithi160 ceph-mon[88407]: pgmap v399: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 6322/297639 objects degraded (2.124%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T03:59:55.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:54 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6322/297639 objects degraded (2.124%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:59:55.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:54 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6322/297639 objects degraded (2.124%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:59:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:54 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6322/297639 objects degraded (2.124%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T03:59:56.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:55 smithi114 ceph-mon[106969]: pgmap v400: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 6322/286908 objects degraded (2.203%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T03:59:56.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:55 smithi114 ceph-mon[103029]: pgmap v400: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 6322/286908 objects degraded (2.203%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T03:59:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:55 smithi160 ceph-mon[88407]: pgmap v400: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 6322/286908 objects degraded (2.203%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T03:59:58.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:57 smithi114 ceph-mon[103029]: pgmap v401: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 823 op/s; 6322/286908 objects degraded (2.203%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T03:59:58.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:57 smithi114 ceph-mon[106969]: pgmap v401: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 823 op/s; 6322/286908 objects degraded (2.203%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T03:59:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:57 smithi160 ceph-mon[88407]: pgmap v401: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 823 op/s; 6322/286908 objects degraded (2.203%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T04:00:00.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:59 smithi114 ceph-mon[106969]: pgmap v402: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 6114/276909 objects degraded (2.208%); 4.0 MiB/s, 64 objects/s recovering 2024-09-14T04:00:00.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 03:59:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6114/276909 objects degraded (2.208%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:00.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:59 smithi114 ceph-mon[103029]: pgmap v402: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 6114/276909 objects degraded (2.208%); 4.0 MiB/s, 64 objects/s recovering 2024-09-14T04:00:00.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 03:59:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6114/276909 objects degraded (2.208%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:00.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:59 smithi160 ceph-mon[88407]: pgmap v402: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 6114/276909 objects degraded (2.208%); 4.0 MiB/s, 64 objects/s recovering 2024-09-14T04:00:00.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 03:59:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6114/276909 objects degraded (2.208%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:00:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:00:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:00:01.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[103029]: Health detail: HEALTH_WARN Degraded data redundancy: 6114/276909 objects degraded (2.208%), 4 pgs degraded, 4 pgs undersized 2024-09-14T04:00:01.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[103029]: [WRN] PG_DEGRADED: Degraded data redundancy: 6114/276909 objects degraded (2.208%), 4 pgs degraded, 4 pgs undersized 2024-09-14T04:00:01.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[103029]: pg 8.7 is stuck undersized for 71s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,6] 2024-09-14T04:00:01.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[103029]: pg 8.8 is stuck undersized for 71s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,7] 2024-09-14T04:00:01.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[103029]: pg 8.c is stuck undersized for 71s, current state active+recovering+undersized+degraded+remapped, last acting [0,7] 2024-09-14T04:00:01.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[103029]: pg 8.f is stuck undersized for 71s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,2] 2024-09-14T04:00:01.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[106969]: Health detail: HEALTH_WARN Degraded data redundancy: 6114/276909 objects degraded (2.208%), 4 pgs degraded, 4 pgs undersized 2024-09-14T04:00:01.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[106969]: [WRN] PG_DEGRADED: Degraded data redundancy: 6114/276909 objects degraded (2.208%), 4 pgs degraded, 4 pgs undersized 2024-09-14T04:00:01.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[106969]: pg 8.7 is stuck undersized for 71s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,6] 2024-09-14T04:00:01.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[106969]: pg 8.8 is stuck undersized for 71s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,7] 2024-09-14T04:00:01.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[106969]: pg 8.c is stuck undersized for 71s, current state active+recovering+undersized+degraded+remapped, last acting [0,7] 2024-09-14T04:00:01.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:00 smithi114 ceph-mon[106969]: pg 8.f is stuck undersized for 71s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,2] 2024-09-14T04:00:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:00 smithi160 ceph-mon[88407]: Health detail: HEALTH_WARN Degraded data redundancy: 6114/276909 objects degraded (2.208%), 4 pgs degraded, 4 pgs undersized 2024-09-14T04:00:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:00 smithi160 ceph-mon[88407]: [WRN] PG_DEGRADED: Degraded data redundancy: 6114/276909 objects degraded (2.208%), 4 pgs degraded, 4 pgs undersized 2024-09-14T04:00:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:00 smithi160 ceph-mon[88407]: pg 8.7 is stuck undersized for 71s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,6] 2024-09-14T04:00:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:00 smithi160 ceph-mon[88407]: pg 8.8 is stuck undersized for 71s, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,7] 2024-09-14T04:00:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:00 smithi160 ceph-mon[88407]: pg 8.c is stuck undersized for 71s, current state active+recovering+undersized+degraded+remapped, last acting [0,7] 2024-09-14T04:00:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:00 smithi160 ceph-mon[88407]: pg 8.f is stuck undersized for 71s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,2] 2024-09-14T04:00:01.736 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:00:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:00:01] "GET /metrics HTTP/1.1" 200 36380 "" "Prometheus/2.51.0" 2024-09-14T04:00:01.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:01 smithi160 ceph-mon[88407]: pgmap v403: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 6114/273627 objects degraded (2.234%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T04:00:01.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:00:02.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:01 smithi114 ceph-mon[103029]: pgmap v403: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 6114/273627 objects degraded (2.234%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T04:00:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:00:02.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:01 smithi114 ceph-mon[106969]: pgmap v403: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 6114/273627 objects degraded (2.234%); 2.6 MiB/s, 42 objects/s recovering 2024-09-14T04:00:02.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:00:04.119 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:03 smithi114 ceph-mon[106969]: pgmap v404: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 5854/271434 objects degraded (2.157%); 4.0 MiB/s, 63 objects/s recovering 2024-09-14T04:00:04.119 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:03 smithi114 ceph-mon[103029]: pgmap v404: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 5854/271434 objects degraded (2.157%); 4.0 MiB/s, 63 objects/s recovering 2024-09-14T04:00:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:03 smithi160 ceph-mon[88407]: pgmap v404: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 785 op/s; 5854/271434 objects degraded (2.157%); 4.0 MiB/s, 63 objects/s recovering 2024-09-14T04:00:05.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 5854/271434 objects degraded (2.157%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:05.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 5854/271434 objects degraded (2.157%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:05.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 5854/271434 objects degraded (2.157%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:06.040 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:05 smithi114 ceph-mon[106969]: pgmap v405: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 977 op/s; 5854/262446 objects degraded (2.231%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:00:06.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:05 smithi114 ceph-mon[103029]: pgmap v405: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 977 op/s; 5854/262446 objects degraded (2.231%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:00:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:05 smithi160 ceph-mon[88407]: pgmap v405: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 977 op/s; 5854/262446 objects degraded (2.231%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:00:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:07 smithi160 ceph-mon[88407]: pgmap v406: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 5854/262446 objects degraded (2.231%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:00:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:07 smithi114 ceph-mon[106969]: pgmap v406: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 5854/262446 objects degraded (2.231%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:00:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:07 smithi114 ceph-mon[103029]: pgmap v406: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 5854/262446 objects degraded (2.231%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:00:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:00:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:00:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:00:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:00:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:00:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:00:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:00:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:09.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:09 smithi114 ceph-mon[106969]: pgmap v407: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 932 op/s; 5602/253338 objects degraded (2.211%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T04:00:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 5854/262446 objects degraded (2.231%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:09 smithi114 ceph-mon[103029]: pgmap v407: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 932 op/s; 5602/253338 objects degraded (2.211%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T04:00:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 5854/262446 objects degraded (2.231%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:09 smithi160 ceph-mon[88407]: pgmap v407: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 932 op/s; 5602/253338 objects degraded (2.211%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T04:00:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 5854/262446 objects degraded (2.231%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:00:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:00:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:00:11.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:11 smithi114 ceph-mon[103029]: pgmap v408: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 5602/250278 objects degraded (2.238%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:00:11.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:00:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:00:11] "GET /metrics HTTP/1.1" 200 36385 "" "Prometheus/2.51.0" 2024-09-14T04:00:11.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:11 smithi114 ceph-mon[106969]: pgmap v408: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 5602/250278 objects degraded (2.238%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:00:11.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:11 smithi160 ceph-mon[88407]: pgmap v408: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 5602/250278 objects degraded (2.238%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:00:13.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:13 smithi114 ceph-mon[103029]: pgmap v409: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 5317/247782 objects degraded (2.146%); 4.1 MiB/s, 66 objects/s recovering 2024-09-14T04:00:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:13 smithi114 ceph-mon[106969]: pgmap v409: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 5317/247782 objects degraded (2.146%); 4.1 MiB/s, 66 objects/s recovering 2024-09-14T04:00:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:13 smithi160 ceph-mon[88407]: pgmap v409: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 5317/247782 objects degraded (2.146%); 4.1 MiB/s, 66 objects/s recovering 2024-09-14T04:00:14.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 5317/247782 objects degraded (2.146%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:14.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 5317/247782 objects degraded (2.146%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 5317/247782 objects degraded (2.146%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:15.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:15 smithi114 ceph-mon[106969]: pgmap v410: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 958 op/s; 5317/236919 objects degraded (2.244%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:00:15.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:15 smithi114 ceph-mon[103029]: pgmap v410: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 958 op/s; 5317/236919 objects degraded (2.244%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:00:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:15 smithi160 ceph-mon[88407]: pgmap v410: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 958 op/s; 5317/236919 objects degraded (2.244%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:00:17.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:17 smithi160 ceph-mon[88407]: pgmap v411: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 708 op/s; 5317/236919 objects degraded (2.244%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:00:17.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:17 smithi160 ceph-mon[88407]: osdmap e143: 8 total, 8 up, 8 in 2024-09-14T04:00:18.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:17 smithi114 ceph-mon[103029]: pgmap v411: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 708 op/s; 5317/236919 objects degraded (2.244%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:00:18.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:17 smithi114 ceph-mon[103029]: osdmap e143: 8 total, 8 up, 8 in 2024-09-14T04:00:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:17 smithi114 ceph-mon[106969]: pgmap v411: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 708 op/s; 5317/236919 objects degraded (2.244%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:00:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:17 smithi114 ceph-mon[106969]: osdmap e143: 8 total, 8 up, 8 in 2024-09-14T04:00:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:18 smithi160 ceph-mon[88407]: osdmap e144: 8 total, 8 up, 8 in 2024-09-14T04:00:19.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:18 smithi114 ceph-mon[106969]: osdmap e144: 8 total, 8 up, 8 in 2024-09-14T04:00:19.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:18 smithi114 ceph-mon[103029]: osdmap e144: 8 total, 8 up, 8 in 2024-09-14T04:00:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:19 smithi160 ceph-mon[88407]: pgmap v414: 89 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 708 op/s; 5051/233271 objects degraded (2.165%); 4.3 MiB/s, 68 objects/s recovering 2024-09-14T04:00:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 5051/233271 objects degraded (2.165%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:20.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:19 smithi114 ceph-mon[103029]: pgmap v414: 89 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 708 op/s; 5051/233271 objects degraded (2.165%); 4.3 MiB/s, 68 objects/s recovering 2024-09-14T04:00:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 5051/233271 objects degraded (2.165%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:20.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:19 smithi114 ceph-mon[106969]: pgmap v414: 89 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 708 op/s; 5051/233271 objects degraded (2.165%); 4.3 MiB/s, 68 objects/s recovering 2024-09-14T04:00:20.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 5051/233271 objects degraded (2.165%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:00:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:00:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:00:21.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:21 smithi114 ceph-mon[103029]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 4870/225453 objects degraded (2.160%); 3.5 MiB/s, 55 objects/s recovering 2024-09-14T04:00:21.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:00:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:00:21] "GET /metrics HTTP/1.1" 200 36385 "" "Prometheus/2.51.0" 2024-09-14T04:00:21.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:21 smithi114 ceph-mon[106969]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 4870/225453 objects degraded (2.160%); 3.5 MiB/s, 55 objects/s recovering 2024-09-14T04:00:21.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:21 smithi160 ceph-mon[88407]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 4870/225453 objects degraded (2.160%); 3.5 MiB/s, 55 objects/s recovering 2024-09-14T04:00:23.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:24.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:23 smithi114 ceph-mon[106969]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 4870/223254 objects degraded (2.181%); 3.5 MiB/s, 55 objects/s recovering 2024-09-14T04:00:24.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:23 smithi114 ceph-mon[103029]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 4870/223254 objects degraded (2.181%); 3.5 MiB/s, 55 objects/s recovering 2024-09-14T04:00:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:23 smithi160 ceph-mon[88407]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 569 op/s; 4870/223254 objects degraded (2.181%); 3.5 MiB/s, 55 objects/s recovering 2024-09-14T04:00:25.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4870/223254 objects degraded (2.181%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:25.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4870/223254 objects degraded (2.181%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4870/223254 objects degraded (2.181%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:26.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:25 smithi114 ceph-mon[106969]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 4664/212166 objects degraded (2.198%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T04:00:26.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:25 smithi114 ceph-mon[103029]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 4664/212166 objects degraded (2.198%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T04:00:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:25 smithi160 ceph-mon[88407]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 4664/212166 objects degraded (2.198%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T04:00:28.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:27 smithi114 ceph-mon[106969]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 4664/212166 objects degraded (2.198%); 4.1 MiB/s, 65 objects/s recovering 2024-09-14T04:00:28.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:27 smithi114 ceph-mon[103029]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 4664/212166 objects degraded (2.198%); 4.1 MiB/s, 65 objects/s recovering 2024-09-14T04:00:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:27 smithi160 ceph-mon[88407]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 4664/212166 objects degraded (2.198%); 4.1 MiB/s, 65 objects/s recovering 2024-09-14T04:00:29.936 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:29 smithi114 ceph-mon[106969]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 4664/209694 objects degraded (2.224%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:00:29.936 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4664/209694 objects degraded (2.224%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:29.936 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:29 smithi114 ceph-mon[103029]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 4664/209694 objects degraded (2.224%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:00:29.937 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4664/209694 objects degraded (2.224%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:30.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:29 smithi160 ceph-mon[88407]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 4664/209694 objects degraded (2.224%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:00:30.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4664/209694 objects degraded (2.224%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:00:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:00:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:00:31.688 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:00:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:00:31] "GET /metrics HTTP/1.1" 200 36384 "" "Prometheus/2.51.0" 2024-09-14T04:00:32.085 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:31 smithi160 ceph-mon[88407]: pgmap v420: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 4454/198579 objects degraded (2.243%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T04:00:32.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:31 smithi114 ceph-mon[106969]: pgmap v420: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 4454/198579 objects degraded (2.243%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T04:00:32.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:31 smithi114 ceph-mon[103029]: pgmap v420: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 4454/198579 objects degraded (2.243%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T04:00:34.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:33 smithi114 ceph-mon[106969]: pgmap v421: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 4454/196119 objects degraded (2.271%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:00:34.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:33 smithi114 ceph-mon[103029]: pgmap v421: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 4454/196119 objects degraded (2.271%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:00:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:33 smithi160 ceph-mon[88407]: pgmap v421: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 4454/196119 objects degraded (2.271%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:00:35.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4454/196119 objects degraded (2.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:35.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4454/196119 objects degraded (2.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4454/196119 objects degraded (2.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:36.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:35 smithi114 ceph-mon[106969]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 4236/185679 objects degraded (2.281%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:00:36.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:35 smithi114 ceph-mon[103029]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 4236/185679 objects degraded (2.281%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:00:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:35 smithi160 ceph-mon[88407]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 4236/185679 objects degraded (2.281%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:00:38.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:37 smithi114 ceph-mon[103029]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 4236/185679 objects degraded (2.281%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:00:38.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:38.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:37 smithi114 ceph-mon[106969]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 4236/185679 objects degraded (2.281%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:00:38.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:37 smithi160 ceph-mon[88407]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 4236/185679 objects degraded (2.281%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:00:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:39 smithi114 ceph-mon[103029]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 4236/183165 objects degraded (2.313%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:00:40.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:39 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4236/183165 objects degraded (2.313%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:39 smithi114 ceph-mon[106969]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 4236/183165 objects degraded (2.313%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:00:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:39 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4236/183165 objects degraded (2.313%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:39 smithi160 ceph-mon[88407]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 805 op/s; 4236/183165 objects degraded (2.313%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:00:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:39 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4236/183165 objects degraded (2.313%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:00:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:00:40.216Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:00:41.720 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:00:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:00:41] "GET /metrics HTTP/1.1" 200 36380 "" "Prometheus/2.51.0" 2024-09-14T04:00:42.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:41 smithi114 ceph-mon[103029]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3999/172401 objects degraded (2.320%); 3.5 MiB/s, 55 objects/s recovering 2024-09-14T04:00:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:41 smithi114 ceph-mon[106969]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3999/172401 objects degraded (2.320%); 3.5 MiB/s, 55 objects/s recovering 2024-09-14T04:00:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:41 smithi160 ceph-mon[88407]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3999/172401 objects degraded (2.320%); 3.5 MiB/s, 55 objects/s recovering 2024-09-14T04:00:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:43 smithi160 ceph-mon[88407]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 3999/169851 objects degraded (2.354%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:00:44.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:43 smithi114 ceph-mon[103029]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 3999/169851 objects degraded (2.354%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:00:44.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:43 smithi114 ceph-mon[106969]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 797 op/s; 3999/169851 objects degraded (2.354%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:00:45.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:44 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3999/169851 objects degraded (2.354%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:45.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:44 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3999/169851 objects degraded (2.354%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:44 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3999/169851 objects degraded (2.354%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:46.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:45 smithi114 ceph-mon[103029]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3711/159144 objects degraded (2.332%); 3.9 MiB/s, 61 objects/s recovering 2024-09-14T04:00:46.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:45 smithi114 ceph-mon[106969]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3711/159144 objects degraded (2.332%); 3.9 MiB/s, 61 objects/s recovering 2024-09-14T04:00:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:45 smithi160 ceph-mon[88407]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 3711/159144 objects degraded (2.332%); 3.9 MiB/s, 61 objects/s recovering 2024-09-14T04:00:48.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:47 smithi114 ceph-mon[103029]: pgmap v428: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 3711/159144 objects degraded (2.332%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:00:48.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:47 smithi114 ceph-mon[106969]: pgmap v428: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 3711/159144 objects degraded (2.332%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:00:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:47 smithi160 ceph-mon[88407]: pgmap v428: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 3711/159144 objects degraded (2.332%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:00:50.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:49 smithi114 ceph-mon[103029]: pgmap v429: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 808 op/s; 3711/156558 objects degraded (2.370%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:00:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:49 smithi114 ceph-mon[103029]: osdmap e145: 8 total, 8 up, 8 in 2024-09-14T04:00:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:49 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3711/156558 objects degraded (2.370%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:49 smithi114 ceph-mon[106969]: pgmap v429: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 808 op/s; 3711/156558 objects degraded (2.370%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:00:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:49 smithi114 ceph-mon[106969]: osdmap e145: 8 total, 8 up, 8 in 2024-09-14T04:00:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:49 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3711/156558 objects degraded (2.370%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:49 smithi160 ceph-mon[88407]: pgmap v429: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 808 op/s; 3711/156558 objects degraded (2.370%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:00:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:49 smithi160 ceph-mon[88407]: osdmap e145: 8 total, 8 up, 8 in 2024-09-14T04:00:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:49 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3711/156558 objects degraded (2.370%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:00:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:00:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:00:51.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:50 smithi114 ceph-mon[103029]: osdmap e146: 8 total, 8 up, 8 in 2024-09-14T04:00:51.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:50 smithi114 ceph-mon[106969]: osdmap e146: 8 total, 8 up, 8 in 2024-09-14T04:00:51.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:50 smithi160 ceph-mon[88407]: osdmap e146: 8 total, 8 up, 8 in 2024-09-14T04:00:51.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:00:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:00:51] "GET /metrics HTTP/1.1" 200 36380 "" "Prometheus/2.51.0" 2024-09-14T04:00:52.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:51 smithi114 ceph-mon[103029]: pgmap v432: 89 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3448/145569 objects degraded (2.369%); 4.3 MiB/s, 68 objects/s recovering 2024-09-14T04:00:52.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:51 smithi114 ceph-mon[106969]: pgmap v432: 89 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3448/145569 objects degraded (2.369%); 4.3 MiB/s, 68 objects/s recovering 2024-09-14T04:00:52.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:51 smithi160 ceph-mon[88407]: pgmap v432: 89 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3448/145569 objects degraded (2.369%); 4.3 MiB/s, 68 objects/s recovering 2024-09-14T04:00:53.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:53.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:52 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:52 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:00:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:53 smithi114 ceph-mon[103029]: pgmap v433: 89 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 3448/143463 objects degraded (2.403%); 2.1 MiB/s, 32 objects/s recovering 2024-09-14T04:00:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:53 smithi114 ceph-mon[106969]: pgmap v433: 89 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 3448/143463 objects degraded (2.403%); 2.1 MiB/s, 32 objects/s recovering 2024-09-14T04:00:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:53 smithi160 ceph-mon[88407]: pgmap v433: 89 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 3448/143463 objects degraded (2.403%); 2.1 MiB/s, 32 objects/s recovering 2024-09-14T04:00:55.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:54 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3448/143463 objects degraded (2.403%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:55.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:54 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3448/143463 objects degraded (2.403%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:54 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3448/143463 objects degraded (2.403%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:00:56.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:55 smithi114 ceph-mon[103029]: pgmap v434: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3141/134169 objects degraded (2.341%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:00:56.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:55 smithi114 ceph-mon[106969]: pgmap v434: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3141/134169 objects degraded (2.341%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:00:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:55 smithi160 ceph-mon[88407]: pgmap v434: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3141/134169 objects degraded (2.341%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:00:58.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:57 smithi114 ceph-mon[103029]: pgmap v435: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 932 op/s; 3141/134169 objects degraded (2.341%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:00:58.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:57 smithi114 ceph-mon[106969]: pgmap v435: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 932 op/s; 3141/134169 objects degraded (2.341%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:00:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:57 smithi160 ceph-mon[88407]: pgmap v435: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 932 op/s; 3141/134169 objects degraded (2.341%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:01:00.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:59 smithi114 ceph-mon[103029]: pgmap v436: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 3141/131637 objects degraded (2.386%); 3.6 MiB/s, 58 objects/s recovering 2024-09-14T04:01:00.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:00:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3141/131637 objects degraded (2.386%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:00.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:59 smithi114 ceph-mon[106969]: pgmap v436: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 3141/131637 objects degraded (2.386%); 3.6 MiB/s, 58 objects/s recovering 2024-09-14T04:01:00.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:00:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3141/131637 objects degraded (2.386%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:00.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:59 smithi160 ceph-mon[88407]: pgmap v436: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 3141/131637 objects degraded (2.386%); 3.6 MiB/s, 58 objects/s recovering 2024-09-14T04:01:00.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:00:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3141/131637 objects degraded (2.386%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:01:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:01:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:01:01.821 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:01:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:01:01] "GET /metrics HTTP/1.1" 200 36379 "" "Prometheus/2.51.0" 2024-09-14T04:01:02.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:01 smithi114 ceph-mon[106969]: pgmap v437: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 2956/120543 objects degraded (2.452%); 2.8 MiB/s, 45 objects/s recovering 2024-09-14T04:01:02.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:01 smithi114 ceph-mon[103029]: pgmap v437: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 2956/120543 objects degraded (2.452%); 2.8 MiB/s, 45 objects/s recovering 2024-09-14T04:01:02.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:01 smithi160 ceph-mon[88407]: pgmap v437: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 2956/120543 objects degraded (2.452%); 2.8 MiB/s, 45 objects/s recovering 2024-09-14T04:01:04.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:03 smithi114 ceph-mon[106969]: pgmap v438: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 2956/117888 objects degraded (2.507%); 2.6 MiB/s, 40 objects/s recovering 2024-09-14T04:01:04.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:03 smithi114 ceph-mon[103029]: pgmap v438: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 2956/117888 objects degraded (2.507%); 2.6 MiB/s, 40 objects/s recovering 2024-09-14T04:01:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:03 smithi160 ceph-mon[88407]: pgmap v438: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 2956/117888 objects degraded (2.507%); 2.6 MiB/s, 40 objects/s recovering 2024-09-14T04:01:05.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2956/117888 objects degraded (2.507%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:05.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2956/117888 objects degraded (2.507%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2956/117888 objects degraded (2.507%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:05 smithi114 ceph-mon[103029]: pgmap v439: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2736/107121 objects degraded (2.554%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T04:01:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:05 smithi114 ceph-mon[106969]: pgmap v439: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2736/107121 objects degraded (2.554%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T04:01:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:05 smithi160 ceph-mon[88407]: pgmap v439: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2736/107121 objects degraded (2.554%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T04:01:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:07 smithi114 ceph-mon[103029]: pgmap v440: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 2736/107121 objects degraded (2.554%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:01:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:01:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:07 smithi114 ceph-mon[106969]: pgmap v440: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 2736/107121 objects degraded (2.554%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:01:08.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:08.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:01:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:07 smithi160 ceph-mon[88407]: pgmap v440: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 2736/107121 objects degraded (2.554%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:01:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:01:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:09 smithi114 ceph-mon[103029]: pgmap v441: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 817 op/s; 2736/104712 objects degraded (2.613%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:01:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2736/104712 objects degraded (2.613%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:09 smithi114 ceph-mon[106969]: pgmap v441: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 817 op/s; 2736/104712 objects degraded (2.613%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:01:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2736/104712 objects degraded (2.613%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:09 smithi160 ceph-mon[88407]: pgmap v441: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 817 op/s; 2736/104712 objects degraded (2.613%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:01:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2736/104712 objects degraded (2.613%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:01:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:01:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:01:11.811 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:01:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:01:11] "GET /metrics HTTP/1.1" 200 36379 "" "Prometheus/2.51.0" 2024-09-14T04:01:12.103 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:11 smithi114 ceph-mon[106969]: pgmap v442: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 2508/93597 objects degraded (2.680%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:01:12.103 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:11 smithi114 ceph-mon[103029]: pgmap v442: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 2508/93597 objects degraded (2.680%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:01:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:11 smithi160 ceph-mon[88407]: pgmap v442: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 2508/93597 objects degraded (2.680%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:01:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:13 smithi160 ceph-mon[88407]: pgmap v443: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 813 op/s; 2508/91239 objects degraded (2.749%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:01:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:13 smithi114 ceph-mon[106969]: pgmap v443: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 813 op/s; 2508/91239 objects degraded (2.749%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:01:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:13.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:13 smithi114 ceph-mon[103029]: pgmap v443: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 813 op/s; 2508/91239 objects degraded (2.749%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:01:14.570 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2508/91239 objects degraded (2.749%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:14.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2508/91239 objects degraded (2.749%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:14.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2508/91239 objects degraded (2.749%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[103029]: pgmap v444: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2206/81189 objects degraded (2.717%); 2.7 MiB/s, 62 objects/s recovering 2024-09-14T04:01:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:01:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:01:16.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[106969]: pgmap v444: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2206/81189 objects degraded (2.717%); 2.7 MiB/s, 62 objects/s recovering 2024-09-14T04:01:16.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:16.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:16.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:01:16.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:01:16.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:15 smithi160 ceph-mon[88407]: pgmap v444: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2206/81189 objects degraded (2.717%); 2.7 MiB/s, 62 objects/s recovering 2024-09-14T04:01:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:01:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:01:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:18.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:17 smithi114 ceph-mon[103029]: pgmap v445: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.7 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 2206/81189 objects degraded (2.717%); 1.5 MiB/s, 44 objects/s recovering 2024-09-14T04:01:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:17 smithi114 ceph-mon[106969]: pgmap v445: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.7 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 2206/81189 objects degraded (2.717%); 1.5 MiB/s, 44 objects/s recovering 2024-09-14T04:01:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:17 smithi160 ceph-mon[88407]: pgmap v445: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.7 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 2206/81189 objects degraded (2.717%); 1.5 MiB/s, 44 objects/s recovering 2024-09-14T04:01:20.215 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:19 smithi160 ceph-mon[88407]: pgmap v446: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 2206/78738 objects degraded (2.802%); 1.5 MiB/s, 44 objects/s recovering 2024-09-14T04:01:20.215 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2206/78738 objects degraded (2.802%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:20.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:19 smithi114 ceph-mon[103029]: pgmap v446: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 2206/78738 objects degraded (2.802%); 1.5 MiB/s, 44 objects/s recovering 2024-09-14T04:01:20.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2206/78738 objects degraded (2.802%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:19 smithi114 ceph-mon[106969]: pgmap v446: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 2206/78738 objects degraded (2.802%); 1.5 MiB/s, 44 objects/s recovering 2024-09-14T04:01:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2206/78738 objects degraded (2.802%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:01:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:01:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:01:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:01:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:01:21] "GET /metrics HTTP/1.1" 200 36379 "" "Prometheus/2.51.0" 2024-09-14T04:01:22.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:21 smithi114 ceph-mon[103029]: pgmap v447: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 1906/67686 objects degraded (2.816%); 1.5 MiB/s, 69 objects/s recovering 2024-09-14T04:01:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:21 smithi114 ceph-mon[106969]: pgmap v447: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 1906/67686 objects degraded (2.816%); 1.5 MiB/s, 69 objects/s recovering 2024-09-14T04:01:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:21 smithi160 ceph-mon[88407]: pgmap v447: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 1906/67686 objects degraded (2.816%); 1.5 MiB/s, 69 objects/s recovering 2024-09-14T04:01:23.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:24.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:23 smithi114 ceph-mon[103029]: pgmap v448: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 790 op/s; 1906/65133 objects degraded (2.926%); 331 KiB/s, 50 objects/s recovering 2024-09-14T04:01:24.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:23 smithi114 ceph-mon[103029]: osdmap e147: 8 total, 8 up, 8 in 2024-09-14T04:01:24.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:23 smithi114 ceph-mon[106969]: pgmap v448: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 790 op/s; 1906/65133 objects degraded (2.926%); 331 KiB/s, 50 objects/s recovering 2024-09-14T04:01:24.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:23 smithi114 ceph-mon[106969]: osdmap e147: 8 total, 8 up, 8 in 2024-09-14T04:01:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:23 smithi160 ceph-mon[88407]: pgmap v448: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 790 op/s; 1906/65133 objects degraded (2.926%); 331 KiB/s, 50 objects/s recovering 2024-09-14T04:01:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:23 smithi160 ceph-mon[88407]: osdmap e147: 8 total, 8 up, 8 in 2024-09-14T04:01:25.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:24 smithi114 ceph-mon[106969]: osdmap e148: 8 total, 8 up, 8 in 2024-09-14T04:01:25.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1906/65133 objects degraded (2.926%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:25.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:24 smithi114 ceph-mon[103029]: osdmap e148: 8 total, 8 up, 8 in 2024-09-14T04:01:25.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1906/65133 objects degraded (2.926%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:25.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:24 smithi160 ceph-mon[88407]: osdmap e148: 8 total, 8 up, 8 in 2024-09-14T04:01:25.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1906/65133 objects degraded (2.926%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:01:26.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:25 smithi114 ceph-mon[103029]: pgmap v451: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 1516/54969 objects degraded (2.758%); 0 B/s, 86 objects/s recovering 2024-09-14T04:01:26.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:25 smithi114 ceph-mon[106969]: pgmap v451: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 1516/54969 objects degraded (2.758%); 0 B/s, 86 objects/s recovering 2024-09-14T04:01:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:25 smithi160 ceph-mon[88407]: pgmap v451: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 1516/54969 objects degraded (2.758%); 0 B/s, 86 objects/s recovering 2024-09-14T04:01:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:27 smithi114 ceph-mon[106969]: pgmap v452: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 990 op/s; 1516/54969 objects degraded (2.758%); 0 B/s, 86 objects/s recovering 2024-09-14T04:01:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:27 smithi114 ceph-mon[103029]: pgmap v452: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 990 op/s; 1516/54969 objects degraded (2.758%); 0 B/s, 86 objects/s recovering 2024-09-14T04:01:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:27 smithi160 ceph-mon[88407]: pgmap v452: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 990 op/s; 1516/54969 objects degraded (2.758%); 0 B/s, 86 objects/s recovering 2024-09-14T04:01:30.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:29 smithi160 ceph-mon[88407]: pgmap v453: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 1516/52698 objects degraded (2.877%); 0 B/s, 48 objects/s recovering 2024-09-14T04:01:30.215 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1516/52698 objects degraded (2.877%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:29 smithi114 ceph-mon[103029]: pgmap v453: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 1516/52698 objects degraded (2.877%); 0 B/s, 48 objects/s recovering 2024-09-14T04:01:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1516/52698 objects degraded (2.877%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:29 smithi114 ceph-mon[106969]: pgmap v453: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 1516/52698 objects degraded (2.877%); 0 B/s, 48 objects/s recovering 2024-09-14T04:01:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1516/52698 objects degraded (2.877%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:30.596 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:01:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:01:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:01:31.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:01:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:01:31] "GET /metrics HTTP/1.1" 200 36377 "" "Prometheus/2.51.0" 2024-09-14T04:01:32.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:31 smithi114 ceph-mon[106969]: pgmap v454: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 880 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 950 op/s; 1211/42306 objects degraded (2.862%); 0 B/s, 86 objects/s recovering 2024-09-14T04:01:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:31 smithi114 ceph-mon[103029]: pgmap v454: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 880 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 950 op/s; 1211/42306 objects degraded (2.862%); 0 B/s, 86 objects/s recovering 2024-09-14T04:01:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:31 smithi160 ceph-mon[88407]: pgmap v454: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 880 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 950 op/s; 1211/42306 objects degraded (2.862%); 0 B/s, 86 objects/s recovering 2024-09-14T04:01:34.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:33 smithi114 ceph-mon[106969]: pgmap v455: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 829 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 1211/39858 objects degraded (3.038%); 0 B/s, 72 objects/s recovering 2024-09-14T04:01:34.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:33 smithi114 ceph-mon[103029]: pgmap v455: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 829 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 1211/39858 objects degraded (3.038%); 0 B/s, 72 objects/s recovering 2024-09-14T04:01:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:33 smithi160 ceph-mon[88407]: pgmap v455: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 829 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 1211/39858 objects degraded (3.038%); 0 B/s, 72 objects/s recovering 2024-09-14T04:01:35.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1211/39858 objects degraded (3.038%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1211/39858 objects degraded (3.038%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1211/39858 objects degraded (3.038%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:35 smithi114 ceph-mon[103029]: pgmap v456: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 606 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 809 op/s; 982/29148 objects degraded (3.369%); 0 B/s, 50 objects/s recovering 2024-09-14T04:01:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:35 smithi114 ceph-mon[106969]: pgmap v456: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 606 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 809 op/s; 982/29148 objects degraded (3.369%); 0 B/s, 50 objects/s recovering 2024-09-14T04:01:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:35 smithi160 ceph-mon[88407]: pgmap v456: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 606 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 809 op/s; 982/29148 objects degraded (3.369%); 0 B/s, 50 objects/s recovering 2024-09-14T04:01:38.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:38 smithi114 ceph-mon[103029]: pgmap v457: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 606 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 982/29148 objects degraded (3.369%); 0 B/s, 44 objects/s recovering 2024-09-14T04:01:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:38 smithi114 ceph-mon[106969]: pgmap v457: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 606 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 982/29148 objects degraded (3.369%); 0 B/s, 44 objects/s recovering 2024-09-14T04:01:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:38 smithi160 ceph-mon[88407]: pgmap v457: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 606 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 982/29148 objects degraded (3.369%); 0 B/s, 44 objects/s recovering 2024-09-14T04:01:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:40 smithi114 ceph-mon[103029]: pgmap v458: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 553 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 982/26604 objects degraded (3.691%); 0 B/s, 44 objects/s recovering 2024-09-14T04:01:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:40 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 982/26604 objects degraded (3.691%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:40 smithi114 ceph-mon[106969]: pgmap v458: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 553 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 982/26604 objects degraded (3.691%); 0 B/s, 44 objects/s recovering 2024-09-14T04:01:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:40 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 982/26604 objects degraded (3.691%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:01:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:01:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:01:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:40 smithi160 ceph-mon[88407]: pgmap v458: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 553 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 982/26604 objects degraded (3.691%); 0 B/s, 44 objects/s recovering 2024-09-14T04:01:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:40 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 982/26604 objects degraded (3.691%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:41.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:01:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:01:41] "GET /metrics HTTP/1.1" 200 36361 "" "Prometheus/2.51.0" 2024-09-14T04:01:42.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:42 smithi114 ceph-mon[103029]: pgmap v459: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 321 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 734/15441 objects degraded (4.754%); 0 B/s, 65 objects/s recovering 2024-09-14T04:01:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:42 smithi114 ceph-mon[106969]: pgmap v459: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 321 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 734/15441 objects degraded (4.754%); 0 B/s, 65 objects/s recovering 2024-09-14T04:01:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:42 smithi160 ceph-mon[88407]: pgmap v459: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 321 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 734/15441 objects degraded (4.754%); 0 B/s, 65 objects/s recovering 2024-09-14T04:01:44.137 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 125969 objects 2024-09-14T04:01:44.137 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :145.316 2024-09-14T04:01:44.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:44 smithi114 ceph-mon[103029]: pgmap v460: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 266 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 734/12825 objects degraded (5.723%); 0 B/s, 39 objects/s recovering 2024-09-14T04:01:44.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:44 smithi114 ceph-mon[106969]: pgmap v460: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 266 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 734/12825 objects degraded (5.723%); 0 B/s, 39 objects/s recovering 2024-09-14T04:01:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:44 smithi160 ceph-mon[88407]: pgmap v460: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 266 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 734/12825 objects degraded (5.723%); 0 B/s, 39 objects/s recovering 2024-09-14T04:01:45.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:45 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 734/12825 objects degraded (5.723%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:45.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:45 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 734/12825 objects degraded (5.723%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:45.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:45 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 734/12825 objects degraded (5.723%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:46.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:46 smithi114 ceph-mon[103029]: pgmap v461: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 36 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 435/1803 objects degraded (24.126%); 0 B/s, 64 objects/s recovering 2024-09-14T04:01:46.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:46 smithi114 ceph-mon[106969]: pgmap v461: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 36 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 435/1803 objects degraded (24.126%); 0 B/s, 64 objects/s recovering 2024-09-14T04:01:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:46 smithi160 ceph-mon[88407]: pgmap v461: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 36 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 435/1803 objects degraded (24.126%); 0 B/s, 64 objects/s recovering 2024-09-14T04:01:46.702 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-14T04:01:46.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-14T04:01:46.961 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:47.332 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:47 smithi114 ceph-mon[103029]: osdmap e149: 8 total, 8 up, 8 in 2024-09-14T04:01:47.333 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:47 smithi114 ceph-mon[106969]: osdmap e149: 8 total, 8 up, 8 in 2024-09-14T04:01:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:47 smithi160 ceph-mon[88407]: osdmap e149: 8 total, 8 up, 8 in 2024-09-14T04:01:48.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:48 smithi114 ceph-mon[106969]: pgmap v463: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 36 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 911 op/s; 435/1803 objects degraded (24.126%); 0 B/s, 54 objects/s recovering 2024-09-14T04:01:48.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:48 smithi114 ceph-mon[106969]: osdmap e150: 8 total, 8 up, 8 in 2024-09-14T04:01:48.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:48 smithi114 ceph-mon[103029]: pgmap v463: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 36 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 911 op/s; 435/1803 objects degraded (24.126%); 0 B/s, 54 objects/s recovering 2024-09-14T04:01:48.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:48 smithi114 ceph-mon[103029]: osdmap e150: 8 total, 8 up, 8 in 2024-09-14T04:01:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:48 smithi160 ceph-mon[88407]: pgmap v463: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 36 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 911 op/s; 435/1803 objects degraded (24.126%); 0 B/s, 54 objects/s recovering 2024-09-14T04:01:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:48 smithi160 ceph-mon[88407]: osdmap e150: 8 total, 8 up, 8 in 2024-09-14T04:01:49.044 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_3' removed 2024-09-14T04:01:49.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:49 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1742745360' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:01:49.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:49 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:01:49.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:49 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1742745360' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:01:49.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:49 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:01:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:49 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1742745360' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:01:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:49 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:01:49.769 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-14T04:01:49.769 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-14T04:01:49.770 DEBUG:teuthology.parallel:result is None 2024-09-14T04:01:49.770 INFO:teuthology.run_tasks:Running task print... 2024-09-14T04:01:49.780 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-09-14T04:01:49.780 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-14T04:01:49.816 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-14T04:01:49.816 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-14T04:01:49.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-14T04:01:50.059 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:50 smithi114 ceph-mon[103029]: pgmap v465: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 18 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 604 op/s; 435/933 objects degraded (46.624%); 0 B/s, 37 objects/s recovering 2024-09-14T04:01:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:50 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:01:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:50 smithi114 ceph-mon[103029]: osdmap e151: 8 total, 8 up, 8 in 2024-09-14T04:01:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:50 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 435/933 objects degraded (46.624%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:50.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:50 smithi114 ceph-mon[106969]: pgmap v465: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 18 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 604 op/s; 435/933 objects degraded (46.624%); 0 B/s, 37 objects/s recovering 2024-09-14T04:01:50.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:50 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:01:50.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:50 smithi114 ceph-mon[106969]: osdmap e151: 8 total, 8 up, 8 in 2024-09-14T04:01:50.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:50 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 435/933 objects degraded (46.624%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:01:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:01:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:01:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:50 smithi160 ceph-mon[88407]: pgmap v465: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 18 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 604 op/s; 435/933 objects degraded (46.624%); 0 B/s, 37 objects/s recovering 2024-09-14T04:01:50.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:50 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:01:50.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:50 smithi160 ceph-mon[88407]: osdmap e151: 8 total, 8 up, 8 in 2024-09-14T04:01:50.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:50 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 435/933 objects degraded (46.624%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:01:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:51 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 435/933 objects degraded (46.624%), 1 pg degraded, 1 pg undersized) 2024-09-14T04:01:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:51 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:01:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:51 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 435/933 objects degraded (46.624%), 1 pg degraded, 1 pg undersized) 2024-09-14T04:01:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:51 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:01:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:51 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 435/933 objects degraded (46.624%), 1 pg degraded, 1 pg undersized) 2024-09-14T04:01:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:51 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:01:51.844 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:01:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:01:51] "GET /metrics HTTP/1.1" 200 36361 "" "Prometheus/2.51.0" 2024-09-14T04:01:52.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:52 smithi114 ceph-mon[103029]: pgmap v467: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:52.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:52 smithi114 ceph-mon[106969]: pgmap v467: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:52 smithi160 ceph-mon[88407]: pgmap v467: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:52.611 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:01:53.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2017620112' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-14T04:01:53.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:53.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2017620112' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-14T04:01:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2017620112' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-14T04:01:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:01:53.411 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":40,"flags":0,"active_gid":34107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2164484837},{"type":"v1","addr":"172.21.15.114:6801","nonce":2164484837}]},"active_addr":"172.21.15.114:6801/2164484837","active_change":"2024-09-14T03:48:52.256267+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":25465,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.114:8443/","prometheus":"http://172.21.15.114:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":76,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2913807509}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2302385312}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4020924495}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":815080595}]}]} 2024-09-14T04:01:53.415 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-14T04:01:53.416 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-14T04:01:53.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-14T04:01:53.660 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:54.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:54 smithi114 ceph-mon[103029]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:54 smithi114 ceph-mon[106969]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:54 smithi160 ceph-mon[88407]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:55.585 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:01:55.585 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":151,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","created":"2024-09-14T03:36:30.445371+0000","modified":"2024-09-14T04:01:49.034415+0000","last_up_change":"2024-09-14T03:58:44.764453+0000","last_in_change":"2024-09-14T03:41:06.908282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T03:39:29.348427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T03:41:54.688727+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-14T03:43:09.784984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-14T03:43:10.496413+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0a021a3a-45d2-4825-a701-d7c3793ef6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":76,"up_from":80,"up_thru":143,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6803","nonce":2022879658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6805","nonce":2022879658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6809","nonce":2022879658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6807","nonce":2022879658}]},"public_addr":"172.21.15.114:6803/2022879658","cluster_addr":"172.21.15.114:6805/2022879658","heartbeat_back_addr":"172.21.15.114:6809/2022879658","heartbeat_front_addr":"172.21.15.114:6807/2022879658","state":["exists","up"]},{"osd":1,"uuid":"076291c4-5927-456d-bf74-2ab79be4c756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":93,"up_from":97,"up_thru":149,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6811","nonce":3980338706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6813","nonce":3980338706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6817","nonce":3980338706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6815","nonce":3980338706}]},"public_addr":"172.21.15.114:6811/3980338706","cluster_addr":"172.21.15.114:6813/3980338706","heartbeat_back_addr":"172.21.15.114:6817/3980338706","heartbeat_front_addr":"172.21.15.114:6815/3980338706","state":["exists","up"]},{"osd":2,"uuid":"d6787400-73ea-4edf-bb17-440ac36bd4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":103,"up_from":109,"up_thru":136,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6819","nonce":3555775440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6821","nonce":3555775440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6825","nonce":3555775440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6823","nonce":3555775440}]},"public_addr":"172.21.15.114:6819/3555775440","cluster_addr":"172.21.15.114:6821/3555775440","heartbeat_back_addr":"172.21.15.114:6825/3555775440","heartbeat_front_addr":"172.21.15.114:6823/3555775440","state":["exists","up"]},{"osd":3,"uuid":"ead755f3-173a-4465-a28f-af7288b0a42b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":132,"up_from":136,"up_thru":147,"down_at":133,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6827","nonce":288809847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6829","nonce":288809847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6833","nonce":288809847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6831","nonce":288809847}]},"public_addr":"172.21.15.114:6827/288809847","cluster_addr":"172.21.15.114:6829/288809847","heartbeat_back_addr":"172.21.15.114:6833/288809847","heartbeat_front_addr":"172.21.15.114:6831/288809847","state":["exists","up"]},{"osd":4,"uuid":"60a0aac2-e235-484f-b25f-f112c83da7f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6801","nonce":825361940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6803","nonce":825361940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6807","nonce":825361940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6805","nonce":825361940}]},"public_addr":"172.21.15.160:6801/825361940","cluster_addr":"172.21.15.160:6803/825361940","heartbeat_back_addr":"172.21.15.160:6807/825361940","heartbeat_front_addr":"172.21.15.160:6805/825361940","state":["exists","up"]},{"osd":5,"uuid":"8f49c7de-c810-4075-b731-022bb5031d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6809","nonce":3752160003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6811","nonce":3752160003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6815","nonce":3752160003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6813","nonce":3752160003}]},"public_addr":"172.21.15.160:6809/3752160003","cluster_addr":"172.21.15.160:6811/3752160003","heartbeat_back_addr":"172.21.15.160:6815/3752160003","heartbeat_front_addr":"172.21.15.160:6813/3752160003","state":["exists","up"]},{"osd":6,"uuid":"455de335-7290-4242-b743-6b0c446683ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716932982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716932982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716932982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716932982}]},"public_addr":"172.21.15.160:6817/3716932982","cluster_addr":"172.21.15.160:6819/3716932982","heartbeat_back_addr":"172.21.15.160:6823/3716932982","heartbeat_front_addr":"172.21.15.160:6821/3716932982","state":["exists","up"]},{"osd":7,"uuid":"7d8b5297-b1db-41b1-be35-073e7780128e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6825","nonce":1323253841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6827","nonce":1323253841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6831","nonce":1323253841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6829","nonce":1323253841}]},"public_addr":"172.21.15.160:6825/1323253841","cluster_addr":"172.21.15.160:6827/1323253841","heartbeat_back_addr":"172.21.15.160:6831/1323253841","heartbeat_front_addr":"172.21.15.160:6829/1323253841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:38:38.363862+0000","dead_epoch":76},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:02.363463+0000","dead_epoch":93},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:26.353194+0000","dead_epoch":103},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:50.132880+0000","dead_epoch":132},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:11.402862+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:34.539123+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:58.191287+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:41:22.159510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.160:6835/3913895673":"2024-09-15T03:47:52.734325+0000","172.21.15.114:6801/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:6800/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1089490408":"2024-09-15T03:36:58.525006+0000","172.21.15.160:0/1361660388":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/767494479":"2024-09-15T03:41:51.162793+0000","172.21.15.160:0/1393128007":"2024-09-15T03:47:52.734325+0000","172.21.15.160:0/710559079":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/2155247703":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/1727106713":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/4089637122":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/2536880143":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/3344792350":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6800/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/426398557":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/1634520325":"2024-09-15T03:48:52.256052+0000","172.21.15.160:6834/3913895673":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/2694023561":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/210127168":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/2067882043":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/3853151361":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/1570092639":"2024-09-15T03:48:52.256052+0000","172.21.15.114:6801/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.160:0/1259038325":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/1481425303":"2024-09-15T03:48:52.256052+0000","172.21.15.114:6800/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6801/1727106713":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/1784685209":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6801/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2995702914":"2024-09-15T03:46:19.354331+0000","172.21.15.114:6800/3774959769":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/386045858":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/546762471":"2024-09-15T03:46:19.354331+0000","172.21.15.114:6801/3774959769":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/1795949490":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/2095546075":"2024-09-15T03:37:21.054968+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T04:01:56.233 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:56 smithi114 ceph-mon[106969]: pgmap v469: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:56.233 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:56 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4262566183' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:01:56.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:56 smithi114 ceph-mon[103029]: pgmap v469: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:56.233 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:56 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4262566183' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:01:56.257 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-14T04:01:56.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-14T04:01:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:56 smithi160 ceph-mon[88407]: pgmap v469: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:56 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4262566183' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:01:56.501 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:01:58.335 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:58 smithi114 ceph-mon[106969]: pgmap v470: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:58.335 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:58 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:58.336 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":151,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","created":"2024-09-14T03:36:30.445371+0000","modified":"2024-09-14T04:01:49.034415+0000","last_up_change":"2024-09-14T03:58:44.764453+0000","last_in_change":"2024-09-14T03:41:06.908282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T03:39:29.348427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T03:41:54.688727+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-14T03:43:09.784984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-14T03:43:10.496413+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0a021a3a-45d2-4825-a701-d7c3793ef6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":76,"up_from":80,"up_thru":143,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6803","nonce":2022879658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6805","nonce":2022879658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6809","nonce":2022879658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6807","nonce":2022879658}]},"public_addr":"172.21.15.114:6803/2022879658","cluster_addr":"172.21.15.114:6805/2022879658","heartbeat_back_addr":"172.21.15.114:6809/2022879658","heartbeat_front_addr":"172.21.15.114:6807/2022879658","state":["exists","up"]},{"osd":1,"uuid":"076291c4-5927-456d-bf74-2ab79be4c756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":93,"up_from":97,"up_thru":149,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6811","nonce":3980338706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6813","nonce":3980338706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6817","nonce":3980338706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6815","nonce":3980338706}]},"public_addr":"172.21.15.114:6811/3980338706","cluster_addr":"172.21.15.114:6813/3980338706","heartbeat_back_addr":"172.21.15.114:6817/3980338706","heartbeat_front_addr":"172.21.15.114:6815/3980338706","state":["exists","up"]},{"osd":2,"uuid":"d6787400-73ea-4edf-bb17-440ac36bd4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":103,"up_from":109,"up_thru":136,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6819","nonce":3555775440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6821","nonce":3555775440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6825","nonce":3555775440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6823","nonce":3555775440}]},"public_addr":"172.21.15.114:6819/3555775440","cluster_addr":"172.21.15.114:6821/3555775440","heartbeat_back_addr":"172.21.15.114:6825/3555775440","heartbeat_front_addr":"172.21.15.114:6823/3555775440","state":["exists","up"]},{"osd":3,"uuid":"ead755f3-173a-4465-a28f-af7288b0a42b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":132,"up_from":136,"up_thru":147,"down_at":133,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6827","nonce":288809847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6829","nonce":288809847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6833","nonce":288809847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6831","nonce":288809847}]},"public_addr":"172.21.15.114:6827/288809847","cluster_addr":"172.21.15.114:6829/288809847","heartbeat_back_addr":"172.21.15.114:6833/288809847","heartbeat_front_addr":"172.21.15.114:6831/288809847","state":["exists","up"]},{"osd":4,"uuid":"60a0aac2-e235-484f-b25f-f112c83da7f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6801","nonce":825361940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6803","nonce":825361940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6807","nonce":825361940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":825361940},{"type":"v1","addr":"172.21.15.160:6805","nonce":825361940}]},"public_addr":"172.21.15.160:6801/825361940","cluster_addr":"172.21.15.160:6803/825361940","heartbeat_back_addr":"172.21.15.160:6807/825361940","heartbeat_front_addr":"172.21.15.160:6805/825361940","state":["exists","up"]},{"osd":5,"uuid":"8f49c7de-c810-4075-b731-022bb5031d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6809","nonce":3752160003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6811","nonce":3752160003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6815","nonce":3752160003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6813","nonce":3752160003}]},"public_addr":"172.21.15.160:6809/3752160003","cluster_addr":"172.21.15.160:6811/3752160003","heartbeat_back_addr":"172.21.15.160:6815/3752160003","heartbeat_front_addr":"172.21.15.160:6813/3752160003","state":["exists","up"]},{"osd":6,"uuid":"455de335-7290-4242-b743-6b0c446683ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716932982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716932982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716932982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716932982}]},"public_addr":"172.21.15.160:6817/3716932982","cluster_addr":"172.21.15.160:6819/3716932982","heartbeat_back_addr":"172.21.15.160:6823/3716932982","heartbeat_front_addr":"172.21.15.160:6821/3716932982","state":["exists","up"]},{"osd":7,"uuid":"7d8b5297-b1db-41b1-be35-073e7780128e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6825","nonce":1323253841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6827","nonce":1323253841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6831","nonce":1323253841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6829","nonce":1323253841}]},"public_addr":"172.21.15.160:6825/1323253841","cluster_addr":"172.21.15.160:6827/1323253841","heartbeat_back_addr":"172.21.15.160:6831/1323253841","heartbeat_front_addr":"172.21.15.160:6829/1323253841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:38:38.363862+0000","dead_epoch":76},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:02.363463+0000","dead_epoch":93},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:26.353194+0000","dead_epoch":103},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:50.132880+0000","dead_epoch":132},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:11.402862+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:34.539123+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:58.191287+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:41:22.159510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.160:6835/3913895673":"2024-09-15T03:47:52.734325+0000","172.21.15.114:6801/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:6800/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1089490408":"2024-09-15T03:36:58.525006+0000","172.21.15.160:0/1361660388":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/767494479":"2024-09-15T03:41:51.162793+0000","172.21.15.160:0/1393128007":"2024-09-15T03:47:52.734325+0000","172.21.15.160:0/710559079":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/2155247703":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/1727106713":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/4089637122":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/2536880143":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/3344792350":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6800/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/426398557":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/1634520325":"2024-09-15T03:48:52.256052+0000","172.21.15.160:6834/3913895673":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/2694023561":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/210127168":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/2067882043":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/3853151361":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/1570092639":"2024-09-15T03:48:52.256052+0000","172.21.15.114:6801/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.160:0/1259038325":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/1481425303":"2024-09-15T03:48:52.256052+0000","172.21.15.114:6800/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6801/1727106713":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/1784685209":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6801/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2995702914":"2024-09-15T03:46:19.354331+0000","172.21.15.114:6800/3774959769":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/386045858":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/546762471":"2024-09-15T03:46:19.354331+0000","172.21.15.114:6801/3774959769":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/1795949490":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/2095546075":"2024-09-15T03:37:21.054968+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T04:01:58.336 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:58 smithi114 ceph-mon[103029]: pgmap v470: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:58.336 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:58 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:58 smithi160 ceph-mon[88407]: pgmap v470: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:01:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:58 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:01:59.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-14T04:01:59.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-14T04:01:59.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-14T04:01:59.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-14T04:01:59.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-14T04:01:59.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-14T04:01:59.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-14T04:01:59.070 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-14T04:01:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:01:59 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2964032564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:01:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:01:59 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2964032564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:01:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:01:59 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2964032564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:01:59.522 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:59.787 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:59.796 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:59.798 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:59.799 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:59.800 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:59.802 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:01:59.804 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:00.186 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:00 smithi114 ceph-mon[103029]: pgmap v471: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:00.187 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:00 smithi114 ceph-mon[106969]: pgmap v471: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:00 smithi160 ceph-mon[88407]: pgmap v471: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:02:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:02:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:02:01.774 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:02:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:02:01] "GET /metrics HTTP/1.1" 200 34002 "" "Prometheus/2.51.0" 2024-09-14T04:02:02.357 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:02 smithi114 ceph-mon[106969]: pgmap v472: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:02.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:02 smithi114 ceph-mon[103029]: pgmap v472: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:02 smithi160 ceph-mon[88407]: pgmap v472: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:04.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:04 smithi160 ceph-mon[88407]: pgmap v473: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:04.444 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:04 smithi114 ceph-mon[106969]: pgmap v473: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:04.444 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:04 smithi114 ceph-mon[103029]: pgmap v473: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:06.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:06 smithi114 ceph-mon[106969]: pgmap v474: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:06.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:06 smithi114 ceph-mon[103029]: pgmap v474: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:06.404 INFO:teuthology.orchestra.run.smithi114.stdout:206158430460 2024-09-14T04:02:06.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-14T04:02:06.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:06 smithi160 ceph-mon[88407]: pgmap v474: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:08.352 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:08 smithi114 ceph-mon[106969]: pgmap v475: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:08.352 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:08.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:08 smithi114 ceph-mon[103029]: pgmap v475: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:08.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:08 smithi160 ceph-mon[88407]: pgmap v475: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:09.780 INFO:teuthology.orchestra.run.smithi114.stdout:416611827810 2024-09-14T04:02:09.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-14T04:02:10.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:10 smithi114 ceph-mon[103029]: pgmap v476: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:10 smithi114 ceph-mon[106969]: pgmap v476: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:10.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:10 smithi160 ceph-mon[88407]: pgmap v476: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:10.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:02:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:02:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:02:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:02:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:02:11] "GET /metrics HTTP/1.1" 200 34000 "" "Prometheus/2.51.0" 2024-09-14T04:02:12.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:12 smithi114 ceph-mon[103029]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:12.346 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:12 smithi114 ceph-mon[106969]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:12 smithi160 ceph-mon[88407]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:13.216 INFO:teuthology.orchestra.run.smithi114.stdout:343597383798 2024-09-14T04:02:13.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-14T04:02:13.607 INFO:teuthology.orchestra.run.smithi114.stdout:128849019146 2024-09-14T04:02:13.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-14T04:02:14.082 INFO:teuthology.orchestra.run.smithi114.stdout:180388626690 2024-09-14T04:02:14.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-14T04:02:14.399 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:14 smithi114 ceph-mon[106969]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:14.399 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:14 smithi114 ceph-mon[103029]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:14 smithi160 ceph-mon[88407]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:15.368 INFO:teuthology.orchestra.run.smithi114.stdout:584115552300 2024-09-14T04:02:15.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-14T04:02:15.512 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:16.234 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:16 smithi114 ceph-mon[106969]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:16.235 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:16 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:02:16.235 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:16 smithi114 ceph-mon[103029]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:16.235 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:16 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:02:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:16 smithi160 ceph-mon[88407]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:16 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:02:16.912 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:16.913 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:16.940 INFO:teuthology.orchestra.run.smithi114.stdout:468151435341 2024-09-14T04:02:16.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-14T04:02:17.990 INFO:teuthology.orchestra.run.smithi114.stdout:154618822918 2024-09-14T04:02:17.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-14T04:02:18.194 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:18 smithi160 ceph-mon[88407]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:18.271 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:18 smithi114 ceph-mon[106969]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:18.271 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:18 smithi114 ceph-mon[103029]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:18.764 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:20.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:20 smithi114 ceph-mon[103029]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:20.352 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:20 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:20.352 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:20 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:20.352 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:20 smithi114 ceph-mon[106969]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:20.352 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:20 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:20.352 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:20 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:20.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:02:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:02:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:02:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:20 smithi160 ceph-mon[88407]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:20 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:20 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:20.451 INFO:teuthology.orchestra.run.smithi114.stdout:206158430462 2024-09-14T04:02:21.170 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430460 got 206158430462 for osd.7 2024-09-14T04:02:21.171 DEBUG:teuthology.parallel:result is None 2024-09-14T04:02:21.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3148908029' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-14T04:02:21.400 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3148908029' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-14T04:02:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3148908029' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-14T04:02:21.711 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:02:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:02:21] "GET /metrics HTTP/1.1" 200 34000 "" "Prometheus/2.51.0" 2024-09-14T04:02:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:22 smithi160 ceph-mon[88407]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:22.484 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:22 smithi114 ceph-mon[106969]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:22.485 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:22 smithi114 ceph-mon[103029]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:22.538 INFO:teuthology.orchestra.run.smithi114.stdout:180388626692 2024-09-14T04:02:23.488 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:23.489 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:23.489 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:23.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:23 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3502005432' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-14T04:02:23.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:23.584 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3502005432' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-14T04:02:23.584 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:23 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3502005432' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-14T04:02:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:24.086 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626690 got 180388626692 for osd.6 2024-09-14T04:02:24.087 DEBUG:teuthology.parallel:result is None 2024-09-14T04:02:24.325 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:24 smithi114 ceph-mon[103029]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:24.326 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:24 smithi114 ceph-mon[106969]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:24 smithi160 ceph-mon[88407]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:24.954 INFO:teuthology.orchestra.run.smithi114.stdout:343597383801 2024-09-14T04:02:25.144 INFO:teuthology.orchestra.run.smithi114.stdout:416611827812 2024-09-14T04:02:25.470 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1112521191' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-14T04:02:25.470 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/570852149' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-14T04:02:25.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1112521191' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-14T04:02:25.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/570852149' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-14T04:02:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1112521191' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-14T04:02:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/570852149' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-14T04:02:25.797 INFO:tasks.cephadm.ceph_manager.ceph:need seq 416611827810 got 416611827812 for osd.1 2024-09-14T04:02:25.797 DEBUG:teuthology.parallel:result is None 2024-09-14T04:02:26.341 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:26 smithi114 ceph-mon[103029]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:26.342 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:26 smithi114 ceph-mon[106969]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:26 smithi160 ceph-mon[88407]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:27.158 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:02:26 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=sqlstore.transactions t=2024-09-14T04:02:26.719505751Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-14T04:02:28.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:28 smithi114 ceph-mon[103029]: pgmap v485: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:28.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:28 smithi114 ceph-mon[106969]: pgmap v485: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:28 smithi160 ceph-mon[88407]: pgmap v485: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:30.317 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:30 smithi114 ceph-mon[103029]: pgmap v486: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:30.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:30 smithi114 ceph-mon[106969]: pgmap v486: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:30.460 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:02:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:02:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:02:30.461 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:30 smithi160 ceph-mon[88407]: pgmap v486: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:31.308 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383798 got 343597383801 for osd.0 2024-09-14T04:02:31.308 DEBUG:teuthology.parallel:result is None 2024-09-14T04:02:31.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:31 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/997534083' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-14T04:02:31.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:31 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/997534083' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-14T04:02:31.481 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:31 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/997534083' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-14T04:02:31.586 INFO:teuthology.orchestra.run.smithi114.stdout:154618822921 2024-09-14T04:02:31.711 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:02:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:02:31] "GET /metrics HTTP/1.1" 200 34003 "" "Prometheus/2.51.0" 2024-09-14T04:02:32.143 INFO:teuthology.orchestra.run.smithi114.stdout:468151435344 2024-09-14T04:02:32.357 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:32 smithi114 ceph-mon[106969]: pgmap v487: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:32.357 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:32 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/778731450' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-14T04:02:32.358 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:32 smithi114 ceph-mon[103029]: pgmap v487: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:32.358 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:32 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/778731450' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-14T04:02:32.382 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822918 got 154618822921 for osd.5 2024-09-14T04:02:32.382 DEBUG:teuthology.parallel:result is None 2024-09-14T04:02:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:32 smithi160 ceph-mon[88407]: pgmap v487: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:32 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/778731450' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-14T04:02:32.725 INFO:teuthology.orchestra.run.smithi114.stdout:128849019150 2024-09-14T04:02:33.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:33 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3514540566' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-14T04:02:33.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:33 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3514540566' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-14T04:02:33.413 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019146 got 128849019150 for osd.4 2024-09-14T04:02:33.413 DEBUG:teuthology.parallel:result is None 2024-09-14T04:02:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:33 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3514540566' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-14T04:02:34.070 INFO:tasks.cephadm.ceph_manager.ceph:need seq 468151435341 got 468151435344 for osd.2 2024-09-14T04:02:34.070 DEBUG:teuthology.parallel:result is None 2024-09-14T04:02:34.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:34 smithi114 ceph-mon[103029]: pgmap v488: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:34.330 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:34 smithi114 ceph-mon[106969]: pgmap v488: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:34 smithi160 ceph-mon[88407]: pgmap v488: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:34.816 INFO:teuthology.orchestra.run.smithi114.stdout:584115552303 2024-09-14T04:02:35.478 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:35 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4128323565' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-14T04:02:35.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:35 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4128323565' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-14T04:02:35.500 INFO:tasks.cephadm.ceph_manager.ceph:need seq 584115552300 got 584115552303 for osd.3 2024-09-14T04:02:35.500 DEBUG:teuthology.parallel:result is None 2024-09-14T04:02:35.501 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-14T04:02:35.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:02:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:35 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4128323565' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-14T04:02:36.288 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:36.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:36 smithi114 ceph-mon[103029]: pgmap v489: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:36.540 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:36 smithi114 ceph-mon[106969]: pgmap v489: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:36 smithi160 ceph-mon[88407]: pgmap v489: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:38.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:38 smithi114 ceph-mon[103029]: pgmap v490: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:38.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:38.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:38 smithi114 ceph-mon[106969]: pgmap v490: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:38.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:38.482 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:02:38.483 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:02:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:38 smithi160 ceph-mon[88407]: pgmap v490: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:39.311 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-14T04:02:36.590801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3545412,"kb_used_data":15452,"kb_used_omap":29,"kb_used_meta":3529826,"kb_avail":746415804,"statfs":{"total":767960285184,"available":764329783296,"internally_reserved":0,"allocated":15822848,"data_stored":9936586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30187,"internal_metadata":3614542357},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002717"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472139+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:01:49.472139+0000","last_peered":"2024-09-14T04:01:49.472139+0000","last_clean":"2024-09-14T04:01:49.472139+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:01:49.472139+0000","last_undegraded":"2024-09-14T04:01:49.472139+0000","last_fullsized":"2024-09-14T04:01:49.472139+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044498+0000","last_change":"2024-09-14T03:55:58.245884+0000","last_active":"2024-09-14T04:01:47.044498+0000","last_peered":"2024-09-14T04:01:47.044498+0000","last_clean":"2024-09-14T04:01:47.044498+0000","last_became_active":"2024-09-14T03:55:58.245547+0000","last_became_peered":"2024-09-14T03:55:58.245547+0000","last_unstale":"2024-09-14T04:01:47.044498+0000","last_undegraded":"2024-09-14T04:01:47.044498+0000","last_fullsized":"2024-09-14T04:01:47.044498+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:30:10.818639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035523+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:01:47.035523+0000","last_peered":"2024-09-14T04:01:47.035523+0000","last_clean":"2024-09-14T04:01:47.035523+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:01:47.035523+0000","last_undegraded":"2024-09-14T04:01:47.035523+0000","last_fullsized":"2024-09-14T04:01:47.035523+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366316+0000","last_change":"2024-09-14T03:58:45.794738+0000","last_active":"2024-09-14T04:01:49.366316+0000","last_peered":"2024-09-14T04:01:49.366316+0000","last_clean":"2024-09-14T04:01:49.366316+0000","last_became_active":"2024-09-14T03:58:45.794111+0000","last_became_peered":"2024-09-14T03:58:45.794111+0000","last_unstale":"2024-09-14T04:01:49.366316+0000","last_undegraded":"2024-09-14T04:01:49.366316+0000","last_fullsized":"2024-09-14T04:01:49.366316+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:12:49.686765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.371005+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:01:49.371005+0000","last_peered":"2024-09-14T04:01:49.371005+0000","last_clean":"2024-09-14T04:01:49.371005+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:01:49.371005+0000","last_undegraded":"2024-09-14T04:01:49.371005+0000","last_fullsized":"2024-09-14T04:01:49.371005+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044358+0000","last_change":"2024-09-14T03:43:11.269452+0000","last_active":"2024-09-14T04:01:47.044358+0000","last_peered":"2024-09-14T04:01:47.044358+0000","last_clean":"2024-09-14T04:01:47.044358+0000","last_became_active":"2024-09-14T03:43:11.269185+0000","last_became_peered":"2024-09-14T03:43:11.269185+0000","last_unstale":"2024-09-14T04:01:47.044358+0000","last_undegraded":"2024-09-14T04:01:47.044358+0000","last_fullsized":"2024-09-14T04:01:47.044358+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:53:12.602272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473429+0000","last_change":"2024-09-14T03:58:45.819617+0000","last_active":"2024-09-14T04:01:49.473429+0000","last_peered":"2024-09-14T04:01:49.473429+0000","last_clean":"2024-09-14T04:01:49.473429+0000","last_became_active":"2024-09-14T03:58:45.819034+0000","last_became_peered":"2024-09-14T03:58:45.819034+0000","last_unstale":"2024-09-14T04:01:49.473429+0000","last_undegraded":"2024-09-14T04:01:49.473429+0000","last_fullsized":"2024-09-14T04:01:49.473429+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:12.478061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":125,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040626+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:01:47.040626+0000","last_peered":"2024-09-14T04:01:47.040626+0000","last_clean":"2024-09-14T04:01:47.040626+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:01:47.040626+0000","last_undegraded":"2024-09-14T04:01:47.040626+0000","last_fullsized":"2024-09-14T04:01:47.040626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044837+0000","last_change":"2024-09-14T03:52:28.189069+0000","last_active":"2024-09-14T04:01:47.044837+0000","last_peered":"2024-09-14T04:01:47.044837+0000","last_clean":"2024-09-14T04:01:47.044837+0000","last_became_active":"2024-09-14T03:52:28.188421+0000","last_became_peered":"2024-09-14T03:52:28.188421+0000","last_unstale":"2024-09-14T04:01:47.044837+0000","last_undegraded":"2024-09-14T04:01:47.044837+0000","last_fullsized":"2024-09-14T04:01:47.044837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:20:09.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035248+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:01:47.035248+0000","last_peered":"2024-09-14T04:01:47.035248+0000","last_clean":"2024-09-14T04:01:47.035248+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:01:47.035248+0000","last_undegraded":"2024-09-14T04:01:47.035248+0000","last_fullsized":"2024-09-14T04:01:47.035248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044841+0000","last_change":"2024-09-14T03:54:09.079772+0000","last_active":"2024-09-14T04:01:47.044841+0000","last_peered":"2024-09-14T04:01:47.044841+0000","last_clean":"2024-09-14T04:01:47.044841+0000","last_became_active":"2024-09-14T03:54:09.079323+0000","last_became_peered":"2024-09-14T03:54:09.079323+0000","last_unstale":"2024-09-14T04:01:47.044841+0000","last_undegraded":"2024-09-14T04:01:47.044841+0000","last_fullsized":"2024-09-14T04:01:47.044841+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:59:58.328220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392726+0000","last_change":"2024-09-14T03:58:45.793252+0000","last_active":"2024-09-14T04:01:47.392726+0000","last_peered":"2024-09-14T04:01:47.392726+0000","last_clean":"2024-09-14T04:01:47.392726+0000","last_became_active":"2024-09-14T03:58:45.792917+0000","last_became_peered":"2024-09-14T03:58:45.792917+0000","last_unstale":"2024-09-14T04:01:47.392726+0000","last_undegraded":"2024-09-14T04:01:47.392726+0000","last_fullsized":"2024-09-14T04:01:47.392726+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:11.399204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044846+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:01:47.044846+0000","last_peered":"2024-09-14T04:01:47.044846+0000","last_clean":"2024-09-14T04:01:47.044846+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:01:47.044846+0000","last_undegraded":"2024-09-14T04:01:47.044846+0000","last_fullsized":"2024-09-14T04:01:47.044846+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040465+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:01:47.040465+0000","last_peered":"2024-09-14T04:01:47.040465+0000","last_clean":"2024-09-14T04:01:47.040465+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:01:47.040465+0000","last_undegraded":"2024-09-14T04:01:47.040465+0000","last_fullsized":"2024-09-14T04:01:47.040465+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035430+0000","last_change":"2024-09-14T03:58:45.810163+0000","last_active":"2024-09-14T04:01:47.035430+0000","last_peered":"2024-09-14T04:01:47.035430+0000","last_clean":"2024-09-14T04:01:47.035430+0000","last_became_active":"2024-09-14T03:58:45.809802+0000","last_became_peered":"2024-09-14T03:58:45.809802+0000","last_unstale":"2024-09-14T04:01:47.035430+0000","last_undegraded":"2024-09-14T04:01:47.035430+0000","last_fullsized":"2024-09-14T04:01:47.035430+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:40:33.911609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366386+0000","last_change":"2024-09-14T03:54:09.079995+0000","last_active":"2024-09-14T04:01:49.366386+0000","last_peered":"2024-09-14T04:01:49.366386+0000","last_clean":"2024-09-14T04:01:49.366386+0000","last_became_active":"2024-09-14T03:54:09.079686+0000","last_became_peered":"2024-09-14T03:54:09.079686+0000","last_unstale":"2024-09-14T04:01:49.366386+0000","last_undegraded":"2024-09-14T04:01:49.366386+0000","last_fullsized":"2024-09-14T04:01:49.366386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:15:31.274947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":134,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472859+0000","last_change":"2024-09-14T03:58:45.788412+0000","last_active":"2024-09-14T04:01:49.472859+0000","last_peered":"2024-09-14T04:01:49.472859+0000","last_clean":"2024-09-14T04:01:49.472859+0000","last_became_active":"2024-09-14T03:58:45.788078+0000","last_became_peered":"2024-09-14T03:58:45.788078+0000","last_unstale":"2024-09-14T04:01:49.472859+0000","last_undegraded":"2024-09-14T04:01:49.472859+0000","last_fullsized":"2024-09-14T04:01:49.472859+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:49:09.289398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392964+0000","last_change":"2024-09-14T03:58:45.812662+0000","last_active":"2024-09-14T04:01:47.392964+0000","last_peered":"2024-09-14T04:01:47.392964+0000","last_clean":"2024-09-14T04:01:47.392964+0000","last_became_active":"2024-09-14T03:58:45.812338+0000","last_became_peered":"2024-09-14T03:58:45.812338+0000","last_unstale":"2024-09-14T04:01:47.392964+0000","last_undegraded":"2024-09-14T04:01:47.392964+0000","last_fullsized":"2024-09-14T04:01:47.392964+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:35:35.258171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040574+0000","last_change":"2024-09-14T03:54:09.089856+0000","last_active":"2024-09-14T04:01:47.040574+0000","last_peered":"2024-09-14T04:01:47.040574+0000","last_clean":"2024-09-14T04:01:47.040574+0000","last_became_active":"2024-09-14T03:54:09.089559+0000","last_became_peered":"2024-09-14T03:54:09.089559+0000","last_unstale":"2024-09-14T04:01:47.040574+0000","last_undegraded":"2024-09-14T04:01:47.040574+0000","last_fullsized":"2024-09-14T04:01:47.040574+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:11:49.651236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393207+0000","last_change":"2024-09-14T03:55:58.249703+0000","last_active":"2024-09-14T04:01:47.393207+0000","last_peered":"2024-09-14T04:01:47.393207+0000","last_clean":"2024-09-14T04:01:47.393207+0000","last_became_active":"2024-09-14T03:55:58.249465+0000","last_became_peered":"2024-09-14T03:55:58.249465+0000","last_unstale":"2024-09-14T04:01:47.393207+0000","last_undegraded":"2024-09-14T04:01:47.393207+0000","last_fullsized":"2024-09-14T04:01:47.393207+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:03.743246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392875+0000","last_change":"2024-09-14T03:55:58.246315+0000","last_active":"2024-09-14T04:01:47.392875+0000","last_peered":"2024-09-14T04:01:47.392875+0000","last_clean":"2024-09-14T04:01:47.392875+0000","last_became_active":"2024-09-14T03:55:58.245997+0000","last_became_peered":"2024-09-14T03:55:58.245997+0000","last_unstale":"2024-09-14T04:01:47.392875+0000","last_undegraded":"2024-09-14T04:01:47.392875+0000","last_fullsized":"2024-09-14T04:01:47.392875+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:11:13.312397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393206+0000","last_change":"2024-09-14T03:54:09.082688+0000","last_active":"2024-09-14T04:01:47.393206+0000","last_peered":"2024-09-14T04:01:47.393206+0000","last_clean":"2024-09-14T04:01:47.393206+0000","last_became_active":"2024-09-14T03:54:09.082402+0000","last_became_peered":"2024-09-14T03:54:09.082402+0000","last_unstale":"2024-09-14T04:01:47.393206+0000","last_undegraded":"2024-09-14T04:01:47.393206+0000","last_fullsized":"2024-09-14T04:01:47.393206+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:28:53.024974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":107,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473967+0000","last_change":"2024-09-14T03:58:45.822672+0000","last_active":"2024-09-14T04:01:49.473967+0000","last_peered":"2024-09-14T04:01:49.473967+0000","last_clean":"2024-09-14T04:01:49.473967+0000","last_became_active":"2024-09-14T03:58:45.822047+0000","last_became_peered":"2024-09-14T03:58:45.822047+0000","last_unstale":"2024-09-14T04:01:49.473967+0000","last_undegraded":"2024-09-14T04:01:49.473967+0000","last_fullsized":"2024-09-14T04:01:49.473967+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:29:49.341047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093884+0000","last_change":"2024-09-14T03:58:45.793497+0000","last_active":"2024-09-14T04:01:49.093884+0000","last_peered":"2024-09-14T04:01:49.093884+0000","last_clean":"2024-09-14T04:01:49.093884+0000","last_became_active":"2024-09-14T03:58:45.793169+0000","last_became_peered":"2024-09-14T03:58:45.793169+0000","last_unstale":"2024-09-14T04:01:49.093884+0000","last_undegraded":"2024-09-14T04:01:49.093884+0000","last_fullsized":"2024-09-14T04:01:49.093884+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:11:06.166164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":233,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044476+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:01:47.044476+0000","last_peered":"2024-09-14T04:01:47.044476+0000","last_clean":"2024-09-14T04:01:47.044476+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:01:47.044476+0000","last_undegraded":"2024-09-14T04:01:47.044476+0000","last_fullsized":"2024-09-14T04:01:47.044476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":130,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035317+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:01:47.035317+0000","last_peered":"2024-09-14T04:01:47.035317+0000","last_clean":"2024-09-14T04:01:47.035317+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:01:47.035317+0000","last_undegraded":"2024-09-14T04:01:47.035317+0000","last_fullsized":"2024-09-14T04:01:47.035317+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":94,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472088+0000","last_change":"2024-09-14T03:58:45.782514+0000","last_active":"2024-09-14T04:01:49.472088+0000","last_peered":"2024-09-14T04:01:49.472088+0000","last_clean":"2024-09-14T04:01:49.472088+0000","last_became_active":"2024-09-14T03:58:45.782196+0000","last_became_peered":"2024-09-14T03:58:45.782196+0000","last_unstale":"2024-09-14T04:01:49.472088+0000","last_undegraded":"2024-09-14T04:01:49.472088+0000","last_fullsized":"2024-09-14T04:01:49.472088+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:59:44.614261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040892+0000","last_change":"2024-09-14T03:43:11.909945+0000","last_active":"2024-09-14T04:01:47.040892+0000","last_peered":"2024-09-14T04:01:47.040892+0000","last_clean":"2024-09-14T04:01:47.040892+0000","last_became_active":"2024-09-14T03:43:11.909666+0000","last_became_peered":"2024-09-14T03:43:11.909666+0000","last_unstale":"2024-09-14T04:01:47.040892+0000","last_undegraded":"2024-09-14T04:01:47.040892+0000","last_fullsized":"2024-09-14T04:01:47.040892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:13:19.629641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":145,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044463+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:01:47.044463+0000","last_peered":"2024-09-14T04:01:47.044463+0000","last_clean":"2024-09-14T04:01:47.044463+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:01:47.044463+0000","last_undegraded":"2024-09-14T04:01:47.044463+0000","last_fullsized":"2024-09-14T04:01:47.044463+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366618+0000","last_change":"2024-09-14T03:58:45.795127+0000","last_active":"2024-09-14T04:01:49.366618+0000","last_peered":"2024-09-14T04:01:49.366618+0000","last_clean":"2024-09-14T04:01:49.366618+0000","last_became_active":"2024-09-14T03:58:45.794831+0000","last_became_peered":"2024-09-14T03:58:45.794831+0000","last_unstale":"2024-09-14T04:01:49.366618+0000","last_undegraded":"2024-09-14T04:01:49.366618+0000","last_fullsized":"2024-09-14T04:01:49.366618+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:22:18.983609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.370773+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:01:49.370773+0000","last_peered":"2024-09-14T04:01:49.370773+0000","last_clean":"2024-09-14T04:01:49.370773+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:01:49.370773+0000","last_undegraded":"2024-09-14T04:01:49.370773+0000","last_fullsized":"2024-09-14T04:01:49.370773+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":168,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471920+0000","last_change":"2024-09-14T03:58:45.791394+0000","last_active":"2024-09-14T04:01:49.471920+0000","last_peered":"2024-09-14T04:01:49.471920+0000","last_clean":"2024-09-14T04:01:49.471920+0000","last_became_active":"2024-09-14T03:58:45.790907+0000","last_became_peered":"2024-09-14T03:58:45.790907+0000","last_unstale":"2024-09-14T04:01:49.471920+0000","last_undegraded":"2024-09-14T04:01:49.471920+0000","last_fullsized":"2024-09-14T04:01:49.471920+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:07.434141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":143,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093019+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:01:49.093019+0000","last_peered":"2024-09-14T04:01:49.093019+0000","last_clean":"2024-09-14T04:01:49.093019+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:01:49.093019+0000","last_undegraded":"2024-09-14T04:01:49.093019+0000","last_fullsized":"2024-09-14T04:01:49.093019+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":168,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093834+0000","last_change":"2024-09-14T03:54:09.087551+0000","last_active":"2024-09-14T04:01:49.093834+0000","last_peered":"2024-09-14T04:01:49.093834+0000","last_clean":"2024-09-14T04:01:49.093834+0000","last_became_active":"2024-09-14T03:54:09.086608+0000","last_became_peered":"2024-09-14T03:54:09.086608+0000","last_unstale":"2024-09-14T04:01:49.093834+0000","last_undegraded":"2024-09-14T04:01:49.093834+0000","last_fullsized":"2024-09-14T04:01:49.093834+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:38:14.920066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035236+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:01:47.035236+0000","last_peered":"2024-09-14T04:01:47.035236+0000","last_clean":"2024-09-14T04:01:47.035236+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:01:47.035236+0000","last_undegraded":"2024-09-14T04:01:47.035236+0000","last_fullsized":"2024-09-14T04:01:47.035236+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393034+0000","last_change":"2024-09-14T03:52:28.187042+0000","last_active":"2024-09-14T04:01:47.393034+0000","last_peered":"2024-09-14T04:01:47.393034+0000","last_clean":"2024-09-14T04:01:47.393034+0000","last_became_active":"2024-09-14T03:52:28.186714+0000","last_became_peered":"2024-09-14T03:52:28.186714+0000","last_unstale":"2024-09-14T04:01:47.393034+0000","last_undegraded":"2024-09-14T04:01:47.393034+0000","last_fullsized":"2024-09-14T04:01:47.393034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:50:57.007224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":154,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093063+0000","last_change":"2024-09-14T03:58:45.812745+0000","last_active":"2024-09-14T04:01:49.093063+0000","last_peered":"2024-09-14T04:01:49.093063+0000","last_clean":"2024-09-14T04:01:49.093063+0000","last_became_active":"2024-09-14T03:58:45.812385+0000","last_became_peered":"2024-09-14T03:58:45.812385+0000","last_unstale":"2024-09-14T04:01:49.093063+0000","last_undegraded":"2024-09-14T04:01:49.093063+0000","last_fullsized":"2024-09-14T04:01:49.093063+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:18:40.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035621+0000","last_change":"2024-09-14T03:58:45.788256+0000","last_active":"2024-09-14T04:01:47.035621+0000","last_peered":"2024-09-14T04:01:47.035621+0000","last_clean":"2024-09-14T04:01:47.035621+0000","last_became_active":"2024-09-14T03:58:45.788052+0000","last_became_peered":"2024-09-14T03:58:45.788052+0000","last_unstale":"2024-09-14T04:01:47.035621+0000","last_undegraded":"2024-09-14T04:01:47.035621+0000","last_fullsized":"2024-09-14T04:01:47.035621+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:38:19.891512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093135+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:01:49.093135+0000","last_peered":"2024-09-14T04:01:49.093135+0000","last_clean":"2024-09-14T04:01:49.093135+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:01:49.093135+0000","last_undegraded":"2024-09-14T04:01:49.093135+0000","last_fullsized":"2024-09-14T04:01:49.093135+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093546+0000","last_change":"2024-09-14T03:54:09.089016+0000","last_active":"2024-09-14T04:01:49.093546+0000","last_peered":"2024-09-14T04:01:49.093546+0000","last_clean":"2024-09-14T04:01:49.093546+0000","last_became_active":"2024-09-14T03:54:09.087653+0000","last_became_peered":"2024-09-14T03:54:09.087653+0000","last_unstale":"2024-09-14T04:01:49.093546+0000","last_undegraded":"2024-09-14T04:01:49.093546+0000","last_fullsized":"2024-09-14T04:01:49.093546+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:07:46.505921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":143,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093259+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:01:49.093259+0000","last_peered":"2024-09-14T04:01:49.093259+0000","last_clean":"2024-09-14T04:01:49.093259+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:01:49.093259+0000","last_undegraded":"2024-09-14T04:01:49.093259+0000","last_fullsized":"2024-09-14T04:01:49.093259+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":128,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044846+0000","last_change":"2024-09-14T03:52:28.190501+0000","last_active":"2024-09-14T04:01:47.044846+0000","last_peered":"2024-09-14T04:01:47.044846+0000","last_clean":"2024-09-14T04:01:47.044846+0000","last_became_active":"2024-09-14T03:52:28.190050+0000","last_became_peered":"2024-09-14T03:52:28.190050+0000","last_unstale":"2024-09-14T04:01:47.044846+0000","last_undegraded":"2024-09-14T04:01:47.044846+0000","last_fullsized":"2024-09-14T04:01:47.044846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:58:22.557305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":166,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366465+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:01:49.366465+0000","last_peered":"2024-09-14T04:01:49.366465+0000","last_clean":"2024-09-14T04:01:49.366465+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:01:49.366465+0000","last_undegraded":"2024-09-14T04:01:49.366465+0000","last_fullsized":"2024-09-14T04:01:49.366465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":134,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473560+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:01:49.473560+0000","last_peered":"2024-09-14T04:01:49.473560+0000","last_clean":"2024-09-14T04:01:49.473560+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:01:49.473560+0000","last_undegraded":"2024-09-14T04:01:49.473560+0000","last_fullsized":"2024-09-14T04:01:49.473560+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093627+0000","last_change":"2024-09-14T03:54:09.082573+0000","last_active":"2024-09-14T04:01:49.093627+0000","last_peered":"2024-09-14T04:01:49.093627+0000","last_clean":"2024-09-14T04:01:49.093627+0000","last_became_active":"2024-09-14T03:54:09.082266+0000","last_became_peered":"2024-09-14T03:54:09.082266+0000","last_unstale":"2024-09-14T04:01:49.093627+0000","last_undegraded":"2024-09-14T04:01:49.093627+0000","last_fullsized":"2024-09-14T04:01:49.093627+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:30:24.244999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":131,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472348+0000","last_change":"2024-09-14T03:58:45.822348+0000","last_active":"2024-09-14T04:01:49.472348+0000","last_peered":"2024-09-14T04:01:49.472348+0000","last_clean":"2024-09-14T04:01:49.472348+0000","last_became_active":"2024-09-14T03:58:45.821621+0000","last_became_peered":"2024-09-14T03:58:45.821621+0000","last_unstale":"2024-09-14T04:01:49.472348+0000","last_undegraded":"2024-09-14T04:01:49.472348+0000","last_fullsized":"2024-09-14T04:01:49.472348+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:02:12.477742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.371059+0000","last_change":"2024-09-14T03:54:09.081554+0000","last_active":"2024-09-14T04:01:49.371059+0000","last_peered":"2024-09-14T04:01:49.371059+0000","last_clean":"2024-09-14T04:01:49.371059+0000","last_became_active":"2024-09-14T03:54:09.081158+0000","last_became_peered":"2024-09-14T03:54:09.081158+0000","last_unstale":"2024-09-14T04:01:49.371059+0000","last_undegraded":"2024-09-14T04:01:49.371059+0000","last_fullsized":"2024-09-14T04:01:49.371059+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:24:43.027300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":129,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040935+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:01:47.040935+0000","last_peered":"2024-09-14T04:01:47.040935+0000","last_clean":"2024-09-14T04:01:47.040935+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:01:47.040935+0000","last_undegraded":"2024-09-14T04:01:47.040935+0000","last_fullsized":"2024-09-14T04:01:47.040935+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":98,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392823+0000","last_change":"2024-09-14T03:43:11.285521+0000","last_active":"2024-09-14T04:01:47.392823+0000","last_peered":"2024-09-14T04:01:47.392823+0000","last_clean":"2024-09-14T04:01:47.392823+0000","last_became_active":"2024-09-14T03:43:11.284778+0000","last_became_peered":"2024-09-14T03:43:11.284778+0000","last_unstale":"2024-09-14T04:01:47.392823+0000","last_undegraded":"2024-09-14T04:01:47.392823+0000","last_fullsized":"2024-09-14T04:01:47.392823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:51:32.744643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":108,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392975+0000","last_change":"2024-09-14T03:43:52.490179+0000","last_active":"2024-09-14T04:01:47.392975+0000","last_peered":"2024-09-14T04:01:47.392975+0000","last_clean":"2024-09-14T04:01:47.392975+0000","last_became_active":"2024-09-14T03:43:52.489786+0000","last_became_peered":"2024-09-14T03:43:52.489786+0000","last_unstale":"2024-09-14T04:01:47.392975+0000","last_undegraded":"2024-09-14T04:01:47.392975+0000","last_fullsized":"2024-09-14T04:01:47.392975+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:55.590247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":121,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393060+0000","last_change":"2024-09-14T03:43:11.274139+0000","last_active":"2024-09-14T04:01:47.393060+0000","last_peered":"2024-09-14T04:01:47.393060+0000","last_clean":"2024-09-14T04:01:47.393060+0000","last_became_active":"2024-09-14T03:43:11.271423+0000","last_became_peered":"2024-09-14T03:43:11.271423+0000","last_unstale":"2024-09-14T04:01:47.393060+0000","last_undegraded":"2024-09-14T04:01:47.393060+0000","last_fullsized":"2024-09-14T04:01:47.393060+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:52.045590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.092936+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:01:49.092936+0000","last_peered":"2024-09-14T04:01:49.092936+0000","last_clean":"2024-09-14T04:01:49.092936+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:01:49.092936+0000","last_undegraded":"2024-09-14T04:01:49.092936+0000","last_fullsized":"2024-09-14T04:01:49.092936+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044710+0000","last_change":"2024-09-14T03:58:45.793790+0000","last_active":"2024-09-14T04:01:47.044710+0000","last_peered":"2024-09-14T04:01:47.044710+0000","last_clean":"2024-09-14T04:01:47.044710+0000","last_became_active":"2024-09-14T03:58:45.793253+0000","last_became_peered":"2024-09-14T03:58:45.793253+0000","last_unstale":"2024-09-14T04:01:47.044710+0000","last_undegraded":"2024-09-14T04:01:47.044710+0000","last_fullsized":"2024-09-14T04:01:47.044710+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:44.988345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035444+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:01:47.035444+0000","last_peered":"2024-09-14T04:01:47.035444+0000","last_clean":"2024-09-14T04:01:47.035444+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:01:47.035444+0000","last_undegraded":"2024-09-14T04:01:47.035444+0000","last_fullsized":"2024-09-14T04:01:47.035444+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":94,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471884+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:01:49.471884+0000","last_peered":"2024-09-14T04:01:49.471884+0000","last_clean":"2024-09-14T04:01:49.471884+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:01:49.471884+0000","last_undegraded":"2024-09-14T04:01:49.471884+0000","last_fullsized":"2024-09-14T04:01:49.471884+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366797+0000","last_change":"2024-09-14T03:58:45.794557+0000","last_active":"2024-09-14T04:01:49.366797+0000","last_peered":"2024-09-14T04:01:49.366797+0000","last_clean":"2024-09-14T04:01:49.366797+0000","last_became_active":"2024-09-14T03:58:45.794173+0000","last_became_peered":"2024-09-14T03:58:45.794173+0000","last_unstale":"2024-09-14T04:01:49.366797+0000","last_undegraded":"2024-09-14T04:01:49.366797+0000","last_fullsized":"2024-09-14T04:01:49.366797+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:33:12.799351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471703+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:01:49.471703+0000","last_peered":"2024-09-14T04:01:49.471703+0000","last_clean":"2024-09-14T04:01:49.471703+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:01:49.471703+0000","last_undegraded":"2024-09-14T04:01:49.471703+0000","last_fullsized":"2024-09-14T04:01:49.471703+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040902+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:01:47.040902+0000","last_peered":"2024-09-14T04:01:47.040902+0000","last_clean":"2024-09-14T04:01:47.040902+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:01:47.040902+0000","last_undegraded":"2024-09-14T04:01:47.040902+0000","last_fullsized":"2024-09-14T04:01:47.040902+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":124,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472437+0000","last_change":"2024-09-14T03:58:45.822679+0000","last_active":"2024-09-14T04:01:49.472437+0000","last_peered":"2024-09-14T04:01:49.472437+0000","last_clean":"2024-09-14T04:01:49.472437+0000","last_became_active":"2024-09-14T03:58:45.822051+0000","last_became_peered":"2024-09-14T03:58:45.822051+0000","last_unstale":"2024-09-14T04:01:49.472437+0000","last_undegraded":"2024-09-14T04:01:49.472437+0000","last_fullsized":"2024-09-14T04:01:49.472437+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:30:52.022607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":178,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.370935+0000","last_change":"2024-09-14T03:58:45.795656+0000","last_active":"2024-09-14T04:01:49.370935+0000","last_peered":"2024-09-14T04:01:49.370935+0000","last_clean":"2024-09-14T04:01:49.370935+0000","last_became_active":"2024-09-14T03:58:45.795369+0000","last_became_peered":"2024-09-14T03:58:45.795369+0000","last_unstale":"2024-09-14T04:01:49.370935+0000","last_undegraded":"2024-09-14T04:01:49.370935+0000","last_fullsized":"2024-09-14T04:01:49.370935+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:45.896015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.092900+0000","last_change":"2024-09-14T03:58:45.793634+0000","last_active":"2024-09-14T04:01:49.092900+0000","last_peered":"2024-09-14T04:01:49.092900+0000","last_clean":"2024-09-14T04:01:49.092900+0000","last_became_active":"2024-09-14T03:58:45.793229+0000","last_became_peered":"2024-09-14T03:58:45.793229+0000","last_unstale":"2024-09-14T04:01:49.092900+0000","last_undegraded":"2024-09-14T04:01:49.092900+0000","last_fullsized":"2024-09-14T04:01:49.092900+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:31.431481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473119+0000","last_change":"2024-09-14T03:58:45.795790+0000","last_active":"2024-09-14T04:01:49.473119+0000","last_peered":"2024-09-14T04:01:49.473119+0000","last_clean":"2024-09-14T04:01:49.473119+0000","last_became_active":"2024-09-14T03:58:45.795468+0000","last_became_peered":"2024-09-14T03:58:45.795468+0000","last_unstale":"2024-09-14T04:01:49.473119+0000","last_undegraded":"2024-09-14T04:01:49.473119+0000","last_fullsized":"2024-09-14T04:01:49.473119+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:50:54.779571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392887+0000","last_change":"2024-09-14T03:55:58.252985+0000","last_active":"2024-09-14T04:01:47.392887+0000","last_peered":"2024-09-14T04:01:47.392887+0000","last_clean":"2024-09-14T04:01:47.392887+0000","last_became_active":"2024-09-14T03:55:58.252479+0000","last_became_peered":"2024-09-14T03:55:58.252479+0000","last_unstale":"2024-09-14T04:01:47.392887+0000","last_undegraded":"2024-09-14T04:01:47.392887+0000","last_fullsized":"2024-09-14T04:01:47.392887+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:46:04.357446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":171,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472629+0000","last_change":"2024-09-14T03:58:45.810289+0000","last_active":"2024-09-14T04:01:49.472629+0000","last_peered":"2024-09-14T04:01:49.472629+0000","last_clean":"2024-09-14T04:01:49.472629+0000","last_became_active":"2024-09-14T03:58:45.809938+0000","last_became_peered":"2024-09-14T03:58:45.809938+0000","last_unstale":"2024-09-14T04:01:49.472629+0000","last_undegraded":"2024-09-14T04:01:49.472629+0000","last_fullsized":"2024-09-14T04:01:49.472629+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:06:18.534582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366607+0000","last_change":"2024-09-14T03:52:28.182347+0000","last_active":"2024-09-14T04:01:49.366607+0000","last_peered":"2024-09-14T04:01:49.366607+0000","last_clean":"2024-09-14T04:01:49.366607+0000","last_became_active":"2024-09-14T03:52:28.181624+0000","last_became_peered":"2024-09-14T03:52:28.181624+0000","last_unstale":"2024-09-14T04:01:49.366607+0000","last_undegraded":"2024-09-14T04:01:49.366607+0000","last_fullsized":"2024-09-14T04:01:49.366607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:18.032619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392932+0000","last_change":"2024-09-14T03:58:45.793294+0000","last_active":"2024-09-14T04:01:47.392932+0000","last_peered":"2024-09-14T04:01:47.392932+0000","last_clean":"2024-09-14T04:01:47.392932+0000","last_became_active":"2024-09-14T03:58:45.793009+0000","last_became_peered":"2024-09-14T03:58:45.793009+0000","last_unstale":"2024-09-14T04:01:47.392932+0000","last_undegraded":"2024-09-14T04:01:47.392932+0000","last_fullsized":"2024-09-14T04:01:47.392932+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:46:00.206467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035459+0000","last_change":"2024-09-14T03:54:09.084402+0000","last_active":"2024-09-14T04:01:47.035459+0000","last_peered":"2024-09-14T04:01:47.035459+0000","last_clean":"2024-09-14T04:01:47.035459+0000","last_became_active":"2024-09-14T03:54:09.084076+0000","last_became_peered":"2024-09-14T03:54:09.084076+0000","last_unstale":"2024-09-14T04:01:47.035459+0000","last_undegraded":"2024-09-14T04:01:47.035459+0000","last_fullsized":"2024-09-14T04:01:47.035459+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:49:51.780807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392790+0000","last_change":"2024-09-14T03:58:45.793059+0000","last_active":"2024-09-14T04:01:47.392790+0000","last_peered":"2024-09-14T04:01:47.392790+0000","last_clean":"2024-09-14T04:01:47.392790+0000","last_became_active":"2024-09-14T03:58:45.792703+0000","last_became_peered":"2024-09-14T03:58:45.792703+0000","last_unstale":"2024-09-14T04:01:47.392790+0000","last_undegraded":"2024-09-14T04:01:47.392790+0000","last_fullsized":"2024-09-14T04:01:47.392790+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:01:17.321430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":102,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035534+0000","last_change":"2024-09-14T03:43:11.278982+0000","last_active":"2024-09-14T04:01:47.035534+0000","last_peered":"2024-09-14T04:01:47.035534+0000","last_clean":"2024-09-14T04:01:47.035534+0000","last_became_active":"2024-09-14T03:43:11.277577+0000","last_became_peered":"2024-09-14T03:43:11.277577+0000","last_unstale":"2024-09-14T04:01:47.035534+0000","last_undegraded":"2024-09-14T04:01:47.035534+0000","last_fullsized":"2024-09-14T04:01:47.035534+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:51.445273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473059+0000","last_change":"2024-09-14T03:58:45.790089+0000","last_active":"2024-09-14T04:01:49.473059+0000","last_peered":"2024-09-14T04:01:49.473059+0000","last_clean":"2024-09-14T04:01:49.473059+0000","last_became_active":"2024-09-14T03:58:45.789707+0000","last_became_peered":"2024-09-14T03:58:45.789707+0000","last_unstale":"2024-09-14T04:01:49.473059+0000","last_undegraded":"2024-09-14T04:01:49.473059+0000","last_fullsized":"2024-09-14T04:01:49.473059+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:03:26.492725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":120,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473285+0000","last_change":"2024-09-14T03:58:45.784441+0000","last_active":"2024-09-14T04:01:49.473285+0000","last_peered":"2024-09-14T04:01:49.473285+0000","last_clean":"2024-09-14T04:01:49.473285+0000","last_became_active":"2024-09-14T03:58:45.783913+0000","last_became_peered":"2024-09-14T03:58:45.783913+0000","last_unstale":"2024-09-14T04:01:49.473285+0000","last_undegraded":"2024-09-14T04:01:49.473285+0000","last_fullsized":"2024-09-14T04:01:49.473285+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:16:11.643108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392630+0000","last_change":"2024-09-14T03:54:09.084713+0000","last_active":"2024-09-14T04:01:47.392630+0000","last_peered":"2024-09-14T04:01:47.392630+0000","last_clean":"2024-09-14T04:01:47.392630+0000","last_became_active":"2024-09-14T03:54:09.082369+0000","last_became_peered":"2024-09-14T03:54:09.082369+0000","last_unstale":"2024-09-14T04:01:47.392630+0000","last_undegraded":"2024-09-14T04:01:47.392630+0000","last_fullsized":"2024-09-14T04:01:47.392630+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:41:39.305100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366512+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:01:49.366512+0000","last_peered":"2024-09-14T04:01:49.366512+0000","last_clean":"2024-09-14T04:01:49.366512+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:01:49.366512+0000","last_undegraded":"2024-09-14T04:01:49.366512+0000","last_fullsized":"2024-09-14T04:01:49.366512+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430466,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600536,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93144616,"statfs":{"total":95995035648,"available":95380086784,"internally_reserved":0,"allocated":2367488,"data_stored":1614300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626695,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299660,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445492,"statfs":{"total":95995035648,"available":95688183808,"internally_reserved":0,"allocated":2355200,"data_stored":1613858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822923,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911160,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":909438,"kb_avail":92833992,"statfs":{"total":95995035648,"available":95062007808,"internally_reserved":0,"allocated":1748992,"data_stored":1019001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931264856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019151,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599944,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93145208,"statfs":{"total":95995035648,"available":95380692992,"internally_reserved":0,"allocated":1765376,"data_stored":1023547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552304,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":201144,"kb_used_data":1708,"kb_used_omap":4,"kb_used_meta":199419,"kb_avail":93544008,"statfs":{"total":95995035648,"available":95789064192,"internally_reserved":0,"allocated":1748992,"data_stored":1019082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":204205501},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435346,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212852,"kb_used_data":1704,"kb_used_omap":6,"kb_used_meta":211129,"kb_avail":93532300,"statfs":{"total":95995035648,"available":95777075200,"internally_reserved":0,"allocated":1744896,"data_stored":1018134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":216196557},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827815,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327416,"kb_used_data":1708,"kb_used_omap":5,"kb_used_meta":325690,"kb_avail":93417736,"statfs":{"total":95995035648,"available":95659761664,"internally_reserved":0,"allocated":1748992,"data_stored":1019011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":333506799},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383803,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392700,"kb_used_data":2288,"kb_used_omap":5,"kb_used_meta":390394,"kb_avail":93352452,"statfs":{"total":95995035648,"available":95592910848,"internally_reserved":0,"allocated":2342912,"data_stored":1609653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":399763516},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:02:39.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:02:39.556 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:39 smithi160 ceph-mon[88407]: from='client.35592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:39 smithi160 ceph-mon[88407]: pgmap v491: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:39 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:39.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:39 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:39.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:39 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:02:39.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:39 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:02:39.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:39 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:39.947 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[103029]: from='client.35592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:39.947 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[103029]: pgmap v491: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:39.947 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:39.947 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:39.947 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:02:39.947 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:02:39.947 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:39.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[106969]: from='client.35592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:39.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[106969]: pgmap v491: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:39.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:39.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:39.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:02:39.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:02:39.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:39 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:02:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:02:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:02:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:02:41.323 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:02:41.324 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:02:41.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:02:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:02:41] "GET /metrics HTTP/1.1" 200 34002 "" "Prometheus/2.51.0" 2024-09-14T04:02:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:41 smithi160 ceph-mon[88407]: pgmap v492: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:41 smithi160 ceph-mon[88407]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:42.083 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:41 smithi114 ceph-mon[103029]: pgmap v492: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:42.083 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:41 smithi114 ceph-mon[103029]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:42.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:41 smithi114 ceph-mon[106969]: pgmap v492: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:42.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:41 smithi114 ceph-mon[106969]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:42.107 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-14T04:02:40.591602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3545412,"kb_used_data":15452,"kb_used_omap":29,"kb_used_meta":3529826,"kb_avail":746415804,"statfs":{"total":767960285184,"available":764329783296,"internally_reserved":0,"allocated":15822848,"data_stored":9936586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30187,"internal_metadata":3614542357},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002669"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472139+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:01:49.472139+0000","last_peered":"2024-09-14T04:01:49.472139+0000","last_clean":"2024-09-14T04:01:49.472139+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:01:49.472139+0000","last_undegraded":"2024-09-14T04:01:49.472139+0000","last_fullsized":"2024-09-14T04:01:49.472139+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044498+0000","last_change":"2024-09-14T03:55:58.245884+0000","last_active":"2024-09-14T04:01:47.044498+0000","last_peered":"2024-09-14T04:01:47.044498+0000","last_clean":"2024-09-14T04:01:47.044498+0000","last_became_active":"2024-09-14T03:55:58.245547+0000","last_became_peered":"2024-09-14T03:55:58.245547+0000","last_unstale":"2024-09-14T04:01:47.044498+0000","last_undegraded":"2024-09-14T04:01:47.044498+0000","last_fullsized":"2024-09-14T04:01:47.044498+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:30:10.818639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035523+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:01:47.035523+0000","last_peered":"2024-09-14T04:01:47.035523+0000","last_clean":"2024-09-14T04:01:47.035523+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:01:47.035523+0000","last_undegraded":"2024-09-14T04:01:47.035523+0000","last_fullsized":"2024-09-14T04:01:47.035523+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366316+0000","last_change":"2024-09-14T03:58:45.794738+0000","last_active":"2024-09-14T04:01:49.366316+0000","last_peered":"2024-09-14T04:01:49.366316+0000","last_clean":"2024-09-14T04:01:49.366316+0000","last_became_active":"2024-09-14T03:58:45.794111+0000","last_became_peered":"2024-09-14T03:58:45.794111+0000","last_unstale":"2024-09-14T04:01:49.366316+0000","last_undegraded":"2024-09-14T04:01:49.366316+0000","last_fullsized":"2024-09-14T04:01:49.366316+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:12:49.686765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.371005+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:01:49.371005+0000","last_peered":"2024-09-14T04:01:49.371005+0000","last_clean":"2024-09-14T04:01:49.371005+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:01:49.371005+0000","last_undegraded":"2024-09-14T04:01:49.371005+0000","last_fullsized":"2024-09-14T04:01:49.371005+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044358+0000","last_change":"2024-09-14T03:43:11.269452+0000","last_active":"2024-09-14T04:01:47.044358+0000","last_peered":"2024-09-14T04:01:47.044358+0000","last_clean":"2024-09-14T04:01:47.044358+0000","last_became_active":"2024-09-14T03:43:11.269185+0000","last_became_peered":"2024-09-14T03:43:11.269185+0000","last_unstale":"2024-09-14T04:01:47.044358+0000","last_undegraded":"2024-09-14T04:01:47.044358+0000","last_fullsized":"2024-09-14T04:01:47.044358+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:53:12.602272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473429+0000","last_change":"2024-09-14T03:58:45.819617+0000","last_active":"2024-09-14T04:01:49.473429+0000","last_peered":"2024-09-14T04:01:49.473429+0000","last_clean":"2024-09-14T04:01:49.473429+0000","last_became_active":"2024-09-14T03:58:45.819034+0000","last_became_peered":"2024-09-14T03:58:45.819034+0000","last_unstale":"2024-09-14T04:01:49.473429+0000","last_undegraded":"2024-09-14T04:01:49.473429+0000","last_fullsized":"2024-09-14T04:01:49.473429+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:12.478061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":125,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040626+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:01:47.040626+0000","last_peered":"2024-09-14T04:01:47.040626+0000","last_clean":"2024-09-14T04:01:47.040626+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:01:47.040626+0000","last_undegraded":"2024-09-14T04:01:47.040626+0000","last_fullsized":"2024-09-14T04:01:47.040626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044837+0000","last_change":"2024-09-14T03:52:28.189069+0000","last_active":"2024-09-14T04:01:47.044837+0000","last_peered":"2024-09-14T04:01:47.044837+0000","last_clean":"2024-09-14T04:01:47.044837+0000","last_became_active":"2024-09-14T03:52:28.188421+0000","last_became_peered":"2024-09-14T03:52:28.188421+0000","last_unstale":"2024-09-14T04:01:47.044837+0000","last_undegraded":"2024-09-14T04:01:47.044837+0000","last_fullsized":"2024-09-14T04:01:47.044837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:20:09.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035248+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:01:47.035248+0000","last_peered":"2024-09-14T04:01:47.035248+0000","last_clean":"2024-09-14T04:01:47.035248+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:01:47.035248+0000","last_undegraded":"2024-09-14T04:01:47.035248+0000","last_fullsized":"2024-09-14T04:01:47.035248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044841+0000","last_change":"2024-09-14T03:54:09.079772+0000","last_active":"2024-09-14T04:01:47.044841+0000","last_peered":"2024-09-14T04:01:47.044841+0000","last_clean":"2024-09-14T04:01:47.044841+0000","last_became_active":"2024-09-14T03:54:09.079323+0000","last_became_peered":"2024-09-14T03:54:09.079323+0000","last_unstale":"2024-09-14T04:01:47.044841+0000","last_undegraded":"2024-09-14T04:01:47.044841+0000","last_fullsized":"2024-09-14T04:01:47.044841+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:59:58.328220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392726+0000","last_change":"2024-09-14T03:58:45.793252+0000","last_active":"2024-09-14T04:01:47.392726+0000","last_peered":"2024-09-14T04:01:47.392726+0000","last_clean":"2024-09-14T04:01:47.392726+0000","last_became_active":"2024-09-14T03:58:45.792917+0000","last_became_peered":"2024-09-14T03:58:45.792917+0000","last_unstale":"2024-09-14T04:01:47.392726+0000","last_undegraded":"2024-09-14T04:01:47.392726+0000","last_fullsized":"2024-09-14T04:01:47.392726+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:11.399204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044846+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:01:47.044846+0000","last_peered":"2024-09-14T04:01:47.044846+0000","last_clean":"2024-09-14T04:01:47.044846+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:01:47.044846+0000","last_undegraded":"2024-09-14T04:01:47.044846+0000","last_fullsized":"2024-09-14T04:01:47.044846+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040465+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:01:47.040465+0000","last_peered":"2024-09-14T04:01:47.040465+0000","last_clean":"2024-09-14T04:01:47.040465+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:01:47.040465+0000","last_undegraded":"2024-09-14T04:01:47.040465+0000","last_fullsized":"2024-09-14T04:01:47.040465+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035430+0000","last_change":"2024-09-14T03:58:45.810163+0000","last_active":"2024-09-14T04:01:47.035430+0000","last_peered":"2024-09-14T04:01:47.035430+0000","last_clean":"2024-09-14T04:01:47.035430+0000","last_became_active":"2024-09-14T03:58:45.809802+0000","last_became_peered":"2024-09-14T03:58:45.809802+0000","last_unstale":"2024-09-14T04:01:47.035430+0000","last_undegraded":"2024-09-14T04:01:47.035430+0000","last_fullsized":"2024-09-14T04:01:47.035430+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:40:33.911609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366386+0000","last_change":"2024-09-14T03:54:09.079995+0000","last_active":"2024-09-14T04:01:49.366386+0000","last_peered":"2024-09-14T04:01:49.366386+0000","last_clean":"2024-09-14T04:01:49.366386+0000","last_became_active":"2024-09-14T03:54:09.079686+0000","last_became_peered":"2024-09-14T03:54:09.079686+0000","last_unstale":"2024-09-14T04:01:49.366386+0000","last_undegraded":"2024-09-14T04:01:49.366386+0000","last_fullsized":"2024-09-14T04:01:49.366386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:15:31.274947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":134,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472859+0000","last_change":"2024-09-14T03:58:45.788412+0000","last_active":"2024-09-14T04:01:49.472859+0000","last_peered":"2024-09-14T04:01:49.472859+0000","last_clean":"2024-09-14T04:01:49.472859+0000","last_became_active":"2024-09-14T03:58:45.788078+0000","last_became_peered":"2024-09-14T03:58:45.788078+0000","last_unstale":"2024-09-14T04:01:49.472859+0000","last_undegraded":"2024-09-14T04:01:49.472859+0000","last_fullsized":"2024-09-14T04:01:49.472859+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:49:09.289398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392964+0000","last_change":"2024-09-14T03:58:45.812662+0000","last_active":"2024-09-14T04:01:47.392964+0000","last_peered":"2024-09-14T04:01:47.392964+0000","last_clean":"2024-09-14T04:01:47.392964+0000","last_became_active":"2024-09-14T03:58:45.812338+0000","last_became_peered":"2024-09-14T03:58:45.812338+0000","last_unstale":"2024-09-14T04:01:47.392964+0000","last_undegraded":"2024-09-14T04:01:47.392964+0000","last_fullsized":"2024-09-14T04:01:47.392964+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:35:35.258171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040574+0000","last_change":"2024-09-14T03:54:09.089856+0000","last_active":"2024-09-14T04:01:47.040574+0000","last_peered":"2024-09-14T04:01:47.040574+0000","last_clean":"2024-09-14T04:01:47.040574+0000","last_became_active":"2024-09-14T03:54:09.089559+0000","last_became_peered":"2024-09-14T03:54:09.089559+0000","last_unstale":"2024-09-14T04:01:47.040574+0000","last_undegraded":"2024-09-14T04:01:47.040574+0000","last_fullsized":"2024-09-14T04:01:47.040574+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:11:49.651236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393207+0000","last_change":"2024-09-14T03:55:58.249703+0000","last_active":"2024-09-14T04:01:47.393207+0000","last_peered":"2024-09-14T04:01:47.393207+0000","last_clean":"2024-09-14T04:01:47.393207+0000","last_became_active":"2024-09-14T03:55:58.249465+0000","last_became_peered":"2024-09-14T03:55:58.249465+0000","last_unstale":"2024-09-14T04:01:47.393207+0000","last_undegraded":"2024-09-14T04:01:47.393207+0000","last_fullsized":"2024-09-14T04:01:47.393207+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:03.743246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392875+0000","last_change":"2024-09-14T03:55:58.246315+0000","last_active":"2024-09-14T04:01:47.392875+0000","last_peered":"2024-09-14T04:01:47.392875+0000","last_clean":"2024-09-14T04:01:47.392875+0000","last_became_active":"2024-09-14T03:55:58.245997+0000","last_became_peered":"2024-09-14T03:55:58.245997+0000","last_unstale":"2024-09-14T04:01:47.392875+0000","last_undegraded":"2024-09-14T04:01:47.392875+0000","last_fullsized":"2024-09-14T04:01:47.392875+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:11:13.312397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393206+0000","last_change":"2024-09-14T03:54:09.082688+0000","last_active":"2024-09-14T04:01:47.393206+0000","last_peered":"2024-09-14T04:01:47.393206+0000","last_clean":"2024-09-14T04:01:47.393206+0000","last_became_active":"2024-09-14T03:54:09.082402+0000","last_became_peered":"2024-09-14T03:54:09.082402+0000","last_unstale":"2024-09-14T04:01:47.393206+0000","last_undegraded":"2024-09-14T04:01:47.393206+0000","last_fullsized":"2024-09-14T04:01:47.393206+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:28:53.024974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":107,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473967+0000","last_change":"2024-09-14T03:58:45.822672+0000","last_active":"2024-09-14T04:01:49.473967+0000","last_peered":"2024-09-14T04:01:49.473967+0000","last_clean":"2024-09-14T04:01:49.473967+0000","last_became_active":"2024-09-14T03:58:45.822047+0000","last_became_peered":"2024-09-14T03:58:45.822047+0000","last_unstale":"2024-09-14T04:01:49.473967+0000","last_undegraded":"2024-09-14T04:01:49.473967+0000","last_fullsized":"2024-09-14T04:01:49.473967+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:29:49.341047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093884+0000","last_change":"2024-09-14T03:58:45.793497+0000","last_active":"2024-09-14T04:01:49.093884+0000","last_peered":"2024-09-14T04:01:49.093884+0000","last_clean":"2024-09-14T04:01:49.093884+0000","last_became_active":"2024-09-14T03:58:45.793169+0000","last_became_peered":"2024-09-14T03:58:45.793169+0000","last_unstale":"2024-09-14T04:01:49.093884+0000","last_undegraded":"2024-09-14T04:01:49.093884+0000","last_fullsized":"2024-09-14T04:01:49.093884+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:11:06.166164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":233,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044476+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:01:47.044476+0000","last_peered":"2024-09-14T04:01:47.044476+0000","last_clean":"2024-09-14T04:01:47.044476+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:01:47.044476+0000","last_undegraded":"2024-09-14T04:01:47.044476+0000","last_fullsized":"2024-09-14T04:01:47.044476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":130,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035317+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:01:47.035317+0000","last_peered":"2024-09-14T04:01:47.035317+0000","last_clean":"2024-09-14T04:01:47.035317+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:01:47.035317+0000","last_undegraded":"2024-09-14T04:01:47.035317+0000","last_fullsized":"2024-09-14T04:01:47.035317+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":94,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472088+0000","last_change":"2024-09-14T03:58:45.782514+0000","last_active":"2024-09-14T04:01:49.472088+0000","last_peered":"2024-09-14T04:01:49.472088+0000","last_clean":"2024-09-14T04:01:49.472088+0000","last_became_active":"2024-09-14T03:58:45.782196+0000","last_became_peered":"2024-09-14T03:58:45.782196+0000","last_unstale":"2024-09-14T04:01:49.472088+0000","last_undegraded":"2024-09-14T04:01:49.472088+0000","last_fullsized":"2024-09-14T04:01:49.472088+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:59:44.614261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040892+0000","last_change":"2024-09-14T03:43:11.909945+0000","last_active":"2024-09-14T04:01:47.040892+0000","last_peered":"2024-09-14T04:01:47.040892+0000","last_clean":"2024-09-14T04:01:47.040892+0000","last_became_active":"2024-09-14T03:43:11.909666+0000","last_became_peered":"2024-09-14T03:43:11.909666+0000","last_unstale":"2024-09-14T04:01:47.040892+0000","last_undegraded":"2024-09-14T04:01:47.040892+0000","last_fullsized":"2024-09-14T04:01:47.040892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:13:19.629641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":145,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044463+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:01:47.044463+0000","last_peered":"2024-09-14T04:01:47.044463+0000","last_clean":"2024-09-14T04:01:47.044463+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:01:47.044463+0000","last_undegraded":"2024-09-14T04:01:47.044463+0000","last_fullsized":"2024-09-14T04:01:47.044463+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366618+0000","last_change":"2024-09-14T03:58:45.795127+0000","last_active":"2024-09-14T04:01:49.366618+0000","last_peered":"2024-09-14T04:01:49.366618+0000","last_clean":"2024-09-14T04:01:49.366618+0000","last_became_active":"2024-09-14T03:58:45.794831+0000","last_became_peered":"2024-09-14T03:58:45.794831+0000","last_unstale":"2024-09-14T04:01:49.366618+0000","last_undegraded":"2024-09-14T04:01:49.366618+0000","last_fullsized":"2024-09-14T04:01:49.366618+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:22:18.983609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.370773+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:01:49.370773+0000","last_peered":"2024-09-14T04:01:49.370773+0000","last_clean":"2024-09-14T04:01:49.370773+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:01:49.370773+0000","last_undegraded":"2024-09-14T04:01:49.370773+0000","last_fullsized":"2024-09-14T04:01:49.370773+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":168,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471920+0000","last_change":"2024-09-14T03:58:45.791394+0000","last_active":"2024-09-14T04:01:49.471920+0000","last_peered":"2024-09-14T04:01:49.471920+0000","last_clean":"2024-09-14T04:01:49.471920+0000","last_became_active":"2024-09-14T03:58:45.790907+0000","last_became_peered":"2024-09-14T03:58:45.790907+0000","last_unstale":"2024-09-14T04:01:49.471920+0000","last_undegraded":"2024-09-14T04:01:49.471920+0000","last_fullsized":"2024-09-14T04:01:49.471920+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:07.434141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":143,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093019+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:01:49.093019+0000","last_peered":"2024-09-14T04:01:49.093019+0000","last_clean":"2024-09-14T04:01:49.093019+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:01:49.093019+0000","last_undegraded":"2024-09-14T04:01:49.093019+0000","last_fullsized":"2024-09-14T04:01:49.093019+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":168,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093834+0000","last_change":"2024-09-14T03:54:09.087551+0000","last_active":"2024-09-14T04:01:49.093834+0000","last_peered":"2024-09-14T04:01:49.093834+0000","last_clean":"2024-09-14T04:01:49.093834+0000","last_became_active":"2024-09-14T03:54:09.086608+0000","last_became_peered":"2024-09-14T03:54:09.086608+0000","last_unstale":"2024-09-14T04:01:49.093834+0000","last_undegraded":"2024-09-14T04:01:49.093834+0000","last_fullsized":"2024-09-14T04:01:49.093834+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:38:14.920066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035236+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:01:47.035236+0000","last_peered":"2024-09-14T04:01:47.035236+0000","last_clean":"2024-09-14T04:01:47.035236+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:01:47.035236+0000","last_undegraded":"2024-09-14T04:01:47.035236+0000","last_fullsized":"2024-09-14T04:01:47.035236+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393034+0000","last_change":"2024-09-14T03:52:28.187042+0000","last_active":"2024-09-14T04:01:47.393034+0000","last_peered":"2024-09-14T04:01:47.393034+0000","last_clean":"2024-09-14T04:01:47.393034+0000","last_became_active":"2024-09-14T03:52:28.186714+0000","last_became_peered":"2024-09-14T03:52:28.186714+0000","last_unstale":"2024-09-14T04:01:47.393034+0000","last_undegraded":"2024-09-14T04:01:47.393034+0000","last_fullsized":"2024-09-14T04:01:47.393034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:50:57.007224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":154,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093063+0000","last_change":"2024-09-14T03:58:45.812745+0000","last_active":"2024-09-14T04:01:49.093063+0000","last_peered":"2024-09-14T04:01:49.093063+0000","last_clean":"2024-09-14T04:01:49.093063+0000","last_became_active":"2024-09-14T03:58:45.812385+0000","last_became_peered":"2024-09-14T03:58:45.812385+0000","last_unstale":"2024-09-14T04:01:49.093063+0000","last_undegraded":"2024-09-14T04:01:49.093063+0000","last_fullsized":"2024-09-14T04:01:49.093063+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:18:40.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035621+0000","last_change":"2024-09-14T03:58:45.788256+0000","last_active":"2024-09-14T04:01:47.035621+0000","last_peered":"2024-09-14T04:01:47.035621+0000","last_clean":"2024-09-14T04:01:47.035621+0000","last_became_active":"2024-09-14T03:58:45.788052+0000","last_became_peered":"2024-09-14T03:58:45.788052+0000","last_unstale":"2024-09-14T04:01:47.035621+0000","last_undegraded":"2024-09-14T04:01:47.035621+0000","last_fullsized":"2024-09-14T04:01:47.035621+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:38:19.891512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093135+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:01:49.093135+0000","last_peered":"2024-09-14T04:01:49.093135+0000","last_clean":"2024-09-14T04:01:49.093135+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:01:49.093135+0000","last_undegraded":"2024-09-14T04:01:49.093135+0000","last_fullsized":"2024-09-14T04:01:49.093135+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093546+0000","last_change":"2024-09-14T03:54:09.089016+0000","last_active":"2024-09-14T04:01:49.093546+0000","last_peered":"2024-09-14T04:01:49.093546+0000","last_clean":"2024-09-14T04:01:49.093546+0000","last_became_active":"2024-09-14T03:54:09.087653+0000","last_became_peered":"2024-09-14T03:54:09.087653+0000","last_unstale":"2024-09-14T04:01:49.093546+0000","last_undegraded":"2024-09-14T04:01:49.093546+0000","last_fullsized":"2024-09-14T04:01:49.093546+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:07:46.505921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":143,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093259+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:01:49.093259+0000","last_peered":"2024-09-14T04:01:49.093259+0000","last_clean":"2024-09-14T04:01:49.093259+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:01:49.093259+0000","last_undegraded":"2024-09-14T04:01:49.093259+0000","last_fullsized":"2024-09-14T04:01:49.093259+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":128,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044846+0000","last_change":"2024-09-14T03:52:28.190501+0000","last_active":"2024-09-14T04:01:47.044846+0000","last_peered":"2024-09-14T04:01:47.044846+0000","last_clean":"2024-09-14T04:01:47.044846+0000","last_became_active":"2024-09-14T03:52:28.190050+0000","last_became_peered":"2024-09-14T03:52:28.190050+0000","last_unstale":"2024-09-14T04:01:47.044846+0000","last_undegraded":"2024-09-14T04:01:47.044846+0000","last_fullsized":"2024-09-14T04:01:47.044846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:58:22.557305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":166,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366465+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:01:49.366465+0000","last_peered":"2024-09-14T04:01:49.366465+0000","last_clean":"2024-09-14T04:01:49.366465+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:01:49.366465+0000","last_undegraded":"2024-09-14T04:01:49.366465+0000","last_fullsized":"2024-09-14T04:01:49.366465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":134,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473560+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:01:49.473560+0000","last_peered":"2024-09-14T04:01:49.473560+0000","last_clean":"2024-09-14T04:01:49.473560+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:01:49.473560+0000","last_undegraded":"2024-09-14T04:01:49.473560+0000","last_fullsized":"2024-09-14T04:01:49.473560+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093627+0000","last_change":"2024-09-14T03:54:09.082573+0000","last_active":"2024-09-14T04:01:49.093627+0000","last_peered":"2024-09-14T04:01:49.093627+0000","last_clean":"2024-09-14T04:01:49.093627+0000","last_became_active":"2024-09-14T03:54:09.082266+0000","last_became_peered":"2024-09-14T03:54:09.082266+0000","last_unstale":"2024-09-14T04:01:49.093627+0000","last_undegraded":"2024-09-14T04:01:49.093627+0000","last_fullsized":"2024-09-14T04:01:49.093627+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:30:24.244999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":131,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472348+0000","last_change":"2024-09-14T03:58:45.822348+0000","last_active":"2024-09-14T04:01:49.472348+0000","last_peered":"2024-09-14T04:01:49.472348+0000","last_clean":"2024-09-14T04:01:49.472348+0000","last_became_active":"2024-09-14T03:58:45.821621+0000","last_became_peered":"2024-09-14T03:58:45.821621+0000","last_unstale":"2024-09-14T04:01:49.472348+0000","last_undegraded":"2024-09-14T04:01:49.472348+0000","last_fullsized":"2024-09-14T04:01:49.472348+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:02:12.477742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.371059+0000","last_change":"2024-09-14T03:54:09.081554+0000","last_active":"2024-09-14T04:01:49.371059+0000","last_peered":"2024-09-14T04:01:49.371059+0000","last_clean":"2024-09-14T04:01:49.371059+0000","last_became_active":"2024-09-14T03:54:09.081158+0000","last_became_peered":"2024-09-14T03:54:09.081158+0000","last_unstale":"2024-09-14T04:01:49.371059+0000","last_undegraded":"2024-09-14T04:01:49.371059+0000","last_fullsized":"2024-09-14T04:01:49.371059+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:24:43.027300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":129,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040935+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:01:47.040935+0000","last_peered":"2024-09-14T04:01:47.040935+0000","last_clean":"2024-09-14T04:01:47.040935+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:01:47.040935+0000","last_undegraded":"2024-09-14T04:01:47.040935+0000","last_fullsized":"2024-09-14T04:01:47.040935+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":98,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392823+0000","last_change":"2024-09-14T03:43:11.285521+0000","last_active":"2024-09-14T04:01:47.392823+0000","last_peered":"2024-09-14T04:01:47.392823+0000","last_clean":"2024-09-14T04:01:47.392823+0000","last_became_active":"2024-09-14T03:43:11.284778+0000","last_became_peered":"2024-09-14T03:43:11.284778+0000","last_unstale":"2024-09-14T04:01:47.392823+0000","last_undegraded":"2024-09-14T04:01:47.392823+0000","last_fullsized":"2024-09-14T04:01:47.392823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:51:32.744643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":108,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392975+0000","last_change":"2024-09-14T03:43:52.490179+0000","last_active":"2024-09-14T04:01:47.392975+0000","last_peered":"2024-09-14T04:01:47.392975+0000","last_clean":"2024-09-14T04:01:47.392975+0000","last_became_active":"2024-09-14T03:43:52.489786+0000","last_became_peered":"2024-09-14T03:43:52.489786+0000","last_unstale":"2024-09-14T04:01:47.392975+0000","last_undegraded":"2024-09-14T04:01:47.392975+0000","last_fullsized":"2024-09-14T04:01:47.392975+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:55.590247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":121,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393060+0000","last_change":"2024-09-14T03:43:11.274139+0000","last_active":"2024-09-14T04:01:47.393060+0000","last_peered":"2024-09-14T04:01:47.393060+0000","last_clean":"2024-09-14T04:01:47.393060+0000","last_became_active":"2024-09-14T03:43:11.271423+0000","last_became_peered":"2024-09-14T03:43:11.271423+0000","last_unstale":"2024-09-14T04:01:47.393060+0000","last_undegraded":"2024-09-14T04:01:47.393060+0000","last_fullsized":"2024-09-14T04:01:47.393060+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:52.045590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.092936+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:01:49.092936+0000","last_peered":"2024-09-14T04:01:49.092936+0000","last_clean":"2024-09-14T04:01:49.092936+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:01:49.092936+0000","last_undegraded":"2024-09-14T04:01:49.092936+0000","last_fullsized":"2024-09-14T04:01:49.092936+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044710+0000","last_change":"2024-09-14T03:58:45.793790+0000","last_active":"2024-09-14T04:01:47.044710+0000","last_peered":"2024-09-14T04:01:47.044710+0000","last_clean":"2024-09-14T04:01:47.044710+0000","last_became_active":"2024-09-14T03:58:45.793253+0000","last_became_peered":"2024-09-14T03:58:45.793253+0000","last_unstale":"2024-09-14T04:01:47.044710+0000","last_undegraded":"2024-09-14T04:01:47.044710+0000","last_fullsized":"2024-09-14T04:01:47.044710+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:44.988345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035444+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:01:47.035444+0000","last_peered":"2024-09-14T04:01:47.035444+0000","last_clean":"2024-09-14T04:01:47.035444+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:01:47.035444+0000","last_undegraded":"2024-09-14T04:01:47.035444+0000","last_fullsized":"2024-09-14T04:01:47.035444+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":94,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471884+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:01:49.471884+0000","last_peered":"2024-09-14T04:01:49.471884+0000","last_clean":"2024-09-14T04:01:49.471884+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:01:49.471884+0000","last_undegraded":"2024-09-14T04:01:49.471884+0000","last_fullsized":"2024-09-14T04:01:49.471884+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366797+0000","last_change":"2024-09-14T03:58:45.794557+0000","last_active":"2024-09-14T04:01:49.366797+0000","last_peered":"2024-09-14T04:01:49.366797+0000","last_clean":"2024-09-14T04:01:49.366797+0000","last_became_active":"2024-09-14T03:58:45.794173+0000","last_became_peered":"2024-09-14T03:58:45.794173+0000","last_unstale":"2024-09-14T04:01:49.366797+0000","last_undegraded":"2024-09-14T04:01:49.366797+0000","last_fullsized":"2024-09-14T04:01:49.366797+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:33:12.799351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471703+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:01:49.471703+0000","last_peered":"2024-09-14T04:01:49.471703+0000","last_clean":"2024-09-14T04:01:49.471703+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:01:49.471703+0000","last_undegraded":"2024-09-14T04:01:49.471703+0000","last_fullsized":"2024-09-14T04:01:49.471703+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040902+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:01:47.040902+0000","last_peered":"2024-09-14T04:01:47.040902+0000","last_clean":"2024-09-14T04:01:47.040902+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:01:47.040902+0000","last_undegraded":"2024-09-14T04:01:47.040902+0000","last_fullsized":"2024-09-14T04:01:47.040902+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":124,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472437+0000","last_change":"2024-09-14T03:58:45.822679+0000","last_active":"2024-09-14T04:01:49.472437+0000","last_peered":"2024-09-14T04:01:49.472437+0000","last_clean":"2024-09-14T04:01:49.472437+0000","last_became_active":"2024-09-14T03:58:45.822051+0000","last_became_peered":"2024-09-14T03:58:45.822051+0000","last_unstale":"2024-09-14T04:01:49.472437+0000","last_undegraded":"2024-09-14T04:01:49.472437+0000","last_fullsized":"2024-09-14T04:01:49.472437+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:30:52.022607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":178,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.370935+0000","last_change":"2024-09-14T03:58:45.795656+0000","last_active":"2024-09-14T04:01:49.370935+0000","last_peered":"2024-09-14T04:01:49.370935+0000","last_clean":"2024-09-14T04:01:49.370935+0000","last_became_active":"2024-09-14T03:58:45.795369+0000","last_became_peered":"2024-09-14T03:58:45.795369+0000","last_unstale":"2024-09-14T04:01:49.370935+0000","last_undegraded":"2024-09-14T04:01:49.370935+0000","last_fullsized":"2024-09-14T04:01:49.370935+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:45.896015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.092900+0000","last_change":"2024-09-14T03:58:45.793634+0000","last_active":"2024-09-14T04:01:49.092900+0000","last_peered":"2024-09-14T04:01:49.092900+0000","last_clean":"2024-09-14T04:01:49.092900+0000","last_became_active":"2024-09-14T03:58:45.793229+0000","last_became_peered":"2024-09-14T03:58:45.793229+0000","last_unstale":"2024-09-14T04:01:49.092900+0000","last_undegraded":"2024-09-14T04:01:49.092900+0000","last_fullsized":"2024-09-14T04:01:49.092900+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:31.431481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473119+0000","last_change":"2024-09-14T03:58:45.795790+0000","last_active":"2024-09-14T04:01:49.473119+0000","last_peered":"2024-09-14T04:01:49.473119+0000","last_clean":"2024-09-14T04:01:49.473119+0000","last_became_active":"2024-09-14T03:58:45.795468+0000","last_became_peered":"2024-09-14T03:58:45.795468+0000","last_unstale":"2024-09-14T04:01:49.473119+0000","last_undegraded":"2024-09-14T04:01:49.473119+0000","last_fullsized":"2024-09-14T04:01:49.473119+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:50:54.779571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392887+0000","last_change":"2024-09-14T03:55:58.252985+0000","last_active":"2024-09-14T04:01:47.392887+0000","last_peered":"2024-09-14T04:01:47.392887+0000","last_clean":"2024-09-14T04:01:47.392887+0000","last_became_active":"2024-09-14T03:55:58.252479+0000","last_became_peered":"2024-09-14T03:55:58.252479+0000","last_unstale":"2024-09-14T04:01:47.392887+0000","last_undegraded":"2024-09-14T04:01:47.392887+0000","last_fullsized":"2024-09-14T04:01:47.392887+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:46:04.357446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":171,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472629+0000","last_change":"2024-09-14T03:58:45.810289+0000","last_active":"2024-09-14T04:01:49.472629+0000","last_peered":"2024-09-14T04:01:49.472629+0000","last_clean":"2024-09-14T04:01:49.472629+0000","last_became_active":"2024-09-14T03:58:45.809938+0000","last_became_peered":"2024-09-14T03:58:45.809938+0000","last_unstale":"2024-09-14T04:01:49.472629+0000","last_undegraded":"2024-09-14T04:01:49.472629+0000","last_fullsized":"2024-09-14T04:01:49.472629+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:06:18.534582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366607+0000","last_change":"2024-09-14T03:52:28.182347+0000","last_active":"2024-09-14T04:01:49.366607+0000","last_peered":"2024-09-14T04:01:49.366607+0000","last_clean":"2024-09-14T04:01:49.366607+0000","last_became_active":"2024-09-14T03:52:28.181624+0000","last_became_peered":"2024-09-14T03:52:28.181624+0000","last_unstale":"2024-09-14T04:01:49.366607+0000","last_undegraded":"2024-09-14T04:01:49.366607+0000","last_fullsized":"2024-09-14T04:01:49.366607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:18.032619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392932+0000","last_change":"2024-09-14T03:58:45.793294+0000","last_active":"2024-09-14T04:01:47.392932+0000","last_peered":"2024-09-14T04:01:47.392932+0000","last_clean":"2024-09-14T04:01:47.392932+0000","last_became_active":"2024-09-14T03:58:45.793009+0000","last_became_peered":"2024-09-14T03:58:45.793009+0000","last_unstale":"2024-09-14T04:01:47.392932+0000","last_undegraded":"2024-09-14T04:01:47.392932+0000","last_fullsized":"2024-09-14T04:01:47.392932+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:46:00.206467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035459+0000","last_change":"2024-09-14T03:54:09.084402+0000","last_active":"2024-09-14T04:01:47.035459+0000","last_peered":"2024-09-14T04:01:47.035459+0000","last_clean":"2024-09-14T04:01:47.035459+0000","last_became_active":"2024-09-14T03:54:09.084076+0000","last_became_peered":"2024-09-14T03:54:09.084076+0000","last_unstale":"2024-09-14T04:01:47.035459+0000","last_undegraded":"2024-09-14T04:01:47.035459+0000","last_fullsized":"2024-09-14T04:01:47.035459+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:49:51.780807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392790+0000","last_change":"2024-09-14T03:58:45.793059+0000","last_active":"2024-09-14T04:01:47.392790+0000","last_peered":"2024-09-14T04:01:47.392790+0000","last_clean":"2024-09-14T04:01:47.392790+0000","last_became_active":"2024-09-14T03:58:45.792703+0000","last_became_peered":"2024-09-14T03:58:45.792703+0000","last_unstale":"2024-09-14T04:01:47.392790+0000","last_undegraded":"2024-09-14T04:01:47.392790+0000","last_fullsized":"2024-09-14T04:01:47.392790+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:01:17.321430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":102,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035534+0000","last_change":"2024-09-14T03:43:11.278982+0000","last_active":"2024-09-14T04:01:47.035534+0000","last_peered":"2024-09-14T04:01:47.035534+0000","last_clean":"2024-09-14T04:01:47.035534+0000","last_became_active":"2024-09-14T03:43:11.277577+0000","last_became_peered":"2024-09-14T03:43:11.277577+0000","last_unstale":"2024-09-14T04:01:47.035534+0000","last_undegraded":"2024-09-14T04:01:47.035534+0000","last_fullsized":"2024-09-14T04:01:47.035534+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:51.445273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473059+0000","last_change":"2024-09-14T03:58:45.790089+0000","last_active":"2024-09-14T04:01:49.473059+0000","last_peered":"2024-09-14T04:01:49.473059+0000","last_clean":"2024-09-14T04:01:49.473059+0000","last_became_active":"2024-09-14T03:58:45.789707+0000","last_became_peered":"2024-09-14T03:58:45.789707+0000","last_unstale":"2024-09-14T04:01:49.473059+0000","last_undegraded":"2024-09-14T04:01:49.473059+0000","last_fullsized":"2024-09-14T04:01:49.473059+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:03:26.492725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":120,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473285+0000","last_change":"2024-09-14T03:58:45.784441+0000","last_active":"2024-09-14T04:01:49.473285+0000","last_peered":"2024-09-14T04:01:49.473285+0000","last_clean":"2024-09-14T04:01:49.473285+0000","last_became_active":"2024-09-14T03:58:45.783913+0000","last_became_peered":"2024-09-14T03:58:45.783913+0000","last_unstale":"2024-09-14T04:01:49.473285+0000","last_undegraded":"2024-09-14T04:01:49.473285+0000","last_fullsized":"2024-09-14T04:01:49.473285+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:16:11.643108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392630+0000","last_change":"2024-09-14T03:54:09.084713+0000","last_active":"2024-09-14T04:01:47.392630+0000","last_peered":"2024-09-14T04:01:47.392630+0000","last_clean":"2024-09-14T04:01:47.392630+0000","last_became_active":"2024-09-14T03:54:09.082369+0000","last_became_peered":"2024-09-14T03:54:09.082369+0000","last_unstale":"2024-09-14T04:01:47.392630+0000","last_undegraded":"2024-09-14T04:01:47.392630+0000","last_fullsized":"2024-09-14T04:01:47.392630+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:41:39.305100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366512+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:01:49.366512+0000","last_peered":"2024-09-14T04:01:49.366512+0000","last_clean":"2024-09-14T04:01:49.366512+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:01:49.366512+0000","last_undegraded":"2024-09-14T04:01:49.366512+0000","last_fullsized":"2024-09-14T04:01:49.366512+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430467,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600536,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93144616,"statfs":{"total":95995035648,"available":95380086784,"internally_reserved":0,"allocated":2367488,"data_stored":1614300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626696,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299660,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445492,"statfs":{"total":95995035648,"available":95688183808,"internally_reserved":0,"allocated":2355200,"data_stored":1613858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822924,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911160,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":909438,"kb_avail":92833992,"statfs":{"total":95995035648,"available":95062007808,"internally_reserved":0,"allocated":1748992,"data_stored":1019001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931264856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019152,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599944,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93145208,"statfs":{"total":95995035648,"available":95380692992,"internally_reserved":0,"allocated":1765376,"data_stored":1023547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552305,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":201144,"kb_used_data":1708,"kb_used_omap":4,"kb_used_meta":199419,"kb_avail":93544008,"statfs":{"total":95995035648,"available":95789064192,"internally_reserved":0,"allocated":1748992,"data_stored":1019082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":204205501},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435346,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212852,"kb_used_data":1704,"kb_used_omap":6,"kb_used_meta":211129,"kb_avail":93532300,"statfs":{"total":95995035648,"available":95777075200,"internally_reserved":0,"allocated":1744896,"data_stored":1018134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":216196557},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827816,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327416,"kb_used_data":1708,"kb_used_omap":5,"kb_used_meta":325690,"kb_avail":93417736,"statfs":{"total":95995035648,"available":95659761664,"internally_reserved":0,"allocated":1748992,"data_stored":1019011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":333506799},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383804,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392700,"kb_used_data":2288,"kb_used_omap":5,"kb_used_meta":390394,"kb_avail":93352452,"statfs":{"total":95995035648,"available":95592910848,"internally_reserved":0,"allocated":2342912,"data_stored":1609653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":399763516},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:02:42.111 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-14T04:02:42.111 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-14T04:02:42.111 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-14T04:02:42.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-14T04:02:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:43 smithi160 ceph-mon[88407]: pgmap v493: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:43.914 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:43 smithi114 ceph-mon[103029]: pgmap v493: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:43.914 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:43 smithi114 ceph-mon[106969]: pgmap v493: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:44.354 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:02:44.354 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-14T04:02:45.057 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:44 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/942313594' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-14T04:02:45.057 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:44 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/942313594' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-14T04:02:45.081 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-14T04:02:45.081 INFO:teuthology.run_tasks:Running task print... 2024-09-14T04:02:45.093 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-09-14T04:02:45.094 INFO:teuthology.run_tasks:Running task parallel... 2024-09-14T04:02:45.135 INFO:teuthology.task.parallel:starting parallel... 2024-09-14T04:02:45.136 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-14T04:02:45.136 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-09-14T04:02:45.138 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-14T04:02:45.139 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': 'a2782315255a3e7d5f6cdc9eb35785187ff6ddf8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a2782315255a3e7d5f6cdc9eb35785187ff6ddf8'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '1c941a66c1b420259b8fc5ac19e792dbbf60aded'}} 2024-09-14T04:02:45.139 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-14T04:02:45.139 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-14T04:02:45.139 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-14T04:02:45.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:02:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:44 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/942313594' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-14T04:02:45.383 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:45.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:45 smithi114 ceph-mon[103029]: pgmap v494: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:45.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:45 smithi114 ceph-mon[106969]: pgmap v494: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:45 smithi160 ceph-mon[88407]: pgmap v494: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:47.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:02:47.299 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:02:47.938 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:47 smithi114 ceph-mon[103029]: pgmap v495: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:47.938 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:47 smithi114 ceph-mon[103029]: from='client.35610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:47.938 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:47 smithi114 ceph-mon[106969]: pgmap v495: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:47.938 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:47 smithi114 ceph-mon[106969]: from='client.35610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:47.962 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-14T04:02:46.593382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3545412,"kb_used_data":15452,"kb_used_omap":29,"kb_used_meta":3529826,"kb_avail":746415804,"statfs":{"total":767960285184,"available":764329783296,"internally_reserved":0,"allocated":15822848,"data_stored":9936586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30187,"internal_metadata":3614542357},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003073"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472139+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:01:49.472139+0000","last_peered":"2024-09-14T04:01:49.472139+0000","last_clean":"2024-09-14T04:01:49.472139+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:01:49.472139+0000","last_undegraded":"2024-09-14T04:01:49.472139+0000","last_fullsized":"2024-09-14T04:01:49.472139+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044498+0000","last_change":"2024-09-14T03:55:58.245884+0000","last_active":"2024-09-14T04:01:47.044498+0000","last_peered":"2024-09-14T04:01:47.044498+0000","last_clean":"2024-09-14T04:01:47.044498+0000","last_became_active":"2024-09-14T03:55:58.245547+0000","last_became_peered":"2024-09-14T03:55:58.245547+0000","last_unstale":"2024-09-14T04:01:47.044498+0000","last_undegraded":"2024-09-14T04:01:47.044498+0000","last_fullsized":"2024-09-14T04:01:47.044498+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:30:10.818639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035523+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:01:47.035523+0000","last_peered":"2024-09-14T04:01:47.035523+0000","last_clean":"2024-09-14T04:01:47.035523+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:01:47.035523+0000","last_undegraded":"2024-09-14T04:01:47.035523+0000","last_fullsized":"2024-09-14T04:01:47.035523+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366316+0000","last_change":"2024-09-14T03:58:45.794738+0000","last_active":"2024-09-14T04:01:49.366316+0000","last_peered":"2024-09-14T04:01:49.366316+0000","last_clean":"2024-09-14T04:01:49.366316+0000","last_became_active":"2024-09-14T03:58:45.794111+0000","last_became_peered":"2024-09-14T03:58:45.794111+0000","last_unstale":"2024-09-14T04:01:49.366316+0000","last_undegraded":"2024-09-14T04:01:49.366316+0000","last_fullsized":"2024-09-14T04:01:49.366316+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:12:49.686765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.371005+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:01:49.371005+0000","last_peered":"2024-09-14T04:01:49.371005+0000","last_clean":"2024-09-14T04:01:49.371005+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:01:49.371005+0000","last_undegraded":"2024-09-14T04:01:49.371005+0000","last_fullsized":"2024-09-14T04:01:49.371005+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044358+0000","last_change":"2024-09-14T03:43:11.269452+0000","last_active":"2024-09-14T04:01:47.044358+0000","last_peered":"2024-09-14T04:01:47.044358+0000","last_clean":"2024-09-14T04:01:47.044358+0000","last_became_active":"2024-09-14T03:43:11.269185+0000","last_became_peered":"2024-09-14T03:43:11.269185+0000","last_unstale":"2024-09-14T04:01:47.044358+0000","last_undegraded":"2024-09-14T04:01:47.044358+0000","last_fullsized":"2024-09-14T04:01:47.044358+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:53:12.602272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473429+0000","last_change":"2024-09-14T03:58:45.819617+0000","last_active":"2024-09-14T04:01:49.473429+0000","last_peered":"2024-09-14T04:01:49.473429+0000","last_clean":"2024-09-14T04:01:49.473429+0000","last_became_active":"2024-09-14T03:58:45.819034+0000","last_became_peered":"2024-09-14T03:58:45.819034+0000","last_unstale":"2024-09-14T04:01:49.473429+0000","last_undegraded":"2024-09-14T04:01:49.473429+0000","last_fullsized":"2024-09-14T04:01:49.473429+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:12.478061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":125,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040626+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:01:47.040626+0000","last_peered":"2024-09-14T04:01:47.040626+0000","last_clean":"2024-09-14T04:01:47.040626+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:01:47.040626+0000","last_undegraded":"2024-09-14T04:01:47.040626+0000","last_fullsized":"2024-09-14T04:01:47.040626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044837+0000","last_change":"2024-09-14T03:52:28.189069+0000","last_active":"2024-09-14T04:01:47.044837+0000","last_peered":"2024-09-14T04:01:47.044837+0000","last_clean":"2024-09-14T04:01:47.044837+0000","last_became_active":"2024-09-14T03:52:28.188421+0000","last_became_peered":"2024-09-14T03:52:28.188421+0000","last_unstale":"2024-09-14T04:01:47.044837+0000","last_undegraded":"2024-09-14T04:01:47.044837+0000","last_fullsized":"2024-09-14T04:01:47.044837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:20:09.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035248+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:01:47.035248+0000","last_peered":"2024-09-14T04:01:47.035248+0000","last_clean":"2024-09-14T04:01:47.035248+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:01:47.035248+0000","last_undegraded":"2024-09-14T04:01:47.035248+0000","last_fullsized":"2024-09-14T04:01:47.035248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044841+0000","last_change":"2024-09-14T03:54:09.079772+0000","last_active":"2024-09-14T04:01:47.044841+0000","last_peered":"2024-09-14T04:01:47.044841+0000","last_clean":"2024-09-14T04:01:47.044841+0000","last_became_active":"2024-09-14T03:54:09.079323+0000","last_became_peered":"2024-09-14T03:54:09.079323+0000","last_unstale":"2024-09-14T04:01:47.044841+0000","last_undegraded":"2024-09-14T04:01:47.044841+0000","last_fullsized":"2024-09-14T04:01:47.044841+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:59:58.328220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392726+0000","last_change":"2024-09-14T03:58:45.793252+0000","last_active":"2024-09-14T04:01:47.392726+0000","last_peered":"2024-09-14T04:01:47.392726+0000","last_clean":"2024-09-14T04:01:47.392726+0000","last_became_active":"2024-09-14T03:58:45.792917+0000","last_became_peered":"2024-09-14T03:58:45.792917+0000","last_unstale":"2024-09-14T04:01:47.392726+0000","last_undegraded":"2024-09-14T04:01:47.392726+0000","last_fullsized":"2024-09-14T04:01:47.392726+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:11.399204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044846+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:01:47.044846+0000","last_peered":"2024-09-14T04:01:47.044846+0000","last_clean":"2024-09-14T04:01:47.044846+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:01:47.044846+0000","last_undegraded":"2024-09-14T04:01:47.044846+0000","last_fullsized":"2024-09-14T04:01:47.044846+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040465+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:01:47.040465+0000","last_peered":"2024-09-14T04:01:47.040465+0000","last_clean":"2024-09-14T04:01:47.040465+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:01:47.040465+0000","last_undegraded":"2024-09-14T04:01:47.040465+0000","last_fullsized":"2024-09-14T04:01:47.040465+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035430+0000","last_change":"2024-09-14T03:58:45.810163+0000","last_active":"2024-09-14T04:01:47.035430+0000","last_peered":"2024-09-14T04:01:47.035430+0000","last_clean":"2024-09-14T04:01:47.035430+0000","last_became_active":"2024-09-14T03:58:45.809802+0000","last_became_peered":"2024-09-14T03:58:45.809802+0000","last_unstale":"2024-09-14T04:01:47.035430+0000","last_undegraded":"2024-09-14T04:01:47.035430+0000","last_fullsized":"2024-09-14T04:01:47.035430+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:40:33.911609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366386+0000","last_change":"2024-09-14T03:54:09.079995+0000","last_active":"2024-09-14T04:01:49.366386+0000","last_peered":"2024-09-14T04:01:49.366386+0000","last_clean":"2024-09-14T04:01:49.366386+0000","last_became_active":"2024-09-14T03:54:09.079686+0000","last_became_peered":"2024-09-14T03:54:09.079686+0000","last_unstale":"2024-09-14T04:01:49.366386+0000","last_undegraded":"2024-09-14T04:01:49.366386+0000","last_fullsized":"2024-09-14T04:01:49.366386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:15:31.274947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":134,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472859+0000","last_change":"2024-09-14T03:58:45.788412+0000","last_active":"2024-09-14T04:01:49.472859+0000","last_peered":"2024-09-14T04:01:49.472859+0000","last_clean":"2024-09-14T04:01:49.472859+0000","last_became_active":"2024-09-14T03:58:45.788078+0000","last_became_peered":"2024-09-14T03:58:45.788078+0000","last_unstale":"2024-09-14T04:01:49.472859+0000","last_undegraded":"2024-09-14T04:01:49.472859+0000","last_fullsized":"2024-09-14T04:01:49.472859+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:49:09.289398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392964+0000","last_change":"2024-09-14T03:58:45.812662+0000","last_active":"2024-09-14T04:01:47.392964+0000","last_peered":"2024-09-14T04:01:47.392964+0000","last_clean":"2024-09-14T04:01:47.392964+0000","last_became_active":"2024-09-14T03:58:45.812338+0000","last_became_peered":"2024-09-14T03:58:45.812338+0000","last_unstale":"2024-09-14T04:01:47.392964+0000","last_undegraded":"2024-09-14T04:01:47.392964+0000","last_fullsized":"2024-09-14T04:01:47.392964+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:35:35.258171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040574+0000","last_change":"2024-09-14T03:54:09.089856+0000","last_active":"2024-09-14T04:01:47.040574+0000","last_peered":"2024-09-14T04:01:47.040574+0000","last_clean":"2024-09-14T04:01:47.040574+0000","last_became_active":"2024-09-14T03:54:09.089559+0000","last_became_peered":"2024-09-14T03:54:09.089559+0000","last_unstale":"2024-09-14T04:01:47.040574+0000","last_undegraded":"2024-09-14T04:01:47.040574+0000","last_fullsized":"2024-09-14T04:01:47.040574+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:11:49.651236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393207+0000","last_change":"2024-09-14T03:55:58.249703+0000","last_active":"2024-09-14T04:01:47.393207+0000","last_peered":"2024-09-14T04:01:47.393207+0000","last_clean":"2024-09-14T04:01:47.393207+0000","last_became_active":"2024-09-14T03:55:58.249465+0000","last_became_peered":"2024-09-14T03:55:58.249465+0000","last_unstale":"2024-09-14T04:01:47.393207+0000","last_undegraded":"2024-09-14T04:01:47.393207+0000","last_fullsized":"2024-09-14T04:01:47.393207+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:03.743246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392875+0000","last_change":"2024-09-14T03:55:58.246315+0000","last_active":"2024-09-14T04:01:47.392875+0000","last_peered":"2024-09-14T04:01:47.392875+0000","last_clean":"2024-09-14T04:01:47.392875+0000","last_became_active":"2024-09-14T03:55:58.245997+0000","last_became_peered":"2024-09-14T03:55:58.245997+0000","last_unstale":"2024-09-14T04:01:47.392875+0000","last_undegraded":"2024-09-14T04:01:47.392875+0000","last_fullsized":"2024-09-14T04:01:47.392875+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:11:13.312397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393206+0000","last_change":"2024-09-14T03:54:09.082688+0000","last_active":"2024-09-14T04:01:47.393206+0000","last_peered":"2024-09-14T04:01:47.393206+0000","last_clean":"2024-09-14T04:01:47.393206+0000","last_became_active":"2024-09-14T03:54:09.082402+0000","last_became_peered":"2024-09-14T03:54:09.082402+0000","last_unstale":"2024-09-14T04:01:47.393206+0000","last_undegraded":"2024-09-14T04:01:47.393206+0000","last_fullsized":"2024-09-14T04:01:47.393206+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:28:53.024974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":107,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473967+0000","last_change":"2024-09-14T03:58:45.822672+0000","last_active":"2024-09-14T04:01:49.473967+0000","last_peered":"2024-09-14T04:01:49.473967+0000","last_clean":"2024-09-14T04:01:49.473967+0000","last_became_active":"2024-09-14T03:58:45.822047+0000","last_became_peered":"2024-09-14T03:58:45.822047+0000","last_unstale":"2024-09-14T04:01:49.473967+0000","last_undegraded":"2024-09-14T04:01:49.473967+0000","last_fullsized":"2024-09-14T04:01:49.473967+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:29:49.341047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093884+0000","last_change":"2024-09-14T03:58:45.793497+0000","last_active":"2024-09-14T04:01:49.093884+0000","last_peered":"2024-09-14T04:01:49.093884+0000","last_clean":"2024-09-14T04:01:49.093884+0000","last_became_active":"2024-09-14T03:58:45.793169+0000","last_became_peered":"2024-09-14T03:58:45.793169+0000","last_unstale":"2024-09-14T04:01:49.093884+0000","last_undegraded":"2024-09-14T04:01:49.093884+0000","last_fullsized":"2024-09-14T04:01:49.093884+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:11:06.166164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":233,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044476+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:01:47.044476+0000","last_peered":"2024-09-14T04:01:47.044476+0000","last_clean":"2024-09-14T04:01:47.044476+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:01:47.044476+0000","last_undegraded":"2024-09-14T04:01:47.044476+0000","last_fullsized":"2024-09-14T04:01:47.044476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":130,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035317+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:01:47.035317+0000","last_peered":"2024-09-14T04:01:47.035317+0000","last_clean":"2024-09-14T04:01:47.035317+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:01:47.035317+0000","last_undegraded":"2024-09-14T04:01:47.035317+0000","last_fullsized":"2024-09-14T04:01:47.035317+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":94,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472088+0000","last_change":"2024-09-14T03:58:45.782514+0000","last_active":"2024-09-14T04:01:49.472088+0000","last_peered":"2024-09-14T04:01:49.472088+0000","last_clean":"2024-09-14T04:01:49.472088+0000","last_became_active":"2024-09-14T03:58:45.782196+0000","last_became_peered":"2024-09-14T03:58:45.782196+0000","last_unstale":"2024-09-14T04:01:49.472088+0000","last_undegraded":"2024-09-14T04:01:49.472088+0000","last_fullsized":"2024-09-14T04:01:49.472088+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:59:44.614261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040892+0000","last_change":"2024-09-14T03:43:11.909945+0000","last_active":"2024-09-14T04:01:47.040892+0000","last_peered":"2024-09-14T04:01:47.040892+0000","last_clean":"2024-09-14T04:01:47.040892+0000","last_became_active":"2024-09-14T03:43:11.909666+0000","last_became_peered":"2024-09-14T03:43:11.909666+0000","last_unstale":"2024-09-14T04:01:47.040892+0000","last_undegraded":"2024-09-14T04:01:47.040892+0000","last_fullsized":"2024-09-14T04:01:47.040892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:13:19.629641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":145,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044463+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:01:47.044463+0000","last_peered":"2024-09-14T04:01:47.044463+0000","last_clean":"2024-09-14T04:01:47.044463+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:01:47.044463+0000","last_undegraded":"2024-09-14T04:01:47.044463+0000","last_fullsized":"2024-09-14T04:01:47.044463+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366618+0000","last_change":"2024-09-14T03:58:45.795127+0000","last_active":"2024-09-14T04:01:49.366618+0000","last_peered":"2024-09-14T04:01:49.366618+0000","last_clean":"2024-09-14T04:01:49.366618+0000","last_became_active":"2024-09-14T03:58:45.794831+0000","last_became_peered":"2024-09-14T03:58:45.794831+0000","last_unstale":"2024-09-14T04:01:49.366618+0000","last_undegraded":"2024-09-14T04:01:49.366618+0000","last_fullsized":"2024-09-14T04:01:49.366618+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:22:18.983609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.370773+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:01:49.370773+0000","last_peered":"2024-09-14T04:01:49.370773+0000","last_clean":"2024-09-14T04:01:49.370773+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:01:49.370773+0000","last_undegraded":"2024-09-14T04:01:49.370773+0000","last_fullsized":"2024-09-14T04:01:49.370773+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":168,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471920+0000","last_change":"2024-09-14T03:58:45.791394+0000","last_active":"2024-09-14T04:01:49.471920+0000","last_peered":"2024-09-14T04:01:49.471920+0000","last_clean":"2024-09-14T04:01:49.471920+0000","last_became_active":"2024-09-14T03:58:45.790907+0000","last_became_peered":"2024-09-14T03:58:45.790907+0000","last_unstale":"2024-09-14T04:01:49.471920+0000","last_undegraded":"2024-09-14T04:01:49.471920+0000","last_fullsized":"2024-09-14T04:01:49.471920+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:07.434141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":143,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093019+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:01:49.093019+0000","last_peered":"2024-09-14T04:01:49.093019+0000","last_clean":"2024-09-14T04:01:49.093019+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:01:49.093019+0000","last_undegraded":"2024-09-14T04:01:49.093019+0000","last_fullsized":"2024-09-14T04:01:49.093019+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":168,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093834+0000","last_change":"2024-09-14T03:54:09.087551+0000","last_active":"2024-09-14T04:01:49.093834+0000","last_peered":"2024-09-14T04:01:49.093834+0000","last_clean":"2024-09-14T04:01:49.093834+0000","last_became_active":"2024-09-14T03:54:09.086608+0000","last_became_peered":"2024-09-14T03:54:09.086608+0000","last_unstale":"2024-09-14T04:01:49.093834+0000","last_undegraded":"2024-09-14T04:01:49.093834+0000","last_fullsized":"2024-09-14T04:01:49.093834+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:38:14.920066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035236+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:01:47.035236+0000","last_peered":"2024-09-14T04:01:47.035236+0000","last_clean":"2024-09-14T04:01:47.035236+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:01:47.035236+0000","last_undegraded":"2024-09-14T04:01:47.035236+0000","last_fullsized":"2024-09-14T04:01:47.035236+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393034+0000","last_change":"2024-09-14T03:52:28.187042+0000","last_active":"2024-09-14T04:01:47.393034+0000","last_peered":"2024-09-14T04:01:47.393034+0000","last_clean":"2024-09-14T04:01:47.393034+0000","last_became_active":"2024-09-14T03:52:28.186714+0000","last_became_peered":"2024-09-14T03:52:28.186714+0000","last_unstale":"2024-09-14T04:01:47.393034+0000","last_undegraded":"2024-09-14T04:01:47.393034+0000","last_fullsized":"2024-09-14T04:01:47.393034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:50:57.007224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":154,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093063+0000","last_change":"2024-09-14T03:58:45.812745+0000","last_active":"2024-09-14T04:01:49.093063+0000","last_peered":"2024-09-14T04:01:49.093063+0000","last_clean":"2024-09-14T04:01:49.093063+0000","last_became_active":"2024-09-14T03:58:45.812385+0000","last_became_peered":"2024-09-14T03:58:45.812385+0000","last_unstale":"2024-09-14T04:01:49.093063+0000","last_undegraded":"2024-09-14T04:01:49.093063+0000","last_fullsized":"2024-09-14T04:01:49.093063+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:18:40.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035621+0000","last_change":"2024-09-14T03:58:45.788256+0000","last_active":"2024-09-14T04:01:47.035621+0000","last_peered":"2024-09-14T04:01:47.035621+0000","last_clean":"2024-09-14T04:01:47.035621+0000","last_became_active":"2024-09-14T03:58:45.788052+0000","last_became_peered":"2024-09-14T03:58:45.788052+0000","last_unstale":"2024-09-14T04:01:47.035621+0000","last_undegraded":"2024-09-14T04:01:47.035621+0000","last_fullsized":"2024-09-14T04:01:47.035621+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:38:19.891512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093135+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:01:49.093135+0000","last_peered":"2024-09-14T04:01:49.093135+0000","last_clean":"2024-09-14T04:01:49.093135+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:01:49.093135+0000","last_undegraded":"2024-09-14T04:01:49.093135+0000","last_fullsized":"2024-09-14T04:01:49.093135+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093546+0000","last_change":"2024-09-14T03:54:09.089016+0000","last_active":"2024-09-14T04:01:49.093546+0000","last_peered":"2024-09-14T04:01:49.093546+0000","last_clean":"2024-09-14T04:01:49.093546+0000","last_became_active":"2024-09-14T03:54:09.087653+0000","last_became_peered":"2024-09-14T03:54:09.087653+0000","last_unstale":"2024-09-14T04:01:49.093546+0000","last_undegraded":"2024-09-14T04:01:49.093546+0000","last_fullsized":"2024-09-14T04:01:49.093546+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:07:46.505921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":143,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093259+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:01:49.093259+0000","last_peered":"2024-09-14T04:01:49.093259+0000","last_clean":"2024-09-14T04:01:49.093259+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:01:49.093259+0000","last_undegraded":"2024-09-14T04:01:49.093259+0000","last_fullsized":"2024-09-14T04:01:49.093259+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":128,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044846+0000","last_change":"2024-09-14T03:52:28.190501+0000","last_active":"2024-09-14T04:01:47.044846+0000","last_peered":"2024-09-14T04:01:47.044846+0000","last_clean":"2024-09-14T04:01:47.044846+0000","last_became_active":"2024-09-14T03:52:28.190050+0000","last_became_peered":"2024-09-14T03:52:28.190050+0000","last_unstale":"2024-09-14T04:01:47.044846+0000","last_undegraded":"2024-09-14T04:01:47.044846+0000","last_fullsized":"2024-09-14T04:01:47.044846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:58:22.557305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":166,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366465+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:01:49.366465+0000","last_peered":"2024-09-14T04:01:49.366465+0000","last_clean":"2024-09-14T04:01:49.366465+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:01:49.366465+0000","last_undegraded":"2024-09-14T04:01:49.366465+0000","last_fullsized":"2024-09-14T04:01:49.366465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":134,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473560+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:01:49.473560+0000","last_peered":"2024-09-14T04:01:49.473560+0000","last_clean":"2024-09-14T04:01:49.473560+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:01:49.473560+0000","last_undegraded":"2024-09-14T04:01:49.473560+0000","last_fullsized":"2024-09-14T04:01:49.473560+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093627+0000","last_change":"2024-09-14T03:54:09.082573+0000","last_active":"2024-09-14T04:01:49.093627+0000","last_peered":"2024-09-14T04:01:49.093627+0000","last_clean":"2024-09-14T04:01:49.093627+0000","last_became_active":"2024-09-14T03:54:09.082266+0000","last_became_peered":"2024-09-14T03:54:09.082266+0000","last_unstale":"2024-09-14T04:01:49.093627+0000","last_undegraded":"2024-09-14T04:01:49.093627+0000","last_fullsized":"2024-09-14T04:01:49.093627+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:30:24.244999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472388+0000","last_change":"2024-09-14T03:58:45.822348+0000","last_active":"2024-09-14T04:01:49.472388+0000","last_peered":"2024-09-14T04:01:49.472388+0000","last_clean":"2024-09-14T04:01:49.472388+0000","last_became_active":"2024-09-14T03:58:45.821621+0000","last_became_peered":"2024-09-14T03:58:45.821621+0000","last_unstale":"2024-09-14T04:01:49.472388+0000","last_undegraded":"2024-09-14T04:01:49.472388+0000","last_fullsized":"2024-09-14T04:01:49.472388+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:02:12.477742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.371059+0000","last_change":"2024-09-14T03:54:09.081554+0000","last_active":"2024-09-14T04:01:49.371059+0000","last_peered":"2024-09-14T04:01:49.371059+0000","last_clean":"2024-09-14T04:01:49.371059+0000","last_became_active":"2024-09-14T03:54:09.081158+0000","last_became_peered":"2024-09-14T03:54:09.081158+0000","last_unstale":"2024-09-14T04:01:49.371059+0000","last_undegraded":"2024-09-14T04:01:49.371059+0000","last_fullsized":"2024-09-14T04:01:49.371059+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:24:43.027300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":129,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040935+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:01:47.040935+0000","last_peered":"2024-09-14T04:01:47.040935+0000","last_clean":"2024-09-14T04:01:47.040935+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:01:47.040935+0000","last_undegraded":"2024-09-14T04:01:47.040935+0000","last_fullsized":"2024-09-14T04:01:47.040935+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":98,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392823+0000","last_change":"2024-09-14T03:43:11.285521+0000","last_active":"2024-09-14T04:01:47.392823+0000","last_peered":"2024-09-14T04:01:47.392823+0000","last_clean":"2024-09-14T04:01:47.392823+0000","last_became_active":"2024-09-14T03:43:11.284778+0000","last_became_peered":"2024-09-14T03:43:11.284778+0000","last_unstale":"2024-09-14T04:01:47.392823+0000","last_undegraded":"2024-09-14T04:01:47.392823+0000","last_fullsized":"2024-09-14T04:01:47.392823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:51:32.744643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":108,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392975+0000","last_change":"2024-09-14T03:43:52.490179+0000","last_active":"2024-09-14T04:01:47.392975+0000","last_peered":"2024-09-14T04:01:47.392975+0000","last_clean":"2024-09-14T04:01:47.392975+0000","last_became_active":"2024-09-14T03:43:52.489786+0000","last_became_peered":"2024-09-14T03:43:52.489786+0000","last_unstale":"2024-09-14T04:01:47.392975+0000","last_undegraded":"2024-09-14T04:01:47.392975+0000","last_fullsized":"2024-09-14T04:01:47.392975+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:55.590247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":121,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393060+0000","last_change":"2024-09-14T03:43:11.274139+0000","last_active":"2024-09-14T04:01:47.393060+0000","last_peered":"2024-09-14T04:01:47.393060+0000","last_clean":"2024-09-14T04:01:47.393060+0000","last_became_active":"2024-09-14T03:43:11.271423+0000","last_became_peered":"2024-09-14T03:43:11.271423+0000","last_unstale":"2024-09-14T04:01:47.393060+0000","last_undegraded":"2024-09-14T04:01:47.393060+0000","last_fullsized":"2024-09-14T04:01:47.393060+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:52.045590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.092936+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:01:49.092936+0000","last_peered":"2024-09-14T04:01:49.092936+0000","last_clean":"2024-09-14T04:01:49.092936+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:01:49.092936+0000","last_undegraded":"2024-09-14T04:01:49.092936+0000","last_fullsized":"2024-09-14T04:01:49.092936+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044710+0000","last_change":"2024-09-14T03:58:45.793790+0000","last_active":"2024-09-14T04:01:47.044710+0000","last_peered":"2024-09-14T04:01:47.044710+0000","last_clean":"2024-09-14T04:01:47.044710+0000","last_became_active":"2024-09-14T03:58:45.793253+0000","last_became_peered":"2024-09-14T03:58:45.793253+0000","last_unstale":"2024-09-14T04:01:47.044710+0000","last_undegraded":"2024-09-14T04:01:47.044710+0000","last_fullsized":"2024-09-14T04:01:47.044710+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:44.988345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035444+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:01:47.035444+0000","last_peered":"2024-09-14T04:01:47.035444+0000","last_clean":"2024-09-14T04:01:47.035444+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:01:47.035444+0000","last_undegraded":"2024-09-14T04:01:47.035444+0000","last_fullsized":"2024-09-14T04:01:47.035444+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":94,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471884+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:01:49.471884+0000","last_peered":"2024-09-14T04:01:49.471884+0000","last_clean":"2024-09-14T04:01:49.471884+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:01:49.471884+0000","last_undegraded":"2024-09-14T04:01:49.471884+0000","last_fullsized":"2024-09-14T04:01:49.471884+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366797+0000","last_change":"2024-09-14T03:58:45.794557+0000","last_active":"2024-09-14T04:01:49.366797+0000","last_peered":"2024-09-14T04:01:49.366797+0000","last_clean":"2024-09-14T04:01:49.366797+0000","last_became_active":"2024-09-14T03:58:45.794173+0000","last_became_peered":"2024-09-14T03:58:45.794173+0000","last_unstale":"2024-09-14T04:01:49.366797+0000","last_undegraded":"2024-09-14T04:01:49.366797+0000","last_fullsized":"2024-09-14T04:01:49.366797+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:33:12.799351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471703+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:01:49.471703+0000","last_peered":"2024-09-14T04:01:49.471703+0000","last_clean":"2024-09-14T04:01:49.471703+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:01:49.471703+0000","last_undegraded":"2024-09-14T04:01:49.471703+0000","last_fullsized":"2024-09-14T04:01:49.471703+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040902+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:01:47.040902+0000","last_peered":"2024-09-14T04:01:47.040902+0000","last_clean":"2024-09-14T04:01:47.040902+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:01:47.040902+0000","last_undegraded":"2024-09-14T04:01:47.040902+0000","last_fullsized":"2024-09-14T04:01:47.040902+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":124,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472437+0000","last_change":"2024-09-14T03:58:45.822679+0000","last_active":"2024-09-14T04:01:49.472437+0000","last_peered":"2024-09-14T04:01:49.472437+0000","last_clean":"2024-09-14T04:01:49.472437+0000","last_became_active":"2024-09-14T03:58:45.822051+0000","last_became_peered":"2024-09-14T03:58:45.822051+0000","last_unstale":"2024-09-14T04:01:49.472437+0000","last_undegraded":"2024-09-14T04:01:49.472437+0000","last_fullsized":"2024-09-14T04:01:49.472437+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:30:52.022607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":178,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.370935+0000","last_change":"2024-09-14T03:58:45.795656+0000","last_active":"2024-09-14T04:01:49.370935+0000","last_peered":"2024-09-14T04:01:49.370935+0000","last_clean":"2024-09-14T04:01:49.370935+0000","last_became_active":"2024-09-14T03:58:45.795369+0000","last_became_peered":"2024-09-14T03:58:45.795369+0000","last_unstale":"2024-09-14T04:01:49.370935+0000","last_undegraded":"2024-09-14T04:01:49.370935+0000","last_fullsized":"2024-09-14T04:01:49.370935+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:45.896015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.092900+0000","last_change":"2024-09-14T03:58:45.793634+0000","last_active":"2024-09-14T04:01:49.092900+0000","last_peered":"2024-09-14T04:01:49.092900+0000","last_clean":"2024-09-14T04:01:49.092900+0000","last_became_active":"2024-09-14T03:58:45.793229+0000","last_became_peered":"2024-09-14T03:58:45.793229+0000","last_unstale":"2024-09-14T04:01:49.092900+0000","last_undegraded":"2024-09-14T04:01:49.092900+0000","last_fullsized":"2024-09-14T04:01:49.092900+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:31.431481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473119+0000","last_change":"2024-09-14T03:58:45.795790+0000","last_active":"2024-09-14T04:01:49.473119+0000","last_peered":"2024-09-14T04:01:49.473119+0000","last_clean":"2024-09-14T04:01:49.473119+0000","last_became_active":"2024-09-14T03:58:45.795468+0000","last_became_peered":"2024-09-14T03:58:45.795468+0000","last_unstale":"2024-09-14T04:01:49.473119+0000","last_undegraded":"2024-09-14T04:01:49.473119+0000","last_fullsized":"2024-09-14T04:01:49.473119+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:50:54.779571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392887+0000","last_change":"2024-09-14T03:55:58.252985+0000","last_active":"2024-09-14T04:01:47.392887+0000","last_peered":"2024-09-14T04:01:47.392887+0000","last_clean":"2024-09-14T04:01:47.392887+0000","last_became_active":"2024-09-14T03:55:58.252479+0000","last_became_peered":"2024-09-14T03:55:58.252479+0000","last_unstale":"2024-09-14T04:01:47.392887+0000","last_undegraded":"2024-09-14T04:01:47.392887+0000","last_fullsized":"2024-09-14T04:01:47.392887+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:46:04.357446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":171,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472629+0000","last_change":"2024-09-14T03:58:45.810289+0000","last_active":"2024-09-14T04:01:49.472629+0000","last_peered":"2024-09-14T04:01:49.472629+0000","last_clean":"2024-09-14T04:01:49.472629+0000","last_became_active":"2024-09-14T03:58:45.809938+0000","last_became_peered":"2024-09-14T03:58:45.809938+0000","last_unstale":"2024-09-14T04:01:49.472629+0000","last_undegraded":"2024-09-14T04:01:49.472629+0000","last_fullsized":"2024-09-14T04:01:49.472629+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:06:18.534582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366607+0000","last_change":"2024-09-14T03:52:28.182347+0000","last_active":"2024-09-14T04:01:49.366607+0000","last_peered":"2024-09-14T04:01:49.366607+0000","last_clean":"2024-09-14T04:01:49.366607+0000","last_became_active":"2024-09-14T03:52:28.181624+0000","last_became_peered":"2024-09-14T03:52:28.181624+0000","last_unstale":"2024-09-14T04:01:49.366607+0000","last_undegraded":"2024-09-14T04:01:49.366607+0000","last_fullsized":"2024-09-14T04:01:49.366607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:18.032619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392932+0000","last_change":"2024-09-14T03:58:45.793294+0000","last_active":"2024-09-14T04:01:47.392932+0000","last_peered":"2024-09-14T04:01:47.392932+0000","last_clean":"2024-09-14T04:01:47.392932+0000","last_became_active":"2024-09-14T03:58:45.793009+0000","last_became_peered":"2024-09-14T03:58:45.793009+0000","last_unstale":"2024-09-14T04:01:47.392932+0000","last_undegraded":"2024-09-14T04:01:47.392932+0000","last_fullsized":"2024-09-14T04:01:47.392932+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:46:00.206467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035459+0000","last_change":"2024-09-14T03:54:09.084402+0000","last_active":"2024-09-14T04:01:47.035459+0000","last_peered":"2024-09-14T04:01:47.035459+0000","last_clean":"2024-09-14T04:01:47.035459+0000","last_became_active":"2024-09-14T03:54:09.084076+0000","last_became_peered":"2024-09-14T03:54:09.084076+0000","last_unstale":"2024-09-14T04:01:47.035459+0000","last_undegraded":"2024-09-14T04:01:47.035459+0000","last_fullsized":"2024-09-14T04:01:47.035459+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:49:51.780807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392790+0000","last_change":"2024-09-14T03:58:45.793059+0000","last_active":"2024-09-14T04:01:47.392790+0000","last_peered":"2024-09-14T04:01:47.392790+0000","last_clean":"2024-09-14T04:01:47.392790+0000","last_became_active":"2024-09-14T03:58:45.792703+0000","last_became_peered":"2024-09-14T03:58:45.792703+0000","last_unstale":"2024-09-14T04:01:47.392790+0000","last_undegraded":"2024-09-14T04:01:47.392790+0000","last_fullsized":"2024-09-14T04:01:47.392790+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:01:17.321430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":102,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035534+0000","last_change":"2024-09-14T03:43:11.278982+0000","last_active":"2024-09-14T04:01:47.035534+0000","last_peered":"2024-09-14T04:01:47.035534+0000","last_clean":"2024-09-14T04:01:47.035534+0000","last_became_active":"2024-09-14T03:43:11.277577+0000","last_became_peered":"2024-09-14T03:43:11.277577+0000","last_unstale":"2024-09-14T04:01:47.035534+0000","last_undegraded":"2024-09-14T04:01:47.035534+0000","last_fullsized":"2024-09-14T04:01:47.035534+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:51.445273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473059+0000","last_change":"2024-09-14T03:58:45.790089+0000","last_active":"2024-09-14T04:01:49.473059+0000","last_peered":"2024-09-14T04:01:49.473059+0000","last_clean":"2024-09-14T04:01:49.473059+0000","last_became_active":"2024-09-14T03:58:45.789707+0000","last_became_peered":"2024-09-14T03:58:45.789707+0000","last_unstale":"2024-09-14T04:01:49.473059+0000","last_undegraded":"2024-09-14T04:01:49.473059+0000","last_fullsized":"2024-09-14T04:01:49.473059+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:03:26.492725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":121,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473333+0000","last_change":"2024-09-14T03:58:45.784441+0000","last_active":"2024-09-14T04:01:49.473333+0000","last_peered":"2024-09-14T04:01:49.473333+0000","last_clean":"2024-09-14T04:01:49.473333+0000","last_became_active":"2024-09-14T03:58:45.783913+0000","last_became_peered":"2024-09-14T03:58:45.783913+0000","last_unstale":"2024-09-14T04:01:49.473333+0000","last_undegraded":"2024-09-14T04:01:49.473333+0000","last_fullsized":"2024-09-14T04:01:49.473333+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:16:11.643108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392630+0000","last_change":"2024-09-14T03:54:09.084713+0000","last_active":"2024-09-14T04:01:47.392630+0000","last_peered":"2024-09-14T04:01:47.392630+0000","last_clean":"2024-09-14T04:01:47.392630+0000","last_became_active":"2024-09-14T03:54:09.082369+0000","last_became_peered":"2024-09-14T03:54:09.082369+0000","last_unstale":"2024-09-14T04:01:47.392630+0000","last_undegraded":"2024-09-14T04:01:47.392630+0000","last_fullsized":"2024-09-14T04:01:47.392630+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:41:39.305100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366512+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:01:49.366512+0000","last_peered":"2024-09-14T04:01:49.366512+0000","last_clean":"2024-09-14T04:01:49.366512+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:01:49.366512+0000","last_undegraded":"2024-09-14T04:01:49.366512+0000","last_fullsized":"2024-09-14T04:01:49.366512+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430468,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600536,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93144616,"statfs":{"total":95995035648,"available":95380086784,"internally_reserved":0,"allocated":2367488,"data_stored":1614300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626697,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299660,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445492,"statfs":{"total":95995035648,"available":95688183808,"internally_reserved":0,"allocated":2355200,"data_stored":1613858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822925,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911160,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":909438,"kb_avail":92833992,"statfs":{"total":95995035648,"available":95062007808,"internally_reserved":0,"allocated":1748992,"data_stored":1019001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931264856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019153,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599944,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93145208,"statfs":{"total":95995035648,"available":95380692992,"internally_reserved":0,"allocated":1765376,"data_stored":1023547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552306,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":201144,"kb_used_data":1708,"kb_used_omap":4,"kb_used_meta":199419,"kb_avail":93544008,"statfs":{"total":95995035648,"available":95789064192,"internally_reserved":0,"allocated":1748992,"data_stored":1019082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":204205501},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435348,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212852,"kb_used_data":1704,"kb_used_omap":6,"kb_used_meta":211129,"kb_avail":93532300,"statfs":{"total":95995035648,"available":95777075200,"internally_reserved":0,"allocated":1744896,"data_stored":1018134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":216196557},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827817,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327416,"kb_used_data":1708,"kb_used_omap":5,"kb_used_meta":325690,"kb_avail":93417736,"statfs":{"total":95995035648,"available":95659761664,"internally_reserved":0,"allocated":1748992,"data_stored":1019011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":333506799},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383805,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392700,"kb_used_data":2288,"kb_used_omap":5,"kb_used_meta":390394,"kb_avail":93352452,"statfs":{"total":95995035648,"available":95592910848,"internally_reserved":0,"allocated":2342912,"data_stored":1609653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":399763516},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:02:47.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:02:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:47 smithi160 ceph-mon[88407]: pgmap v495: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:47 smithi160 ceph-mon[88407]: from='client.35610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:48.209 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:50.026 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:02:50.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:49 smithi114 ceph-mon[103029]: pgmap v496: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:50.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:49 smithi114 ceph-mon[106969]: pgmap v496: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:50.029 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:02:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:49 smithi160 ceph-mon[88407]: pgmap v496: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:50.612 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-14T04:02:48.593935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3545412,"kb_used_data":15452,"kb_used_omap":29,"kb_used_meta":3529826,"kb_avail":746415804,"statfs":{"total":767960285184,"available":764329783296,"internally_reserved":0,"allocated":15822848,"data_stored":9936586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30187,"internal_metadata":3614542357},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472139+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:01:49.472139+0000","last_peered":"2024-09-14T04:01:49.472139+0000","last_clean":"2024-09-14T04:01:49.472139+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:01:49.472139+0000","last_undegraded":"2024-09-14T04:01:49.472139+0000","last_fullsized":"2024-09-14T04:01:49.472139+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044498+0000","last_change":"2024-09-14T03:55:58.245884+0000","last_active":"2024-09-14T04:01:47.044498+0000","last_peered":"2024-09-14T04:01:47.044498+0000","last_clean":"2024-09-14T04:01:47.044498+0000","last_became_active":"2024-09-14T03:55:58.245547+0000","last_became_peered":"2024-09-14T03:55:58.245547+0000","last_unstale":"2024-09-14T04:01:47.044498+0000","last_undegraded":"2024-09-14T04:01:47.044498+0000","last_fullsized":"2024-09-14T04:01:47.044498+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:30:10.818639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035523+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:01:47.035523+0000","last_peered":"2024-09-14T04:01:47.035523+0000","last_clean":"2024-09-14T04:01:47.035523+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:01:47.035523+0000","last_undegraded":"2024-09-14T04:01:47.035523+0000","last_fullsized":"2024-09-14T04:01:47.035523+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366316+0000","last_change":"2024-09-14T03:58:45.794738+0000","last_active":"2024-09-14T04:01:49.366316+0000","last_peered":"2024-09-14T04:01:49.366316+0000","last_clean":"2024-09-14T04:01:49.366316+0000","last_became_active":"2024-09-14T03:58:45.794111+0000","last_became_peered":"2024-09-14T03:58:45.794111+0000","last_unstale":"2024-09-14T04:01:49.366316+0000","last_undegraded":"2024-09-14T04:01:49.366316+0000","last_fullsized":"2024-09-14T04:01:49.366316+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:12:49.686765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.371005+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:01:49.371005+0000","last_peered":"2024-09-14T04:01:49.371005+0000","last_clean":"2024-09-14T04:01:49.371005+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:01:49.371005+0000","last_undegraded":"2024-09-14T04:01:49.371005+0000","last_fullsized":"2024-09-14T04:01:49.371005+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044358+0000","last_change":"2024-09-14T03:43:11.269452+0000","last_active":"2024-09-14T04:01:47.044358+0000","last_peered":"2024-09-14T04:01:47.044358+0000","last_clean":"2024-09-14T04:01:47.044358+0000","last_became_active":"2024-09-14T03:43:11.269185+0000","last_became_peered":"2024-09-14T03:43:11.269185+0000","last_unstale":"2024-09-14T04:01:47.044358+0000","last_undegraded":"2024-09-14T04:01:47.044358+0000","last_fullsized":"2024-09-14T04:01:47.044358+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:53:12.602272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473429+0000","last_change":"2024-09-14T03:58:45.819617+0000","last_active":"2024-09-14T04:01:49.473429+0000","last_peered":"2024-09-14T04:01:49.473429+0000","last_clean":"2024-09-14T04:01:49.473429+0000","last_became_active":"2024-09-14T03:58:45.819034+0000","last_became_peered":"2024-09-14T03:58:45.819034+0000","last_unstale":"2024-09-14T04:01:49.473429+0000","last_undegraded":"2024-09-14T04:01:49.473429+0000","last_fullsized":"2024-09-14T04:01:49.473429+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:12.478061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":125,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040626+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:01:47.040626+0000","last_peered":"2024-09-14T04:01:47.040626+0000","last_clean":"2024-09-14T04:01:47.040626+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:01:47.040626+0000","last_undegraded":"2024-09-14T04:01:47.040626+0000","last_fullsized":"2024-09-14T04:01:47.040626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044837+0000","last_change":"2024-09-14T03:52:28.189069+0000","last_active":"2024-09-14T04:01:47.044837+0000","last_peered":"2024-09-14T04:01:47.044837+0000","last_clean":"2024-09-14T04:01:47.044837+0000","last_became_active":"2024-09-14T03:52:28.188421+0000","last_became_peered":"2024-09-14T03:52:28.188421+0000","last_unstale":"2024-09-14T04:01:47.044837+0000","last_undegraded":"2024-09-14T04:01:47.044837+0000","last_fullsized":"2024-09-14T04:01:47.044837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:20:09.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035248+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:01:47.035248+0000","last_peered":"2024-09-14T04:01:47.035248+0000","last_clean":"2024-09-14T04:01:47.035248+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:01:47.035248+0000","last_undegraded":"2024-09-14T04:01:47.035248+0000","last_fullsized":"2024-09-14T04:01:47.035248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":118,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044841+0000","last_change":"2024-09-14T03:54:09.079772+0000","last_active":"2024-09-14T04:01:47.044841+0000","last_peered":"2024-09-14T04:01:47.044841+0000","last_clean":"2024-09-14T04:01:47.044841+0000","last_became_active":"2024-09-14T03:54:09.079323+0000","last_became_peered":"2024-09-14T03:54:09.079323+0000","last_unstale":"2024-09-14T04:01:47.044841+0000","last_undegraded":"2024-09-14T04:01:47.044841+0000","last_fullsized":"2024-09-14T04:01:47.044841+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:59:58.328220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392726+0000","last_change":"2024-09-14T03:58:45.793252+0000","last_active":"2024-09-14T04:01:47.392726+0000","last_peered":"2024-09-14T04:01:47.392726+0000","last_clean":"2024-09-14T04:01:47.392726+0000","last_became_active":"2024-09-14T03:58:45.792917+0000","last_became_peered":"2024-09-14T03:58:45.792917+0000","last_unstale":"2024-09-14T04:01:47.392726+0000","last_undegraded":"2024-09-14T04:01:47.392726+0000","last_fullsized":"2024-09-14T04:01:47.392726+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:11.399204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":100,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044846+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:01:47.044846+0000","last_peered":"2024-09-14T04:01:47.044846+0000","last_clean":"2024-09-14T04:01:47.044846+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:01:47.044846+0000","last_undegraded":"2024-09-14T04:01:47.044846+0000","last_fullsized":"2024-09-14T04:01:47.044846+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040465+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:01:47.040465+0000","last_peered":"2024-09-14T04:01:47.040465+0000","last_clean":"2024-09-14T04:01:47.040465+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:01:47.040465+0000","last_undegraded":"2024-09-14T04:01:47.040465+0000","last_fullsized":"2024-09-14T04:01:47.040465+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035430+0000","last_change":"2024-09-14T03:58:45.810163+0000","last_active":"2024-09-14T04:01:47.035430+0000","last_peered":"2024-09-14T04:01:47.035430+0000","last_clean":"2024-09-14T04:01:47.035430+0000","last_became_active":"2024-09-14T03:58:45.809802+0000","last_became_peered":"2024-09-14T03:58:45.809802+0000","last_unstale":"2024-09-14T04:01:47.035430+0000","last_undegraded":"2024-09-14T04:01:47.035430+0000","last_fullsized":"2024-09-14T04:01:47.035430+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:40:33.911609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366386+0000","last_change":"2024-09-14T03:54:09.079995+0000","last_active":"2024-09-14T04:01:49.366386+0000","last_peered":"2024-09-14T04:01:49.366386+0000","last_clean":"2024-09-14T04:01:49.366386+0000","last_became_active":"2024-09-14T03:54:09.079686+0000","last_became_peered":"2024-09-14T03:54:09.079686+0000","last_unstale":"2024-09-14T04:01:49.366386+0000","last_undegraded":"2024-09-14T04:01:49.366386+0000","last_fullsized":"2024-09-14T04:01:49.366386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:15:31.274947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":134,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472859+0000","last_change":"2024-09-14T03:58:45.788412+0000","last_active":"2024-09-14T04:01:49.472859+0000","last_peered":"2024-09-14T04:01:49.472859+0000","last_clean":"2024-09-14T04:01:49.472859+0000","last_became_active":"2024-09-14T03:58:45.788078+0000","last_became_peered":"2024-09-14T03:58:45.788078+0000","last_unstale":"2024-09-14T04:01:49.472859+0000","last_undegraded":"2024-09-14T04:01:49.472859+0000","last_fullsized":"2024-09-14T04:01:49.472859+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:49:09.289398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392964+0000","last_change":"2024-09-14T03:58:45.812662+0000","last_active":"2024-09-14T04:01:47.392964+0000","last_peered":"2024-09-14T04:01:47.392964+0000","last_clean":"2024-09-14T04:01:47.392964+0000","last_became_active":"2024-09-14T03:58:45.812338+0000","last_became_peered":"2024-09-14T03:58:45.812338+0000","last_unstale":"2024-09-14T04:01:47.392964+0000","last_undegraded":"2024-09-14T04:01:47.392964+0000","last_fullsized":"2024-09-14T04:01:47.392964+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:35:35.258171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040574+0000","last_change":"2024-09-14T03:54:09.089856+0000","last_active":"2024-09-14T04:01:47.040574+0000","last_peered":"2024-09-14T04:01:47.040574+0000","last_clean":"2024-09-14T04:01:47.040574+0000","last_became_active":"2024-09-14T03:54:09.089559+0000","last_became_peered":"2024-09-14T03:54:09.089559+0000","last_unstale":"2024-09-14T04:01:47.040574+0000","last_undegraded":"2024-09-14T04:01:47.040574+0000","last_fullsized":"2024-09-14T04:01:47.040574+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:11:49.651236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393207+0000","last_change":"2024-09-14T03:55:58.249703+0000","last_active":"2024-09-14T04:01:47.393207+0000","last_peered":"2024-09-14T04:01:47.393207+0000","last_clean":"2024-09-14T04:01:47.393207+0000","last_became_active":"2024-09-14T03:55:58.249465+0000","last_became_peered":"2024-09-14T03:55:58.249465+0000","last_unstale":"2024-09-14T04:01:47.393207+0000","last_undegraded":"2024-09-14T04:01:47.393207+0000","last_fullsized":"2024-09-14T04:01:47.393207+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:03.743246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392875+0000","last_change":"2024-09-14T03:55:58.246315+0000","last_active":"2024-09-14T04:01:47.392875+0000","last_peered":"2024-09-14T04:01:47.392875+0000","last_clean":"2024-09-14T04:01:47.392875+0000","last_became_active":"2024-09-14T03:55:58.245997+0000","last_became_peered":"2024-09-14T03:55:58.245997+0000","last_unstale":"2024-09-14T04:01:47.392875+0000","last_undegraded":"2024-09-14T04:01:47.392875+0000","last_fullsized":"2024-09-14T04:01:47.392875+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:11:13.312397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393206+0000","last_change":"2024-09-14T03:54:09.082688+0000","last_active":"2024-09-14T04:01:47.393206+0000","last_peered":"2024-09-14T04:01:47.393206+0000","last_clean":"2024-09-14T04:01:47.393206+0000","last_became_active":"2024-09-14T03:54:09.082402+0000","last_became_peered":"2024-09-14T03:54:09.082402+0000","last_unstale":"2024-09-14T04:01:47.393206+0000","last_undegraded":"2024-09-14T04:01:47.393206+0000","last_fullsized":"2024-09-14T04:01:47.393206+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:28:53.024974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":107,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473967+0000","last_change":"2024-09-14T03:58:45.822672+0000","last_active":"2024-09-14T04:01:49.473967+0000","last_peered":"2024-09-14T04:01:49.473967+0000","last_clean":"2024-09-14T04:01:49.473967+0000","last_became_active":"2024-09-14T03:58:45.822047+0000","last_became_peered":"2024-09-14T03:58:45.822047+0000","last_unstale":"2024-09-14T04:01:49.473967+0000","last_undegraded":"2024-09-14T04:01:49.473967+0000","last_fullsized":"2024-09-14T04:01:49.473967+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:29:49.341047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093884+0000","last_change":"2024-09-14T03:58:45.793497+0000","last_active":"2024-09-14T04:01:49.093884+0000","last_peered":"2024-09-14T04:01:49.093884+0000","last_clean":"2024-09-14T04:01:49.093884+0000","last_became_active":"2024-09-14T03:58:45.793169+0000","last_became_peered":"2024-09-14T03:58:45.793169+0000","last_unstale":"2024-09-14T04:01:49.093884+0000","last_undegraded":"2024-09-14T04:01:49.093884+0000","last_fullsized":"2024-09-14T04:01:49.093884+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:11:06.166164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":233,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044476+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:01:47.044476+0000","last_peered":"2024-09-14T04:01:47.044476+0000","last_clean":"2024-09-14T04:01:47.044476+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:01:47.044476+0000","last_undegraded":"2024-09-14T04:01:47.044476+0000","last_fullsized":"2024-09-14T04:01:47.044476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":130,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035317+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:01:47.035317+0000","last_peered":"2024-09-14T04:01:47.035317+0000","last_clean":"2024-09-14T04:01:47.035317+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:01:47.035317+0000","last_undegraded":"2024-09-14T04:01:47.035317+0000","last_fullsized":"2024-09-14T04:01:47.035317+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":94,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472088+0000","last_change":"2024-09-14T03:58:45.782514+0000","last_active":"2024-09-14T04:01:49.472088+0000","last_peered":"2024-09-14T04:01:49.472088+0000","last_clean":"2024-09-14T04:01:49.472088+0000","last_became_active":"2024-09-14T03:58:45.782196+0000","last_became_peered":"2024-09-14T03:58:45.782196+0000","last_unstale":"2024-09-14T04:01:49.472088+0000","last_undegraded":"2024-09-14T04:01:49.472088+0000","last_fullsized":"2024-09-14T04:01:49.472088+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:59:44.614261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":99,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040892+0000","last_change":"2024-09-14T03:43:11.909945+0000","last_active":"2024-09-14T04:01:47.040892+0000","last_peered":"2024-09-14T04:01:47.040892+0000","last_clean":"2024-09-14T04:01:47.040892+0000","last_became_active":"2024-09-14T03:43:11.909666+0000","last_became_peered":"2024-09-14T03:43:11.909666+0000","last_unstale":"2024-09-14T04:01:47.040892+0000","last_undegraded":"2024-09-14T04:01:47.040892+0000","last_fullsized":"2024-09-14T04:01:47.040892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:13:19.629641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":145,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044463+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:01:47.044463+0000","last_peered":"2024-09-14T04:01:47.044463+0000","last_clean":"2024-09-14T04:01:47.044463+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:01:47.044463+0000","last_undegraded":"2024-09-14T04:01:47.044463+0000","last_fullsized":"2024-09-14T04:01:47.044463+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366618+0000","last_change":"2024-09-14T03:58:45.795127+0000","last_active":"2024-09-14T04:01:49.366618+0000","last_peered":"2024-09-14T04:01:49.366618+0000","last_clean":"2024-09-14T04:01:49.366618+0000","last_became_active":"2024-09-14T03:58:45.794831+0000","last_became_peered":"2024-09-14T03:58:45.794831+0000","last_unstale":"2024-09-14T04:01:49.366618+0000","last_undegraded":"2024-09-14T04:01:49.366618+0000","last_fullsized":"2024-09-14T04:01:49.366618+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:22:18.983609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.370773+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:01:49.370773+0000","last_peered":"2024-09-14T04:01:49.370773+0000","last_clean":"2024-09-14T04:01:49.370773+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:01:49.370773+0000","last_undegraded":"2024-09-14T04:01:49.370773+0000","last_fullsized":"2024-09-14T04:01:49.370773+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":168,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471920+0000","last_change":"2024-09-14T03:58:45.791394+0000","last_active":"2024-09-14T04:01:49.471920+0000","last_peered":"2024-09-14T04:01:49.471920+0000","last_clean":"2024-09-14T04:01:49.471920+0000","last_became_active":"2024-09-14T03:58:45.790907+0000","last_became_peered":"2024-09-14T03:58:45.790907+0000","last_unstale":"2024-09-14T04:01:49.471920+0000","last_undegraded":"2024-09-14T04:01:49.471920+0000","last_fullsized":"2024-09-14T04:01:49.471920+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:07.434141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":143,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093019+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:01:49.093019+0000","last_peered":"2024-09-14T04:01:49.093019+0000","last_clean":"2024-09-14T04:01:49.093019+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:01:49.093019+0000","last_undegraded":"2024-09-14T04:01:49.093019+0000","last_fullsized":"2024-09-14T04:01:49.093019+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":168,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093834+0000","last_change":"2024-09-14T03:54:09.087551+0000","last_active":"2024-09-14T04:01:49.093834+0000","last_peered":"2024-09-14T04:01:49.093834+0000","last_clean":"2024-09-14T04:01:49.093834+0000","last_became_active":"2024-09-14T03:54:09.086608+0000","last_became_peered":"2024-09-14T03:54:09.086608+0000","last_unstale":"2024-09-14T04:01:49.093834+0000","last_undegraded":"2024-09-14T04:01:49.093834+0000","last_fullsized":"2024-09-14T04:01:49.093834+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:38:14.920066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":131,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035236+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:01:47.035236+0000","last_peered":"2024-09-14T04:01:47.035236+0000","last_clean":"2024-09-14T04:01:47.035236+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:01:47.035236+0000","last_undegraded":"2024-09-14T04:01:47.035236+0000","last_fullsized":"2024-09-14T04:01:47.035236+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":116,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393034+0000","last_change":"2024-09-14T03:52:28.187042+0000","last_active":"2024-09-14T04:01:47.393034+0000","last_peered":"2024-09-14T04:01:47.393034+0000","last_clean":"2024-09-14T04:01:47.393034+0000","last_became_active":"2024-09-14T03:52:28.186714+0000","last_became_peered":"2024-09-14T03:52:28.186714+0000","last_unstale":"2024-09-14T04:01:47.393034+0000","last_undegraded":"2024-09-14T04:01:47.393034+0000","last_fullsized":"2024-09-14T04:01:47.393034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:50:57.007224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":154,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093063+0000","last_change":"2024-09-14T03:58:45.812745+0000","last_active":"2024-09-14T04:01:49.093063+0000","last_peered":"2024-09-14T04:01:49.093063+0000","last_clean":"2024-09-14T04:01:49.093063+0000","last_became_active":"2024-09-14T03:58:45.812385+0000","last_became_peered":"2024-09-14T03:58:45.812385+0000","last_unstale":"2024-09-14T04:01:49.093063+0000","last_undegraded":"2024-09-14T04:01:49.093063+0000","last_fullsized":"2024-09-14T04:01:49.093063+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:18:40.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035621+0000","last_change":"2024-09-14T03:58:45.788256+0000","last_active":"2024-09-14T04:01:47.035621+0000","last_peered":"2024-09-14T04:01:47.035621+0000","last_clean":"2024-09-14T04:01:47.035621+0000","last_became_active":"2024-09-14T03:58:45.788052+0000","last_became_peered":"2024-09-14T03:58:45.788052+0000","last_unstale":"2024-09-14T04:01:47.035621+0000","last_undegraded":"2024-09-14T04:01:47.035621+0000","last_fullsized":"2024-09-14T04:01:47.035621+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:38:19.891512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":182,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093135+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:01:49.093135+0000","last_peered":"2024-09-14T04:01:49.093135+0000","last_clean":"2024-09-14T04:01:49.093135+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:01:49.093135+0000","last_undegraded":"2024-09-14T04:01:49.093135+0000","last_fullsized":"2024-09-14T04:01:49.093135+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093546+0000","last_change":"2024-09-14T03:54:09.089016+0000","last_active":"2024-09-14T04:01:49.093546+0000","last_peered":"2024-09-14T04:01:49.093546+0000","last_clean":"2024-09-14T04:01:49.093546+0000","last_became_active":"2024-09-14T03:54:09.087653+0000","last_became_peered":"2024-09-14T03:54:09.087653+0000","last_unstale":"2024-09-14T04:01:49.093546+0000","last_undegraded":"2024-09-14T04:01:49.093546+0000","last_fullsized":"2024-09-14T04:01:49.093546+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:07:46.505921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":143,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093259+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:01:49.093259+0000","last_peered":"2024-09-14T04:01:49.093259+0000","last_clean":"2024-09-14T04:01:49.093259+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:01:49.093259+0000","last_undegraded":"2024-09-14T04:01:49.093259+0000","last_fullsized":"2024-09-14T04:01:49.093259+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":128,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044846+0000","last_change":"2024-09-14T03:52:28.190501+0000","last_active":"2024-09-14T04:01:47.044846+0000","last_peered":"2024-09-14T04:01:47.044846+0000","last_clean":"2024-09-14T04:01:47.044846+0000","last_became_active":"2024-09-14T03:52:28.190050+0000","last_became_peered":"2024-09-14T03:52:28.190050+0000","last_unstale":"2024-09-14T04:01:47.044846+0000","last_undegraded":"2024-09-14T04:01:47.044846+0000","last_fullsized":"2024-09-14T04:01:47.044846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:58:22.557305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":166,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366465+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:01:49.366465+0000","last_peered":"2024-09-14T04:01:49.366465+0000","last_clean":"2024-09-14T04:01:49.366465+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:01:49.366465+0000","last_undegraded":"2024-09-14T04:01:49.366465+0000","last_fullsized":"2024-09-14T04:01:49.366465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":134,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473560+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:01:49.473560+0000","last_peered":"2024-09-14T04:01:49.473560+0000","last_clean":"2024-09-14T04:01:49.473560+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:01:49.473560+0000","last_undegraded":"2024-09-14T04:01:49.473560+0000","last_fullsized":"2024-09-14T04:01:49.473560+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.093627+0000","last_change":"2024-09-14T03:54:09.082573+0000","last_active":"2024-09-14T04:01:49.093627+0000","last_peered":"2024-09-14T04:01:49.093627+0000","last_clean":"2024-09-14T04:01:49.093627+0000","last_became_active":"2024-09-14T03:54:09.082266+0000","last_became_peered":"2024-09-14T03:54:09.082266+0000","last_unstale":"2024-09-14T04:01:49.093627+0000","last_undegraded":"2024-09-14T04:01:49.093627+0000","last_fullsized":"2024-09-14T04:01:49.093627+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:30:24.244999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472388+0000","last_change":"2024-09-14T03:58:45.822348+0000","last_active":"2024-09-14T04:01:49.472388+0000","last_peered":"2024-09-14T04:01:49.472388+0000","last_clean":"2024-09-14T04:01:49.472388+0000","last_became_active":"2024-09-14T03:58:45.821621+0000","last_became_peered":"2024-09-14T03:58:45.821621+0000","last_unstale":"2024-09-14T04:01:49.472388+0000","last_undegraded":"2024-09-14T04:01:49.472388+0000","last_fullsized":"2024-09-14T04:01:49.472388+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:02:12.477742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.371059+0000","last_change":"2024-09-14T03:54:09.081554+0000","last_active":"2024-09-14T04:01:49.371059+0000","last_peered":"2024-09-14T04:01:49.371059+0000","last_clean":"2024-09-14T04:01:49.371059+0000","last_became_active":"2024-09-14T03:54:09.081158+0000","last_became_peered":"2024-09-14T03:54:09.081158+0000","last_unstale":"2024-09-14T04:01:49.371059+0000","last_undegraded":"2024-09-14T04:01:49.371059+0000","last_fullsized":"2024-09-14T04:01:49.371059+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:24:43.027300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":129,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040935+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:01:47.040935+0000","last_peered":"2024-09-14T04:01:47.040935+0000","last_clean":"2024-09-14T04:01:47.040935+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:01:47.040935+0000","last_undegraded":"2024-09-14T04:01:47.040935+0000","last_fullsized":"2024-09-14T04:01:47.040935+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":98,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392823+0000","last_change":"2024-09-14T03:43:11.285521+0000","last_active":"2024-09-14T04:01:47.392823+0000","last_peered":"2024-09-14T04:01:47.392823+0000","last_clean":"2024-09-14T04:01:47.392823+0000","last_became_active":"2024-09-14T03:43:11.284778+0000","last_became_peered":"2024-09-14T03:43:11.284778+0000","last_unstale":"2024-09-14T04:01:47.392823+0000","last_undegraded":"2024-09-14T04:01:47.392823+0000","last_fullsized":"2024-09-14T04:01:47.392823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:51:32.744643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":108,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392975+0000","last_change":"2024-09-14T03:43:52.490179+0000","last_active":"2024-09-14T04:01:47.392975+0000","last_peered":"2024-09-14T04:01:47.392975+0000","last_clean":"2024-09-14T04:01:47.392975+0000","last_became_active":"2024-09-14T03:43:52.489786+0000","last_became_peered":"2024-09-14T03:43:52.489786+0000","last_unstale":"2024-09-14T04:01:47.392975+0000","last_undegraded":"2024-09-14T04:01:47.392975+0000","last_fullsized":"2024-09-14T04:01:47.392975+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:55.590247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":121,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.393060+0000","last_change":"2024-09-14T03:43:11.274139+0000","last_active":"2024-09-14T04:01:47.393060+0000","last_peered":"2024-09-14T04:01:47.393060+0000","last_clean":"2024-09-14T04:01:47.393060+0000","last_became_active":"2024-09-14T03:43:11.271423+0000","last_became_peered":"2024-09-14T03:43:11.271423+0000","last_unstale":"2024-09-14T04:01:47.393060+0000","last_undegraded":"2024-09-14T04:01:47.393060+0000","last_fullsized":"2024-09-14T04:01:47.393060+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:52.045590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.092936+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:01:49.092936+0000","last_peered":"2024-09-14T04:01:49.092936+0000","last_clean":"2024-09-14T04:01:49.092936+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:01:49.092936+0000","last_undegraded":"2024-09-14T04:01:49.092936+0000","last_fullsized":"2024-09-14T04:01:49.092936+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.044710+0000","last_change":"2024-09-14T03:58:45.793790+0000","last_active":"2024-09-14T04:01:47.044710+0000","last_peered":"2024-09-14T04:01:47.044710+0000","last_clean":"2024-09-14T04:01:47.044710+0000","last_became_active":"2024-09-14T03:58:45.793253+0000","last_became_peered":"2024-09-14T03:58:45.793253+0000","last_unstale":"2024-09-14T04:01:47.044710+0000","last_undegraded":"2024-09-14T04:01:47.044710+0000","last_fullsized":"2024-09-14T04:01:47.044710+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:44.988345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":117,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035444+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:01:47.035444+0000","last_peered":"2024-09-14T04:01:47.035444+0000","last_clean":"2024-09-14T04:01:47.035444+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:01:47.035444+0000","last_undegraded":"2024-09-14T04:01:47.035444+0000","last_fullsized":"2024-09-14T04:01:47.035444+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":94,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471884+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:01:49.471884+0000","last_peered":"2024-09-14T04:01:49.471884+0000","last_clean":"2024-09-14T04:01:49.471884+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:01:49.471884+0000","last_undegraded":"2024-09-14T04:01:49.471884+0000","last_fullsized":"2024-09-14T04:01:49.471884+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":177,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366797+0000","last_change":"2024-09-14T03:58:45.794557+0000","last_active":"2024-09-14T04:01:49.366797+0000","last_peered":"2024-09-14T04:01:49.366797+0000","last_clean":"2024-09-14T04:01:49.366797+0000","last_became_active":"2024-09-14T03:58:45.794173+0000","last_became_peered":"2024-09-14T03:58:45.794173+0000","last_unstale":"2024-09-14T04:01:49.366797+0000","last_undegraded":"2024-09-14T04:01:49.366797+0000","last_fullsized":"2024-09-14T04:01:49.366797+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:33:12.799351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.471703+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:01:49.471703+0000","last_peered":"2024-09-14T04:01:49.471703+0000","last_clean":"2024-09-14T04:01:49.471703+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:01:49.471703+0000","last_undegraded":"2024-09-14T04:01:49.471703+0000","last_fullsized":"2024-09-14T04:01:49.471703+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":126,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.040902+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:01:47.040902+0000","last_peered":"2024-09-14T04:01:47.040902+0000","last_clean":"2024-09-14T04:01:47.040902+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:01:47.040902+0000","last_undegraded":"2024-09-14T04:01:47.040902+0000","last_fullsized":"2024-09-14T04:01:47.040902+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":124,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472437+0000","last_change":"2024-09-14T03:58:45.822679+0000","last_active":"2024-09-14T04:01:49.472437+0000","last_peered":"2024-09-14T04:01:49.472437+0000","last_clean":"2024-09-14T04:01:49.472437+0000","last_became_active":"2024-09-14T03:58:45.822051+0000","last_became_peered":"2024-09-14T03:58:45.822051+0000","last_unstale":"2024-09-14T04:01:49.472437+0000","last_undegraded":"2024-09-14T04:01:49.472437+0000","last_fullsized":"2024-09-14T04:01:49.472437+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:30:52.022607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":178,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.370935+0000","last_change":"2024-09-14T03:58:45.795656+0000","last_active":"2024-09-14T04:01:49.370935+0000","last_peered":"2024-09-14T04:01:49.370935+0000","last_clean":"2024-09-14T04:01:49.370935+0000","last_became_active":"2024-09-14T03:58:45.795369+0000","last_became_peered":"2024-09-14T03:58:45.795369+0000","last_unstale":"2024-09-14T04:01:49.370935+0000","last_undegraded":"2024-09-14T04:01:49.370935+0000","last_fullsized":"2024-09-14T04:01:49.370935+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:45.896015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.092900+0000","last_change":"2024-09-14T03:58:45.793634+0000","last_active":"2024-09-14T04:01:49.092900+0000","last_peered":"2024-09-14T04:01:49.092900+0000","last_clean":"2024-09-14T04:01:49.092900+0000","last_became_active":"2024-09-14T03:58:45.793229+0000","last_became_peered":"2024-09-14T03:58:45.793229+0000","last_unstale":"2024-09-14T04:01:49.092900+0000","last_undegraded":"2024-09-14T04:01:49.092900+0000","last_fullsized":"2024-09-14T04:01:49.092900+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:31.431481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":106,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473119+0000","last_change":"2024-09-14T03:58:45.795790+0000","last_active":"2024-09-14T04:01:49.473119+0000","last_peered":"2024-09-14T04:01:49.473119+0000","last_clean":"2024-09-14T04:01:49.473119+0000","last_became_active":"2024-09-14T03:58:45.795468+0000","last_became_peered":"2024-09-14T03:58:45.795468+0000","last_unstale":"2024-09-14T04:01:49.473119+0000","last_undegraded":"2024-09-14T04:01:49.473119+0000","last_fullsized":"2024-09-14T04:01:49.473119+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:50:54.779571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":134,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392887+0000","last_change":"2024-09-14T03:55:58.252985+0000","last_active":"2024-09-14T04:01:47.392887+0000","last_peered":"2024-09-14T04:01:47.392887+0000","last_clean":"2024-09-14T04:01:47.392887+0000","last_became_active":"2024-09-14T03:55:58.252479+0000","last_became_peered":"2024-09-14T03:55:58.252479+0000","last_unstale":"2024-09-14T04:01:47.392887+0000","last_undegraded":"2024-09-14T04:01:47.392887+0000","last_fullsized":"2024-09-14T04:01:47.392887+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:46:04.357446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":171,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.472629+0000","last_change":"2024-09-14T03:58:45.810289+0000","last_active":"2024-09-14T04:01:49.472629+0000","last_peered":"2024-09-14T04:01:49.472629+0000","last_clean":"2024-09-14T04:01:49.472629+0000","last_became_active":"2024-09-14T03:58:45.809938+0000","last_became_peered":"2024-09-14T03:58:45.809938+0000","last_unstale":"2024-09-14T04:01:49.472629+0000","last_undegraded":"2024-09-14T04:01:49.472629+0000","last_fullsized":"2024-09-14T04:01:49.472629+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:06:18.534582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":169,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366607+0000","last_change":"2024-09-14T03:52:28.182347+0000","last_active":"2024-09-14T04:01:49.366607+0000","last_peered":"2024-09-14T04:01:49.366607+0000","last_clean":"2024-09-14T04:01:49.366607+0000","last_became_active":"2024-09-14T03:52:28.181624+0000","last_became_peered":"2024-09-14T03:52:28.181624+0000","last_unstale":"2024-09-14T04:01:49.366607+0000","last_undegraded":"2024-09-14T04:01:49.366607+0000","last_fullsized":"2024-09-14T04:01:49.366607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:18.032619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392932+0000","last_change":"2024-09-14T03:58:45.793294+0000","last_active":"2024-09-14T04:01:47.392932+0000","last_peered":"2024-09-14T04:01:47.392932+0000","last_clean":"2024-09-14T04:01:47.392932+0000","last_became_active":"2024-09-14T03:58:45.793009+0000","last_became_peered":"2024-09-14T03:58:45.793009+0000","last_unstale":"2024-09-14T04:01:47.392932+0000","last_undegraded":"2024-09-14T04:01:47.392932+0000","last_fullsized":"2024-09-14T04:01:47.392932+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:46:00.206467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":119,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035459+0000","last_change":"2024-09-14T03:54:09.084402+0000","last_active":"2024-09-14T04:01:47.035459+0000","last_peered":"2024-09-14T04:01:47.035459+0000","last_clean":"2024-09-14T04:01:47.035459+0000","last_became_active":"2024-09-14T03:54:09.084076+0000","last_became_peered":"2024-09-14T03:54:09.084076+0000","last_unstale":"2024-09-14T04:01:47.035459+0000","last_undegraded":"2024-09-14T04:01:47.035459+0000","last_fullsized":"2024-09-14T04:01:47.035459+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:49:51.780807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":132,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392790+0000","last_change":"2024-09-14T03:58:45.793059+0000","last_active":"2024-09-14T04:01:47.392790+0000","last_peered":"2024-09-14T04:01:47.392790+0000","last_clean":"2024-09-14T04:01:47.392790+0000","last_became_active":"2024-09-14T03:58:45.792703+0000","last_became_peered":"2024-09-14T03:58:45.792703+0000","last_unstale":"2024-09-14T04:01:47.392790+0000","last_undegraded":"2024-09-14T04:01:47.392790+0000","last_fullsized":"2024-09-14T04:01:47.392790+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:01:17.321430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":102,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.035534+0000","last_change":"2024-09-14T03:43:11.278982+0000","last_active":"2024-09-14T04:01:47.035534+0000","last_peered":"2024-09-14T04:01:47.035534+0000","last_clean":"2024-09-14T04:01:47.035534+0000","last_became_active":"2024-09-14T03:43:11.277577+0000","last_became_peered":"2024-09-14T03:43:11.277577+0000","last_unstale":"2024-09-14T04:01:47.035534+0000","last_undegraded":"2024-09-14T04:01:47.035534+0000","last_fullsized":"2024-09-14T04:01:47.035534+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:51.445273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473059+0000","last_change":"2024-09-14T03:58:45.790089+0000","last_active":"2024-09-14T04:01:49.473059+0000","last_peered":"2024-09-14T04:01:49.473059+0000","last_clean":"2024-09-14T04:01:49.473059+0000","last_became_active":"2024-09-14T03:58:45.789707+0000","last_became_peered":"2024-09-14T03:58:45.789707+0000","last_unstale":"2024-09-14T04:01:49.473059+0000","last_undegraded":"2024-09-14T04:01:49.473059+0000","last_fullsized":"2024-09-14T04:01:49.473059+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:03:26.492725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":121,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.473333+0000","last_change":"2024-09-14T03:58:45.784441+0000","last_active":"2024-09-14T04:01:49.473333+0000","last_peered":"2024-09-14T04:01:49.473333+0000","last_clean":"2024-09-14T04:01:49.473333+0000","last_became_active":"2024-09-14T03:58:45.783913+0000","last_became_peered":"2024-09-14T03:58:45.783913+0000","last_unstale":"2024-09-14T04:01:49.473333+0000","last_undegraded":"2024-09-14T04:01:49.473333+0000","last_fullsized":"2024-09-14T04:01:49.473333+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:16:11.643108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":115,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-14T04:01:47.392630+0000","last_change":"2024-09-14T03:54:09.084713+0000","last_active":"2024-09-14T04:01:47.392630+0000","last_peered":"2024-09-14T04:01:47.392630+0000","last_clean":"2024-09-14T04:01:47.392630+0000","last_became_active":"2024-09-14T03:54:09.082369+0000","last_became_peered":"2024-09-14T03:54:09.082369+0000","last_unstale":"2024-09-14T04:01:47.392630+0000","last_undegraded":"2024-09-14T04:01:47.392630+0000","last_fullsized":"2024-09-14T04:01:47.392630+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:41:39.305100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":180,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-09-14T04:01:49.366512+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:01:49.366512+0000","last_peered":"2024-09-14T04:01:49.366512+0000","last_clean":"2024-09-14T04:01:49.366512+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:01:49.366512+0000","last_undegraded":"2024-09-14T04:01:49.366512+0000","last_fullsized":"2024-09-14T04:01:49.366512+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430468,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600536,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93144616,"statfs":{"total":95995035648,"available":95380086784,"internally_reserved":0,"allocated":2367488,"data_stored":1614300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626697,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299660,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445492,"statfs":{"total":95995035648,"available":95688183808,"internally_reserved":0,"allocated":2355200,"data_stored":1613858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822925,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911160,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":909438,"kb_avail":92833992,"statfs":{"total":95995035648,"available":95062007808,"internally_reserved":0,"allocated":1748992,"data_stored":1019001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931264856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019153,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":599944,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93145208,"statfs":{"total":95995035648,"available":95380692992,"internally_reserved":0,"allocated":1765376,"data_stored":1023547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552306,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":201144,"kb_used_data":1708,"kb_used_omap":4,"kb_used_meta":199419,"kb_avail":93544008,"statfs":{"total":95995035648,"available":95789064192,"internally_reserved":0,"allocated":1748992,"data_stored":1019082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":204205501},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435348,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212852,"kb_used_data":1704,"kb_used_omap":6,"kb_used_meta":211129,"kb_avail":93532300,"statfs":{"total":95995035648,"available":95777075200,"internally_reserved":0,"allocated":1744896,"data_stored":1018134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":216196557},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827817,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327416,"kb_used_data":1708,"kb_used_omap":5,"kb_used_meta":325690,"kb_avail":93417736,"statfs":{"total":95995035648,"available":95659761664,"internally_reserved":0,"allocated":1748992,"data_stored":1019011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":333506799},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383806,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392700,"kb_used_data":2288,"kb_used_omap":5,"kb_used_meta":390394,"kb_avail":93352452,"statfs":{"total":95995035648,"available":95592910848,"internally_reserved":0,"allocated":2342912,"data_stored":1609653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":399763516},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:02:50.615 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-14T04:02:50.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-14T04:02:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:02:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:02:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:02:50.866 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:50.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:50 smithi114 ceph-mon[103029]: from='client.45509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:50.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:50 smithi114 ceph-mon[106969]: from='client.45509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:51.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:50 smithi160 ceph-mon[88407]: from='client.45509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:02:51.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:02:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:02:51] "GET /metrics HTTP/1.1" 200 34002 "" "Prometheus/2.51.0" 2024-09-14T04:02:52.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:51 smithi114 ceph-mon[103029]: pgmap v497: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:52.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:51 smithi114 ceph-mon[106969]: pgmap v497: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:52.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:51 smithi160 ceph-mon[88407]: pgmap v497: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:52.826 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 151 2024-09-14T04:02:52.826 INFO:teuthology.orchestra.run.smithi114.stdout:fsid 63f4e762-724a-11ef-bceb-c7b262605968 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-09-14T03:36:30.445371+0000 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-09-14T04:01:49.034415+0000 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:crush_version 22 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:full_ratio 0.95 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:backfillfull_ratio 0.9 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:nearfull_ratio 0.85 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:require_min_compat_client luminous 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:min_compat_client luminous 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:require_osd_release quincy 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:stretch_mode_enabled false 2024-09-14T04:02:52.827 INFO:teuthology.orchestra.run.smithi114.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 57 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:max_osd 8 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 up in weight 1 up_from 80 up_thru 143 down_at 77 last_clean_interval [8,76) [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658] [v2:172.21.15.114:6804/2022879658,v1:172.21.15.114:6805/2022879658] exists,up 0a021a3a-45d2-4825-a701-d7c3793ef6b2 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 up in weight 1 up_from 97 up_thru 149 down_at 94 last_clean_interval [13,93) [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706] [v2:172.21.15.114:6812/3980338706,v1:172.21.15.114:6813/3980338706] exists,up 076291c4-5927-456d-bf74-2ab79be4c756 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 up in weight 1 up_from 109 up_thru 136 down_at 104 last_clean_interval [18,103) [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440] [v2:172.21.15.114:6820/3555775440,v1:172.21.15.114:6821/3555775440] exists,up d6787400-73ea-4edf-bb17-440ac36bd4ff 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 up in weight 1 up_from 136 up_thru 147 down_at 133 last_clean_interval [25,132) [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847] [v2:172.21.15.114:6828/288809847,v1:172.21.15.114:6829/288809847] exists,up ead755f3-173a-4465-a28f-af7288b0a42b 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 up in weight 1 up_from 30 up_thru 136 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940] [v2:172.21.15.160:6802/825361940,v1:172.21.15.160:6803/825361940] exists,up 60a0aac2-e235-484f-b25f-f112c83da7f7 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 up in weight 1 up_from 36 up_thru 145 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003] [v2:172.21.15.160:6810/3752160003,v1:172.21.15.160:6811/3752160003] exists,up 8f49c7de-c810-4075-b731-022bb5031d2a 2024-09-14T04:02:52.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 up in weight 1 up_from 42 up_thru 133 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982] [v2:172.21.15.160:6818/3716932982,v1:172.21.15.160:6819/3716932982] exists,up 455de335-7290-4242-b743-6b0c446683ad 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 up in weight 1 up_from 48 up_thru 136 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841] [v2:172.21.15.160:6826/1323253841,v1:172.21.15.160:6827/1323253841] exists,up 7d8b5297-b1db-41b1-be35-073e7780128e 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.a [1,2] 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.13 [1,5] 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.1f [1,2] 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1795949490 expires 2024-09-15T03:48:52.256052+0000 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/386045858 expires 2024-09-15T03:48:52.256052+0000 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/3774959769 expires 2024-09-15T03:48:52.256052+0000 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1481425303 expires 2024-09-15T03:48:52.256052+0000 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/210127168 expires 2024-09-15T03:48:52.256052+0000 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.160:0/1259038325 expires 2024-09-15T03:47:52.734325+0000 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.160:0/710559079 expires 2024-09-15T03:47:52.734325+0000 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.160:0/1393128007 expires 2024-09-15T03:47:52.734325+0000 2024-09-14T04:02:52.829 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/426398557 expires 2024-09-15T03:41:51.162793+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1634520325 expires 2024-09-15T03:48:52.256052+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/2959862646 expires 2024-09-15T03:37:21.054968+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/1727106713 expires 2024-09-15T03:46:19.354331+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/767494479 expires 2024-09-15T03:41:51.162793+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2155247703 expires 2024-09-15T03:37:21.054968+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/1727106713 expires 2024-09-15T03:46:19.354331+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/4089637122 expires 2024-09-15T03:46:19.354331+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2536880143 expires 2024-09-15T03:36:58.525006+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/3344792350 expires 2024-09-15T03:41:51.162793+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/441101312 expires 2024-09-15T03:41:51.162793+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1570092639 expires 2024-09-15T03:48:52.256052+0000 2024-09-14T04:02:52.830 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2067882043 expires 2024-09-15T03:37:21.054968+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/3853151361 expires 2024-09-15T03:46:19.354331+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/441101312 expires 2024-09-15T03:41:51.162793+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1784685209 expires 2024-09-15T03:41:51.162793+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/2959862646 expires 2024-09-15T03:37:21.054968+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2995702914 expires 2024-09-15T03:46:19.354331+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2095546075 expires 2024-09-15T03:37:21.054968+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.160:0/1361660388 expires 2024-09-15T03:47:52.734325+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.160:6835/3913895673 expires 2024-09-15T03:47:52.734325+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/3774959769 expires 2024-09-15T03:48:52.256052+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/546762471 expires 2024-09-15T03:46:19.354331+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/1434563052 expires 2024-09-15T03:36:58.525006+0000 2024-09-14T04:02:52.831 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1089490408 expires 2024-09-15T03:36:58.525006+0000 2024-09-14T04:02:52.832 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/1434563052 expires 2024-09-15T03:36:58.525006+0000 2024-09-14T04:02:52.832 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2694023561 expires 2024-09-15T03:36:58.525006+0000 2024-09-14T04:02:52.832 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.160:6834/3913895673 expires 2024-09-15T03:47:52.734325+0000 2024-09-14T04:02:53.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:52 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:53.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:52 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:02:53.484 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 80 up_thru 143 down_at 77 last_clean_interval [8,76) [v2:172.21.15.114:6802/2022879658,v1:172.21.15.114:6803/2022879658] [v2:172.21.15.114:6804/2022879658,v1:172.21.15.114:6805/2022879658] exists,up 0a021a3a-45d2-4825-a701-d7c3793ef6b2', 'osd.1 up in weight 1 up_from 97 up_thru 149 down_at 94 last_clean_interval [13,93) [v2:172.21.15.114:6810/3980338706,v1:172.21.15.114:6811/3980338706] [v2:172.21.15.114:6812/3980338706,v1:172.21.15.114:6813/3980338706] exists,up 076291c4-5927-456d-bf74-2ab79be4c756', 'osd.2 up in weight 1 up_from 109 up_thru 136 down_at 104 last_clean_interval [18,103) [v2:172.21.15.114:6818/3555775440,v1:172.21.15.114:6819/3555775440] [v2:172.21.15.114:6820/3555775440,v1:172.21.15.114:6821/3555775440] exists,up d6787400-73ea-4edf-bb17-440ac36bd4ff', 'osd.3 up in weight 1 up_from 136 up_thru 147 down_at 133 last_clean_interval [25,132) [v2:172.21.15.114:6826/288809847,v1:172.21.15.114:6827/288809847] [v2:172.21.15.114:6828/288809847,v1:172.21.15.114:6829/288809847] exists,up ead755f3-173a-4465-a28f-af7288b0a42b', 'osd.4 up in weight 1 up_from 30 up_thru 136 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6800/825361940,v1:172.21.15.160:6801/825361940] [v2:172.21.15.160:6802/825361940,v1:172.21.15.160:6803/825361940] exists,up 60a0aac2-e235-484f-b25f-f112c83da7f7', 'osd.5 up in weight 1 up_from 36 up_thru 145 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6808/3752160003,v1:172.21.15.160:6809/3752160003] [v2:172.21.15.160:6810/3752160003,v1:172.21.15.160:6811/3752160003] exists,up 8f49c7de-c810-4075-b731-022bb5031d2a', 'osd.6 up in weight 1 up_from 42 up_thru 133 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6816/3716932982,v1:172.21.15.160:6817/3716932982] [v2:172.21.15.160:6818/3716932982,v1:172.21.15.160:6819/3716932982] exists,up 455de335-7290-4242-b743-6b0c446683ad', 'osd.7 up in weight 1 up_from 48 up_thru 136 down_at 0 last_clean_interval [0,0) [v2:172.21.15.160:6824/1323253841,v1:172.21.15.160:6825/1323253841] [v2:172.21.15.160:6826/1323253841,v1:172.21.15.160:6827/1323253841] exists,up 7d8b5297-b1db-41b1-be35-073e7780128e'] 2024-09-14T04:02:53.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-14T04:02:53.728 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:53.849 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:53 smithi114 ceph-mon[103029]: pgmap v498: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:53.849 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1531272049' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-14T04:02:53.850 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:53 smithi114 ceph-mon[106969]: pgmap v498: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:53.850 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1531272049' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-14T04:02:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:53 smithi160 ceph-mon[88407]: pgmap v498: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1531272049' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-14T04:02:55.669 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mon.a", 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster": "ceph", 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "admin_socket_mode": "", 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_client_required": "cephx, none", 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_cluster_required": "cephx", 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_debug": "true", 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-14T04:02:55.670 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_service_required": "cephx", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_supported": "", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_aio": "true", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_aio_poll_ms": "250", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_aio_reap_max": "16", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_async_discard_threads": "0", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_block_size": "4096", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_debug_aio": "false", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_debug_inflight_ios": "false", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_enable_discard": "false", 2024-09-14T04:02:55.671 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_flock_retry": "3", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_inject_crash": "0", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_ioring": "false", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_ioring_hipri": "false", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-14T04:02:55.672 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-14T04:02:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "bdev_type": "", 2024-09-14T04:02:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_alloc_size": "1048576", 2024-09-14T04:02:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_allocator": "hybrid", 2024-09-14T04:02:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_buffered_io": "true", 2024-09-14T04:02:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_check_for_zeros": "false", 2024-09-14T04:02:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-14T04:02:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-14T04:02:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_compact_log_sync": "false", 2024-09-14T04:02:55.673 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_max_log_runway": "4194304", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_max_prefetch": "1048576", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_min_flush_size": "524288", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_min_log_runway": "1048576", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_replay_recovery": "false", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_sync_write": "false", 2024-09-14T04:02:55.674 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_allocation_from_file": "true", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_allocator": "hybrid", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_async_db_compaction": "true", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bdev_label_multi": "true", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-14T04:02:55.675 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_create": "true", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_db_create": "false", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_db_path": "", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_db_size": "0", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_path": "", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_preallocate_file": "false", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_size": "107374182400", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_wal_create": "false", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_wal_path": "", 2024-09-14T04:02:55.676 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_block_wal_size": "100663296", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bluefs": "true", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_autotune": "true", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-14T04:02:55.677 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-14T04:02:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-14T04:02:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-14T04:02:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_size": "0", 2024-09-14T04:02:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-14T04:02:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-14T04:02:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-14T04:02:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-14T04:02:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_type": "2q", 2024-09-14T04:02:55.678 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_clone_cow": "true", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_mode": "none", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-14T04:02:55.679 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_csum_type": "crc32c", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_freelist": "false", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-14T04:02:55.680 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_small_allocations": "0", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_default_buffered_read": "true", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_default_buffered_write": "false", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-14T04:02:55.681 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fail_eio": "false", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-14T04:02:55.682 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_mount": "false", 2024-09-14T04:02:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-14T04:02:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_umount": "false", 2024-09-14T04:02:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-14T04:02:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-14T04:02:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-14T04:02:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-14T04:02:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-14T04:02:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-14T04:02:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-14T04:02:55.683 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-14T04:02:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_ignore_data_csum": "false", 2024-09-14T04:02:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-14T04:02:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-14T04:02:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-14T04:02:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-14T04:02:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_log_op_age": "5.000000", 2024-09-14T04:02:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_alloc_size": "0", 2024-09-14T04:02:55.684 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_blob_size": "0", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_max_deferred_txc": "32", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_min_alloc_size": "0", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_nid_prealloc": "1024", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-14T04:02:55.685 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_retry_disk_reads": "3", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_rocksdb_cf": "true", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_spdk_mem": "512", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-14T04:02:55.686 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_tracing": "false", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-14T04:02:55.687 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_write_v2": "false", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_write_v2_random": "false", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_zero_block_detection": "false", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-14T04:02:55.688 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_cluster_require_signatures": "false", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_cluster_require_version": "2", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_require_signatures": "false", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_require_version": "2", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_service_require_signatures": "false", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_service_require_version": "2", 2024-09-14T04:02:55.689 INFO:teuthology.orchestra.run.smithi114.stdout: "cephx_sign_messages": "true", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "chdir": "", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_acl_type": "", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_asio_thread_count": "2", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_cache_mid": "0.750000", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_cache_size": "16384", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_caps_release_delay": "5", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_check_pool_perm": "true", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_debug_force_sync_read": "false", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_debug_getattr_caps": "false", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_debug_inject_features": "", 2024-09-14T04:02:55.690 INFO:teuthology.orchestra.run.smithi114.stdout: "client_debug_inject_tick_delay": "0", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_die_on_failed_remount": "false", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_dirsize_rbytes": "true", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_force_lazyio": "false", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_fs": "", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_inject_release_failure": "false", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_max_inline_size": "4096", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_mds_namespace": "", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_metadata": "", 2024-09-14T04:02:55.691 INFO:teuthology.orchestra.run.smithi114.stdout: "client_mount_gid": "-1", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_mount_timeout": "300", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_mount_uid": "-1", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_mountpoint": "/", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_notify_timeout": "10", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc": "true", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc_max_dirty": "104857600", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc_max_objects": "1000", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc_size": "209715200", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_oc_target_dirty": "8388608", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_permissions": "true", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_quota": "true", 2024-09-14T04:02:55.692 INFO:teuthology.orchestra.run.smithi114.stdout: "client_quota_df": "true", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_readahead_max_bytes": "0", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_readahead_max_periods": "4", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_readahead_min": "131072", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_reconnect_stale": "false", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_shutdown_timeout": "30", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_snapdir": ".snap", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_tick_interval": "1", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_trace": "", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_try_dentry_invalidate": "false", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_use_faked_inos": "false", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "client_use_random_mds": "false", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_graylog": "false", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-14T04:02:55.693 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_graylog_port": "12201", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_monitors": "default=true", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_syslog": "false", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "clog_to_syslog_level": "info", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_addr": "-", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_network": "", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_network_interface": "", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "compressor_zlib_isal": "false", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "compressor_zlib_level": "5", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "compressor_zlib_winsize": "-15", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "compressor_zstd_level": "1", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-14T04:02:55.694 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_alien_op_num_threads": "6", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_osd_stat_interval": "0", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_seastar_cpu_cores": "", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "crimson_seastar_num_threads": "0", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location": "", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location_hook": "", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_location_hook_timeout": "10", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "daemonize": "false", 2024-09-14T04:02:55.695 INFO:teuthology.orchestra.run.smithi114.stdout: "daos_pool": "tank", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_alienstore": "0/5", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_allow_any_pool_priority": "false", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_asok": "1/5", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_asok_assert_abort": "false", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_asserts_on_shutdown": "true", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_auth": "1/5", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_bdev": "1/3", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_bluefs": "1/5", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_bluestore": "1/5", 2024-09-14T04:02:55.696 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_buffer": "0/1", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_ceph_dedup": "0/5", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_ceph_exporter": "1/5", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_cephfs_mirror": "0/5", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_cephsqlite": "0/5", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_client": "0/5", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_compressor": "1/5", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_context": "0/1", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_crimson_interrupt": "0/5", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_crush": "1/1", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_crypto": "1/5", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_cyanstore": "0/5", 2024-09-14T04:02:55.697 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_deliberately_leak_memory": "false", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_disable_randomized_ping": "false", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_dpdk": "1/5", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_eventtrace": "1/5", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_filer": "0/1", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_filestore": "1/3", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_finisher": "1/1", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_fuse": "1/5", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_heartbeat_testing_span": "0", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_heartbeatmap": "1/5", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_javaclient": "1/5", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_journal": "1/3", 2024-09-14T04:02:55.698 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_journaler": "0/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_kstore": "1/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_lockdep": "0/1", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mclock": "1/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds": "1/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_balancer": "1/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_locker": "1/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_log": "1/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_log_expire": "1/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_migrator": "1/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mds_quiesce": "3/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_memstore": "1/5", 2024-09-14T04:02:55.699 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mgr": "2/5", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mgrc": "1/5", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_mon": "20/20", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_monc": "0/10", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_ms": "1/1", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_none": "0/5", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_objclass": "0/5", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_objectcacher": "0/5", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_objecter": "0/1", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_optracker": "0/5", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_osd": "1/5", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_paxos": "20/20", 2024-09-14T04:02:55.700 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_perfcounter": "1/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_prioritycache": "1/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rados": "0/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rbd": "0/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rbd_mirror": "0/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rbd_pwl": "0/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rbd_replay": "0/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_refs": "0/0", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_reserver": "1/1", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw": "1/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_access": "1/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_datacache": "1/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_dbstore": "1/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_flight": "1/5", 2024-09-14T04:02:55.701 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_notification": "1/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rgw_sync": "1/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_rocksdb": "4/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore": "0/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_backref": "0/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_cache": "0/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_cleaner": "0/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_device": "0/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_epm": "0/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_journal": "0/5", 2024-09-14T04:02:55.702 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_lba": "0/5", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_odata": "0/5", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_omap": "0/5", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_onode": "0/5", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_t": "0/5", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_seastore_tm": "0/5", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_striper": "0/1", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_test": "0/5", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_throttle": "1/1", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_timer": "0/1", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_tp": "0/5", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "debug_trace": "1/5", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-14T04:02:55.703 INFO:teuthology.orchestra.run.smithi114.stdout: "device_failure_prediction_mode": "none", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "err_to_graylog": "false", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "err_to_journald": "false", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "err_to_stderr": "true", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "err_to_syslog": "false", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event_tracing": "false", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_addr": "0.0.0.0", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_cert_file": "", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_http_port": "9926", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_key_file": "", 2024-09-14T04:02:55.704 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_prio_limit": "5", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_sort_metrics": "true", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "exporter_stats_period": "5", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "fake_statfs_for_testing": "0", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "fatal_signal_handlers": "true", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "filer_max_purge_ops": "10", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "filer_max_truncate_ops": "128", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_apply_finisher_threads": "1", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_blackhole": "false", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_btrfs_clone_range": "true", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_btrfs_snap": "true", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_caller_concurrency": "10", 2024-09-14T04:02:55.705 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_collect_device_partition_information": "true", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_commit_timeout": "600.000000", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_debug_inject_read_err": "false", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_debug_omap_check": "false", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_debug_verify_split": "false", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_dump_file": "", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fadvise": "true", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fail_eio": "true", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fd_cache_shards": "16", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fd_cache_size": "128", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fiemap": "false", 2024-09-14T04:02:55.706 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fiemap_threshold": "4096", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_inject_stall": "0", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_journal_parallel": "false", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_journal_trailing": "false", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_journal_writeahead": "false", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_kill_at": "0", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattrs": "0", 2024-09-14T04:02:55.707 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_xattr_value_size": "0", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_merge_threshold": "-10", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_odsync_write": "false", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_omap_backend": "rocksdb", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_omap_backend_path": "", 2024-09-14T04:02:55.708 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_op_thread_timeout": "60", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_op_threads": "2", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_punch_hole": "false", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-14T04:02:55.709 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_queue_max_ops": "50", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_seek_data_hole": "false", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_sloppy_crc": "false", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_splice": "false", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_split_multiple": "2", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_split_rand_factor": "20", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_update_to": "1000", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-14T04:02:55.710 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_enable": "true", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_xfs_extsize": "false", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "filestore_zfs_snap": "false", 2024-09-14T04:02:55.711 INFO:teuthology.orchestra.run.smithi114.stdout: "fio_dir": "/tmp/fio", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fsid": "63f4e762-724a-11ef-bceb-c7b262605968", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_allow_other": "true", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_atomic_o_trunc": "true", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_big_writes": "true", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_debug": "false", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_default_permissions": "false", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_disable_pagecache": "false", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_max_write": "0", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_multithreaded": "true", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_require_active_mds": "true", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_set_user_groups": "true", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_splice_move": "true", 2024-09-14T04:02:55.712 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_splice_read": "true", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_splice_write": "true", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "fuse_use_invalidate_cb": "true", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "gss_target_name": "ceph", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_file": "", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_inject_failure": "0", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "heartbeat_interval": "5", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "host": "smithi114", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-14T04:02:55.713 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "inject_early_sigterm": "false", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "jaeger_agent_port": "6799", 2024-09-14T04:02:55.714 INFO:teuthology.orchestra.run.smithi114.stdout: "jaeger_tracing_enable": "false", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_aio": "true", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_align_min_size": "65536", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_block_align": "true", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_block_size": "4096", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_dio": "true", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_discard": "false", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_force_aio": "false", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_ignore_corruption": "false", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_max_write_bytes": "10485760", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_max_write_entries": "100", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_replay_from": "0", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-14T04:02:55.715 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_write_header_frequency": "0", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "journal_zero_on_create": "false", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "journaler_prefetch_periods": "10", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "journaler_prezero_periods": "5", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "journaler_write_head_interval": "15", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "key": "", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "keyfile": "", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_backend": "rocksdb", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_default_stripe_size": "65536", 2024-09-14T04:02:55.716 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_fsck_on_mount": "false", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_max_bytes": "67108864", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_max_ops": "512", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_nid_prealloc": "1024", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_onode_map_size": "1024", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_sync_submit_transaction": "false", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "kstore_sync_transaction": "false", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "librados_thread_count": "2", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "lockdep": "false", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "lockdep_force_backtrace": "false", 2024-09-14T04:02:55.717 INFO:teuthology.orchestra.run.smithi114.stdout: "log_coarse_timestamps": "true", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_flush_on_exit": "false", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_graylog_host": "127.0.0.1", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_graylog_port": "12201", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_max_new": "1000", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_max_recent": "10000", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_stderr_prefix": "", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_stop_at_utilization": "0.970000", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": "true", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_graylog": "false", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_journald": "false", 2024-09-14T04:02:55.718 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_stderr": "false", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_syslog": "false", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "max_rotating_auth_attempts": "10", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_action_on_write_error": "1", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_alternate_name_max": "8192", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_asio_thread_count": "2", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_export_pin": "true", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_fragment_dirs": "true", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_fragment_interval": "5", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-14T04:02:55.719 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_interval": "10", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_max": "-1", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_max_until": "-1", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_merge_size": "50", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_midchunk": "0.300000", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_min_start": "0.200000", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_minchunk": "0.001000", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_mode": "0", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_need_max": "1.200000", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_need_min": "0.800000", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_overload_epochs": "2", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-14T04:02:55.720 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_split_bits": "3", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_split_size": "10000", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_target_decay": "10.000000", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_beacon_grace": "15.000000", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_beacon_interval": "4.000000", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_mid": "0.700000", 2024-09-14T04:02:55.721 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_quiesce_delay": "0", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_release_free_interval": "10", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_reservation": "0.050000", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_trim_interval": "1", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache_trim_threshold": "262144", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_client_prealloc_inos": "1000", 2024-09-14T04:02:55.722 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_connect_bootstrapping": "false", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_damage_table_max_entries": "10000", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_debug_auth_pins": "false", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_debug_frag": "false", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_debug_scatterstat": "false", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_debug_subtrees": "false", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_decay_halflife": "5.000000", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_default_dir_hash": "2", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_defer_session_stale": "true", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_deny_all_reconnect": "false", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dir_keys_per_op": "16384", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dir_max_commit_size": "10", 2024-09-14T04:02:55.723 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dir_max_entries": "0", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dir_prefetch": "true", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dump_cache_on_map": "false", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_early_reply": "true", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_enable_op_tracker": "true", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_enforce_unique_name": "true", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_export_ephemeral_random": "true", 2024-09-14T04:02:55.724 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_health_summarize_threshold": "10", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_health_dummy": "false", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-14T04:02:55.725 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_migrator_session_race": "false", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_join_fs": "", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_journal_format": "1", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_create_at": "0", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_dirfrag_at": "0", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_export_at": "0", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_import_at": "0", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_journal_at": "0", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_journal_expire_at": "0", 2024-09-14T04:02:55.726 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_journal_replay_at": "0", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_link_at": "0", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_mdstable_at": "0", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_openc_at": "0", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_rename_at": "0", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_kill_shutdown_at": "0", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_event_large_threshold": "524288", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_events_per_segment": "1024", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_max_events": "-1", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_max_segments": "128", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_pause": "false", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_segment_size": "0", 2024-09-14T04:02:55.727 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_trim_threshold": "128", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log_warn_factor": "2.000000", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_caps_per_client": "1048576", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_completed_flushes": "100000", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_completed_requests": "100000", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_export_size": "20971520", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_file_recover": "32", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_purge_files": "64", 2024-09-14T04:02:55.728 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_purge_ops": "8192", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_max_snaps_per_dir": "100", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_metrics_update_interval": "2", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_min_caps_per_client": "100", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_min_caps_working_set": "10000", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_numa_node": "-1", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_complaint_time": "30.000000", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_history_duration": "600", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_history_size": "20", 2024-09-14T04:02:55.729 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_history_slow_op_size": "20", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_op_log_threshold": "5", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_ping_grace": "15", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_ping_interval": "5", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_max_caps": "30000", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_recall_warning_threshold": "262144", 2024-09-14T04:02:55.730 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_replay_interval": "1.000000", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_root_ino_gid": "0", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_root_ino_uid": "0", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_scrub_stats_review_period": "1", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-14T04:02:55.731 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_shutdown_check": "0", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_skip_ino": "0", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_snap_max_uid": "4294967294", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_snap_min_uid": "0", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_snap_rstat": "false", 2024-09-14T04:02:55.732 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_standby_replay_damaged": "false", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_symlink_recovery": "true", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_thrash_exports": "0", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_thrash_fragments": "0", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_tick_interval": "5.000000", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_valgrind_exit": "false", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_verify_backtrace": "1", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_verify_scatter": "false", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_wipe_sessions": "false", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool_debug": "false", 2024-09-14T04:02:55.733 INFO:teuthology.orchestra.run.smithi114.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "memstore_device_bytes": "1073741824", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "memstore_page_set": "false", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "memstore_page_size": "65536", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_client_bytes": "134217728", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_client_messages": "512", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_disabled_modules": "", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_enable_op_tracker": "true", 2024-09-14T04:02:55.734 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_max_pg_creating": "1024", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_max_pg_num_change": "128", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_mds_bytes": "134217728", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_mds_messages": "128", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_mon_bytes": "134217728", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_mon_messages": "128", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_history_duration": "600", 2024-09-14T04:02:55.735 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_history_size": "20", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_op_log_threshold": "5", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_osd_bytes": "536870912", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_osd_messages": "8192", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_pool": "true", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_standby_modules": "true", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_stats_period": "5", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_stats_threshold": "5", 2024-09-14T04:02:55.736 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_tick_period": "2", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_allow_pool_delete": "true", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_allow_pool_size_one": "false", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_auth_validate_all_caps": "true", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_bytes": "104857600", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_directed_command_retry": "2", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-14T04:02:55.737 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_hunt_parallel": "3", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_log_interval": "1.000000", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_ping_interval": "10.000000", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_client_target_rank": "-1", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_level": "debug", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_file": "true", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-14T04:02:55.738 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-14T04:02:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-14T04:02:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_journald": "true", 2024-09-14T04:02:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-14T04:02:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-14T04:02:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-14T04:02:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_compact_on_bootstrap": "false", 2024-09-14T04:02:55.739 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_compact_on_start": "false", 2024-09-14T04:02:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_compact_on_trim": "true", 2024-09-14T04:02:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-14T04:02:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-14T04:02:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-14T04:02:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_cpu_threads": "4", 2024-09-14T04:02:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_crush_min_required_version": "hammer", 2024-09-14T04:02:55.741 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_daemon_bytes": "419430400", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_data_avail_crit": "5", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_data_avail_warn": "5", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_data_size_warn": "16106127360", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_dump_json": "false", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_dump_transactions": "false", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_extra_checks": "false", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-14T04:02:55.742 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_no_require_reef": "false", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_no_require_squid": "false", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_down_mkfs_grace": "60", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_down_uptime_grace": "60", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_election_default_strategy": "3", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_election_timeout": "5.000000", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_elector_ping_divisor": "2", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-14T04:02:55.743 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_enable_op_tracker": "true", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_fake_pool_delete": "false", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_force_quorum_join": "false", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_globalid_prealloc": "10000", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_detail_to_clog": "true", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_log_update_period": "5", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_max_detail": "50", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_to_clog": "true", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_to_clog_interval": "600", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_host": "[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] [v2:172.21.15.114:3301/0,v1:172.21.15.114:6790/0]", 2024-09-14T04:02:55.744 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_host_override": "", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_initial_members": "", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_lease": "5.000000", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_log_full_interval": "50", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_log_max": "10000", 2024-09-14T04:02:55.745 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_log_max_summary": "50", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_log_epochs": "500", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_osd": "10000", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_pg_per_osd": "10000", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_pool_pg_num": "65536", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mds_force_trim_to": "0", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mds_skip_sanity": "true", 2024-09-14T04:02:55.746 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_memory_autotune": "true", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_memory_target": "2147483648", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_beacon_grace": "30", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_digest_period": "5", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_inactive_grace": "60", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_min_osdmap_epochs": "500", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_complaint_time": "30", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_history_duration": "600", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_history_size": "20", 2024-09-14T04:02:55.747 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_history_slow_op_size": "20", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_op_log_threshold": "5", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_auto_mark_in": "false", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_cache_size": "500", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-14T04:02:55.748 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_down_out_interval": "600", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_force_trim_to": "0", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_min_down_reporters": "2", 2024-09-14T04:02:55.749 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-14T04:02:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-14T04:02:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-14T04:02:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-14T04:02:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-14T04:02:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-14T04:02:55.750 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_report_timeout": "900", 2024-09-14T04:02:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-14T04:02:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-14T04:02:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-14T04:02:55.751 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_stuck_threshold": "60", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-14T04:02:55.752 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_probe_timeout": "2.000000", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_reweight_max_change": "0.050000", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_reweight_max_osds": "4", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_scrub_interval": "86400", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_scrub_max_keys": "100", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_scrub_timeout": "300", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_session_timeout": "300", 2024-09-14T04:02:55.753 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_smart_report_timeout": "5", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_stat_smooth_intervals": "6", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_stretch_pool_min_size": "2", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_stretch_pool_size": "4", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_debug": "false", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_provider_kill_at": "0", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_requester_kill_at": "0", 2024-09-14T04:02:55.754 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_sync_timeout": "60.000000", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_target_pg_per_osd": "100", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_tick_interval": "5", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_timecheck_interval": "300.000000", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_use_min_delay_socket": "false", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_older_version_delay": "604800", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-14T04:02:55.755 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_misplaced": "false", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_older_version": "true", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-14T04:02:55.756 INFO:teuthology.orchestra.run.smithi114.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "monmap": "", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "motr_tracing_enabled": "false", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_op_threads": "3", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_cm": "false", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_device_name": "", 2024-09-14T04:02:55.757 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_dscp": "96", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_local_gid": "", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_port_num": "1", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_sl": "3", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_support_srq": "true", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_rdma_type": "ib", 2024-09-14T04:02:55.758 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_async_reap_threshold": "5", 2024-09-14T04:02:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_before_connect": "false", 2024-09-14T04:02:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_ipv4": "true", 2024-09-14T04:02:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_ipv6": "false", 2024-09-14T04:02:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_msgr1": "true", 2024-09-14T04:02:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_msgr2": "true", 2024-09-14T04:02:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_port_max": "7568", 2024-09-14T04:02:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_port_min": "6800", 2024-09-14T04:02:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-14T04:02:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_retry_count": "3", 2024-09-14T04:02:55.759 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_bind_retry_delay": "5", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_blackhole_client": "false", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_blackhole_mds": "false", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_blackhole_mgr": "false", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_blackhole_mon": "false", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_blackhole_osd": "false", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_client_mode": "crc secure", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_cluster_mode": "crc secure", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_cluster_type": "", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_compress_secure": "false", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_connection_idle_timeout": "900", 2024-09-14T04:02:55.760 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_connection_ready_timeout": "10", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_crc_data": "true", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_crc_header": "true", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_die_on_bad_msg": "false", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_die_on_bug": "true", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_die_on_old_message": "true", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_die_on_skipped_message": "false", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_coremask": "0xF", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_enable_tso": "true", 2024-09-14T04:02:55.761 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_hugepages": "", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_lro": "true", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_memory_channel": "4", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_pmd": "", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_port_id": "0", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-14T04:02:55.762 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_dump_on_send": "false", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_initial_backoff": "0.200000", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_delay_max": "1.000000", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_delay_type": "", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_network_congestion": "0", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_inject_socket_failures": "0", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_learn_addr_from_peer": "true", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_max_accept_failures": "4", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_max_backoff": "15.000000", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_mon_client_mode": "secure crc", 2024-09-14T04:02:55.763 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_mon_service_mode": "secure crc", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_osd_compress_min_size": "1024", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_osd_compress_mode": "none", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_pq_min_cost": "65536", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_public_type": "", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_service_mode": "crc secure", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_tcp_listen_backlog": "512", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_tcp_nodelay": "true", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-14T04:02:55.764 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_tcp_rcvbuf": "0", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "ms_type": "async+posix", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "no_config_file": "false", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_completion_locks_per_session": "32", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_inflight_ops": "1024", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_tick_interval": "5.000000", 2024-09-14T04:02:55.765 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter_timeout": "10.000000", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "objectstore_blackhole": "false", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "openssl_engine_opts": "", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_delay_time": "5.000000", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_hist_halflife": "1000", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_max_low_ops": "2", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_max_ops": "4", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_agent_slop": "0.020000", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-14T04:02:55.766 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_asio_thread_count": "2", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_async_recovery_min_cost": "100", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backfill_scan_max": "512", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backfill_scan_min": "64", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backoff_on_degraded": "false", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backoff_on_peering": "false", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_backoff_on_unfound": "true", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_beacon_report_interval": "300", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bench_duration": "30", 2024-09-14T04:02:55.767 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bench_max_block_size": "67108864", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_blkin_trace_all": "false", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_check_for_log_corruption": "false", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-14T04:02:55.768 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_class_update_on_start": "true", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_client_message_cap": "256", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_client_message_size_cap": "524288000", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_client_op_priority": "63", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_client_watch_timeout": "30", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_command_max_records": "256", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_compact_on_start": "false", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_crush_update_on_start": "true", 2024-09-14T04:02:55.769 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_crush_update_weight_set": "true", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_feed_pullee": "-1", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_misdirected_ops": "false", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_no_acting_change": "false", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_no_purge_strays": "false", 2024-09-14T04:02:55.770 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_op_order": "false", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_shutdown": "false", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_trim_objects": "false", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_verify_snaps": "false", 2024-09-14T04:02:55.771 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_keys": "1024", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_stride": "524288", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_default_notify_timeout": "30", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_delete_sleep": "0.000000", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-14T04:02:55.772 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-14T04:02:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-14T04:02:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_discard_disconnected_ops": "true", 2024-09-14T04:02:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_ec_partial_reads": "true", 2024-09-14T04:02:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_enable_op_tracker": "true", 2024-09-14T04:02:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-14T04:02:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-14T04:02:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-14T04:02:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-14T04:02:55.773 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_fast_info": "true", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_fast_shutdown": "true", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_function_tracing": "false", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_grace": "20", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_interval": "6", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_min_peers": "10", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_min_size": "2000", 2024-09-14T04:02:55.774 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_stale": "600", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_hit_set_max_size": "100000", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_hit_set_min_size": "1000", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_journal_size": "5120", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_kill_backfill_at": "0", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-14T04:02:55.775 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_map_cache_size": "50", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_map_dedup": "true", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_map_message_max": "40", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_map_share_max_epochs": "40", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_attr_name_len": "100", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_attr_size": "0", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_backfills": "1", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_markdown_count": "5", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_markdown_period": "600", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_object_name_len": "2048", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_object_namespace_len": "256", 2024-09-14T04:02:55.776 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_object_size": "134217728", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_pg_blocked_by": "16", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_pg_log_entries": "10000", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_pgls": "1024", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_push_cost": "8388608", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_push_objects": "10", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_scrubs": "3", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_trimming_pgs": "2", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_write_op_reply_len": "64", 2024-09-14T04:02:55.777 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_max_write_size": "90", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_profile": "balanced", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-14T04:02:55.778 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_base": "805306368", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_cache_min": "134217728", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_target": "4294967296", 2024-09-14T04:02:55.779 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_target_autotune": "false", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_min_pg_log_entries": "250", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_min_recovery_priority": "0", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mon_report_interval": "5", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_num_cache_shards": "32", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_num_op_tracker_shard": "32", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_numa_auto_affinity": "true", 2024-09-14T04:02:55.780 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_numa_node": "-1", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_numa_prefer_iface": "true", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_objecter_finishers": "1", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_objectstore": "bluestore", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_objectstore_fuse": "false", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_objectstore_tracing": "false", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_complaint_time": "30.000000", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_history_duration": "600", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_history_size": "20", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_history_slow_op_size": "20", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-14T04:02:55.781 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_log_threshold": "5", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_shards": "0", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_shards_hdd": "5", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_shards_ssd": "8", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_pq_min_cost": "65536", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_queue_cut_off": "high", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_op_thread_timeout": "15", 2024-09-14T04:02:55.782 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_open_classes_on_start": "true", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_os_flags": "0", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_peering_op_priority": "255", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_delete_cost": "1048576", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_delete_priority": "5", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_log_trim_max": "10000", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_log_trim_min": "100", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-14T04:02:55.783 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_crimson": "false", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-14T04:02:55.784 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_flags": "0", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_min_size": "0", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_pg_num": "32", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_pgp_num": "0", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_read_ratio": "70", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_size": "3", 2024-09-14T04:02:55.785 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_default_type": "replicated", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_push_per_object_cost": "1000", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recover_clone_overlap": "true", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_cost": "20971520", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_active": "0", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-14T04:02:55.786 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_max_single_start": "1", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_op_priority": "3", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_priority": "5", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_sleep": "0.000000", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_repair_during_recovery": "false", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_requested_scrub_priority": "120", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-14T04:02:55.787 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_auto_repair": "false", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_begin_hour": "0", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_begin_week_day": "0", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_chunk_max": "25", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_chunk_min": "5", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_cost": "52428800", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_during_recovery": "false", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_end_hour": "0", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_end_week_day": "0", 2024-09-14T04:02:55.788 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_event_cost": "4096", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_invalid_stats": "true", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_max_preemptions": "5", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_priority": "5", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_retry_delay": "30", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-14T04:02:55.789 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_retry_trimming": "10", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_scrub_sleep": "0.000000", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_skip_data_digest": "false", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_smart_report_timeout": "5", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_cost": "1048576", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_priority": "5", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-14T04:02:55.790 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_target_transaction_size": "30", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-14T04:02:55.791 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_tracing": "false", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_use_stale_snap": "false", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "osdc_blkin_trace_all": "false", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_kill_at": "0", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_max_join_drift": "10", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_min": "500", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_min_wait": "0.050000", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_propose_interval": "1.000000", 2024-09-14T04:02:55.792 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_service_trim_max": "500", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_service_trim_min": "250", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_stash_full_interval": "25", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_trim_max": "500", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "paxos_trim_min": "250", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "perf": "true", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "pid_file": "", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addr": "-", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "public_addrv": "", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "public_bind_addr": "-", 2024-09-14T04:02:55.793 INFO:teuthology.orchestra.run.smithi114.stdout: "public_network": "172.21.0.0/20", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "public_network_interface": "", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "qat_compressor_busy_polling": "false", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "qat_compressor_enabled": "false", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "qat_compressor_session_max_number": "256", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "rados_mon_op_timeout": "0", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "rados_osd_op_timeout": "0", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "rados_tracing": "false", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_atime_update_interval": "60", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_balance_parent_reads": "false", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_balance_snap_reads": "false", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_blkin_trace_all": "false", 2024-09-14T04:02:55.794 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache": "true", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_policy": "writearound", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_size": "33554432", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_clone_copy_on_read": "false", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_compression_hint": "none", 2024-09-14T04:02:55.795 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_concurrent_management_ops": "10", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_clone_format": "auto", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_data_pool": "", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_features": "61", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_format": "2", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_map_options": "", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_order": "22", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_pool": "rbd", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_stripe_count": "0", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_default_stripe_unit": "0", 2024-09-14T04:02:55.796 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_enable_alloc_hint": "true", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_io_scheduler": "simple", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-14T04:02:55.797 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_order": "24", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_pool": "", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_journal_splay_width": "4", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_localize_parent_reads": "false", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_localize_snap_reads": "false", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-14T04:02:55.798 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-14T04:02:55.799 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_mtime_update_interval": "60", 2024-09-14T04:02:55.800 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_non_blocking_aio": "true", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_op_thread_timeout": "60", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_op_threads": "1", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_parent_cache_enabled": "false", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_plugins": "", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_bps_burst": "0", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_bps_limit": "0", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_exclude_ops": "0", 2024-09-14T04:02:55.801 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_iops_burst": "0", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_iops_limit": "0", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-14T04:02:55.802 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_read_from_replica_policy": "default", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_skip_partial_discard": "true", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_tracing": "false", 2024-09-14T04:02:55.803 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_validate_names": "true", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_validate_pool": "true", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "restapi_base_url": "", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "restapi_log_level": "", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_acl_grants_max_num": "100", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_admin_entry": "admin", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_asio_assert_yielding": "false", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_backend_store": "rados", 2024-09-14T04:02:55.804 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_barbican_url": "", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_beast_enable_async": "true", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_counters_cache": "false", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_cache_enabled": "true", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_cache_expiry_interval": "900", 2024-09-14T04:02:55.805 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_cache_lru_size": "25000", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_config_store": "rados", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_content_length_compat": "false", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_copy_obj_progress": "true", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_cors_rules_max_num": "100", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_cross_domain_policy": "", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_addr": "", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-14T04:02:55.806 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_password": "", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_kmip_username": "", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_require_ssl": "true", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-14T04:02:55.807 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_addr": "", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_auth": "token", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_namespace": "", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_prefix": "", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-14T04:02:55.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_token_file": "", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_curl_buffersize": "524288", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_curl_low_speed_time": "300", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-14T04:02:55.809 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_log_changes_size": "1000", 2024-09-14T04:02:55.810 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_log_num_shards": "128", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_log_window": "30", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-14T04:02:55.811 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_disable_s3select": "false", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-14T04:02:55.812 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dns_name": "", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dns_s3website_name": "", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dynamic_resharding": "true", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_gc_threads": "true", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_lc_threads": "true", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_mdsearch": "true", 2024-09-14T04:02:55.813 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_ops_log": "false", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_quota_threads": "true", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_static_website": "false", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enable_usage_log": "false", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_enforce_swift_acls": "true", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_exit_timeout_secs": "120", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_expose_bucket": "false", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_extended_http_attrs": "", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_filter": "none", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_frontends": "beast port=7480", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-14T04:02:55.814 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_deferred": "50", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_objs": "32", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_gc_processor_period": "3600", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_init_timeout": "300", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-14T04:02:55.815 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_kafka_connection_idle": "300", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_domain": "", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_password": "", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_password_path": "", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_project": "", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_tenant": "", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_token": "", 2024-09-14T04:02:55.816 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_token_path": "", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_admin_user": "", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_barbican_domain": "", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_barbican_password": "", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_barbican_project": "", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_barbican_user": "", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_url": "", 2024-09-14T04:02:55.817 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_debug_interval": "-1", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_lock_max_time": "90", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_max_objs": "32", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_max_rules": "1000", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_max_worker": "3", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lc_thread_delay": "0", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_dnattr": "uid", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_searchfilter": "", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ldap_uri": "", 2024-09-14T04:02:55.818 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_log_http_headers": "", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_log_object_name_utc": "false", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_attr_name_len": "0", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_attr_size": "0", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-14T04:02:55.819 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_chunk_size": "4194304", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_listing_results": "5000", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_notify_retries": "10", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_put_param_size": "1048576", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_put_size": "5368709120", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_max_slo_entries": "1000", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_md_log_max_shards": "64", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-14T04:02:55.820 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_mp_lock_max_time": "600", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_max_gc": "300", 2024-09-14T04:02:55.821 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_num_async_rados_threads": "32", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_num_control_oids": "8", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_numa_node": "-1", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_objexp_chunk_size": "100", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_objexp_gc_interval": "600", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-14T04:02:55.822 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_op_thread_timeout": "600", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_op_tracing": "false", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_opa_token": "", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_opa_url": "", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_opa_verify_ssl": "true", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ops_log_rados": "false", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_ops_log_socket_path": "", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-14T04:02:55.823 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_period_push_interval": "2.000000", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_cache_lanes": "3", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_cache_partitions": "3", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_print_continue": "true", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-14T04:02:55.824 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_rados_tracing": "false", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_realm": "", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_realm_id": "", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_region": "", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-14T04:02:55.825 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_request_uri": "", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_batch_size": "64", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_max_aio": "128", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_num_logs": "16", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshard_thread_interval": "600", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_resolve_cname": "false", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-14T04:02:55.826 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_run_sync_thread": "true", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_scheduler_type": "throttler", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_script_uri": "", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_service_provider_name": "", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-14T04:02:55.827 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_client_id": "", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_client_secret": "", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_entry": "sts", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_key": "sts", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_min_session_duration": "900", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sts_token_introspection_url": "", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_account_in_url": "false", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_auth_entry": "auth", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_auth_url": "", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_custom_header": "", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-14T04:02:55.828 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_need_stats": "true", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_tenant_name": "", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_token_expiration": "86400", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_url": "", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_url_prefix": "swift", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_lease_period": "120", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-14T04:02:55.829 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_thread_pool_size": "512", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_comment": "", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_createby": "", 2024-09-14T04:02:55.830 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_encoding": "", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_flag": "false", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_origin": "", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_torrent_tracker": "", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_trust_forwarded_https": "false", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_usage_max_shards": "32", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_usage_max_user_shards": "1", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_use_opa_authz": "false", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_counters_cache": "false", 2024-09-14T04:02:55.831 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_max_buckets": "1000", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_policies_max_num": "100", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_user_unique_email": "true", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_verify_ssl": "true", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zone": "", 2024-09-14T04:02:55.832 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zone_id": "", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zonegroup": "", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zonegroup_id": "", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rgwlc_auto_session_clear": "true", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_block_size": "4096", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-14T04:02:55.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_size": "536870912", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_index_type": "binary_search", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-14T04:02:55.834 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_partition_filters": "false", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_perf": "false", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "run_dir": "/var/run/ceph", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_block_create": "true", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_cache_lru_size": "67108864", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_default_max_object_size": "16777216", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-14T04:02:55.835 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_device_size": "53687091200", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_full_integrity_check": "false", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_journal_batch_capacity": "16", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_main_device_type": "SSD", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-14T04:02:55.836 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "seastore_segment_size": "67108864", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "set_keepcaps": "false", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "setgroup": "ceph", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "setuser": "ceph", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "setuser_match_path": "", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "thp": "false", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "threadpool_default_timeout": "60", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "throttler_perf_counter": "true", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "tmp_dir": "/tmp", 2024-09-14T04:02:55.837 INFO:teuthology.orchestra.run.smithi114.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-14T04:02:55.838 INFO:teuthology.orchestra.run.smithi114.stdout: "uadk_compressor_enabled": "false", 2024-09-14T04:02:55.838 INFO:teuthology.orchestra.run.smithi114.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-14T04:02:55.838 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:02:56.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:55 smithi114 ceph-mon[103029]: pgmap v499: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:56.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:55 smithi114 ceph-mon[106969]: pgmap v499: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:55 smithi160 ceph-mon[88407]: pgmap v499: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:56.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-14T04:02:57.008 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:58.109 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:57 smithi114 ceph-mon[106969]: pgmap v500: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:58.109 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:57 smithi114 ceph-mon[103029]: pgmap v500: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:57 smithi160 ceph-mon[88407]: pgmap v500: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:02:58.622 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a: {} 2024-09-14T04:02:58.622 INFO:teuthology.orchestra.run.smithi114.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-14T04:02:58.628 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b: {} 2024-09-14T04:02:58.628 INFO:teuthology.orchestra.run.smithi114.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-14T04:02:58.636 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c: {} 2024-09-14T04:02:58.637 INFO:teuthology.orchestra.run.smithi114.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-14T04:02:58.959 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:58 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1586071030' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-14T04:02:58.959 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:58 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1586071030' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-14T04:02:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:58 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1586071030' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-14T04:02:59.259 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-09-14T04:02:59.260 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-14T04:02:59.260 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-14T04:02:59.260 INFO:tasks.radosbench:Beginning radosbench... 2024-09-14T04:02:59.260 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-14T04:02:59.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-14T04:02:59.261 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-14T04:02:59.261 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-14T04:02:59.261 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:02:59.262 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-14T04:02:59.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:02:59.262 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-14T04:02:59.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:02:59.263 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-14T04:02:59.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-14T04:02:59.263 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-14T04:02:59.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:02:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:02:59.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:02:59.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:02:59.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:02:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:02:59.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:02:59.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:02:59.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:02:59.573 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:59.575 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:59.577 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:59.577 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:02:59.589 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:02:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:02:59.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:02:59.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:02:59.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:02:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:02:59.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:02:59.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:02:59.843 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:02:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:02:59.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:02:59.944 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:02:59.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:02:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:02:59.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:00.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:02:59 smithi114 ceph-mon[103029]: pgmap v501: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:00.046 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:02:59 smithi114 ceph-mon[106969]: pgmap v501: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:02:59 smithi160 ceph-mon[88407]: pgmap v501: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:00.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:00.095 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:00.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:00.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:00.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:00.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:00.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:00.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:00.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:00.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:03:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:03:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:03:00.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:00.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:00.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:00.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:00.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:00.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:00.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:00.724 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:00.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:00.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:00.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:00.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:00.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:00.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:00.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:00.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:00.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:01.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:01.098 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:01.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:01.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:01.221 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:01.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:01.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:01.344 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:01.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:01.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:01.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:01.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:01.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:01.592 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:01.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:01.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:01.699 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:03:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:03:01] "GET /metrics HTTP/1.1" 200 34000 "" "Prometheus/2.51.0" 2024-09-14T04:03:01.716 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:01.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:01.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:01.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:01.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:01.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:01.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:01.967 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:01.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:02.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:01 smithi114 ceph-mon[103029]: pgmap v502: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:02.029 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:01 smithi114 ceph-mon[106969]: pgmap v502: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:02.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:02.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:02.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:01 smithi160 ceph-mon[88407]: pgmap v502: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:02.192 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:02.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:02.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:02.267 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:03:02.268 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:03:02.288 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:03:02.288 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:03:02.303 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:03:02.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:03:02.320 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:03:02.320 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:03:02.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:02.336 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:03:02.336 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:03:02.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:02.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:02.352 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:03:02.352 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:03:02.368 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:03:02.368 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:03:02.385 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:03:02.385 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:03:02.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:02.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:02.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:02.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:02.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:02.596 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:02.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:02.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:02.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:02.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:02.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:02.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:02.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:02.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:02.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:02.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:03.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:03.097 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:03.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:02 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/819909940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:02 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/819909940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:02 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/819909940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:03.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:03.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:03.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:03.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:03.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:03.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:03.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:03.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:03.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:03.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:03.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:03.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:03.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:03.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:03.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:03.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:03 smithi114 ceph-mon[106969]: pgmap v503: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:03.785 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:03 smithi114 ceph-mon[103029]: pgmap v503: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:03.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:03.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:03.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:03.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:03.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:03.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:04.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:04.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:04.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:03 smithi160 ceph-mon[88407]: pgmap v503: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:04.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:04.219 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:04.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:04.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:04.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:04.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:04.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:04.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:04.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:04.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:04.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:04.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:04.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:04.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:04.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:04.727 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:04.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:04.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:04.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:04.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:04.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:04.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:04.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:04.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:05.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:05.101 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:05.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:05.138 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:03:05.138 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:03:05.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:05.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:05.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:05.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:05.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:05.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:05.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:05.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:05.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:05.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:05.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:05.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:05.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:05.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:05.724 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:05.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:05.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:05.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-14T04:03:05.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:03:05.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:05.875 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:05.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:05.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:05.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:05 smithi114 ceph-mon[106969]: pgmap v504: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:05.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:05 smithi114 ceph-mon[103029]: pgmap v504: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:06.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:06.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:06.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:06.127 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:06.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:05 smithi160 ceph-mon[88407]: pgmap v504: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:06.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:06.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:06.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:06.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:06.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:06.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:06.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:06.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:06.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:06.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:06.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:06.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:06.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:06.725 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_4' created 2024-09-14T04:03:06.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:06.764 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:06.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:06.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:06.887 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:06.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:06.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:06.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:07.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:07.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:07.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/426400776' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-14T04:03:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:06 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-14T04:03:07.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/426400776' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-14T04:03:07.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:06 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-14T04:03:07.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:07.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:07.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:07.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/426400776' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-14T04:03:07.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:06 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-14T04:03:07.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:07.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:07.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:07.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:07.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:07.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:07.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:07.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:07.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:07.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-14T04:03:07.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:07.685 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:07.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:07.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:07.810 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:07.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:07.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:07 smithi114 ceph-mon[106969]: pgmap v505: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:07 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-14T04:03:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:07 smithi114 ceph-mon[106969]: osdmap e152: 8 total, 8 up, 8 in 2024-09-14T04:03:07.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:07 smithi114 ceph-mon[103029]: pgmap v505: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:07 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-14T04:03:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:07 smithi114 ceph-mon[103029]: osdmap e152: 8 total, 8 up, 8 in 2024-09-14T04:03:07.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:07.906 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:07.906 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:07.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:07.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:07.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:08.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:08.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:08.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:07 smithi160 ceph-mon[88407]: pgmap v505: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:07 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-14T04:03:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:07 smithi160 ceph-mon[88407]: osdmap e152: 8 total, 8 up, 8 in 2024-09-14T04:03:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:08.166 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:08.191 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:08.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:08.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:08.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:08.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:08.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:08.438 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:08.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:08.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:08.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:08.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:08.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:08.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:08.689 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:08.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:08.732 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:03:08.744 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:08 smithi114 ceph-mon[106969]: osdmap e153: 8 total, 8 up, 8 in 2024-09-14T04:03:08.745 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2575291497' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:03:08.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:08 smithi114 ceph-mon[103029]: osdmap e153: 8 total, 8 up, 8 in 2024-09-14T04:03:08.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2575291497' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:03:08.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:08.812 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:08.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:08.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:08.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:08.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:09.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:09.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:09.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:08 smithi160 ceph-mon[88407]: osdmap e153: 8 total, 8 up, 8 in 2024-09-14T04:03:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2575291497' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:03:09.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:09.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:09.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:09.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:09.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:09.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:09.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:09.440 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:09.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:09.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:09.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:09.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:09.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:09.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:09.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:09.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:09.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:09.814 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:09.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:09.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:09.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:09.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:09.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:10.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:10.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:10.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:10.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:10.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:09 smithi114 ceph-mon[103029]: pgmap v508: 89 pgs: 3 creating+peering, 13 unknown, 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:10.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:09 smithi114 ceph-mon[103029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:03:10.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:09 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:03:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:09 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2575291497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:03:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:09 smithi114 ceph-mon[103029]: osdmap e154: 8 total, 8 up, 8 in 2024-09-14T04:03:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:09 smithi114 ceph-mon[106969]: pgmap v508: 89 pgs: 3 creating+peering, 13 unknown, 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:09 smithi114 ceph-mon[106969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:03:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:09 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:03:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:09 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2575291497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:03:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:09 smithi114 ceph-mon[106969]: osdmap e154: 8 total, 8 up, 8 in 2024-09-14T04:03:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:09 smithi160 ceph-mon[88407]: pgmap v508: 89 pgs: 3 creating+peering, 13 unknown, 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:09 smithi160 ceph-mon[88407]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:03:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:09 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:03:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:09 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2575291497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:03:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:09 smithi160 ceph-mon[88407]: osdmap e154: 8 total, 8 up, 8 in 2024-09-14T04:03:10.166 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:10.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:10.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:10.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:10.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:10.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:10.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:10.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:10.432 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:10.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:10.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:10.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:10.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:10.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:03:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:03:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:03:10.679 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:10.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:10.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:10.807 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:10.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:10.909 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:10.931 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:10.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:10.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:11.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:11.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:11.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:11.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:11.157 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:11.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:11.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:11.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:11.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:11.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:11.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:11.430 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:11.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:11.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:11.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:11.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:11.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:11.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:11.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:11.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:11.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:11.686 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:03:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:03:11] "GET /metrics HTTP/1.1" 200 36290 "" "Prometheus/2.51.0" 2024-09-14T04:03:11.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:11.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:11.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:11.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:11.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:11.933 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:11.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:12.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:12.055 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:12.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:12.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:11 smithi114 ceph-mon[106969]: pgmap v510: 89 pgs: 3 creating+peering, 86 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:12.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:11 smithi114 ceph-mon[103029]: pgmap v510: 89 pgs: 3 creating+peering, 86 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:12.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:11 smithi160 ceph-mon[88407]: pgmap v510: 89 pgs: 3 creating+peering, 86 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:12.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:12.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:12.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:12.291 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:03:12.291 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:03:12.307 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:03:12.308 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:03:12.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:12.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:12.324 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:03:12.324 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:03:12.340 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:03:12.340 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:03:12.356 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:03:12.356 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:03:12.372 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:03:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:03:12.389 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:03:12.389 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:03:12.405 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:03:12.405 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:03:12.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:12.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:12.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:12.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:03:12.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:12.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:12.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:12.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:12.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:12.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:12.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:12.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:12 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2292563138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:12.784 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:12 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2292563138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:12.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:12.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:12.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:12.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:12.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:12.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:12.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:13.049 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:13.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:13.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:12 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2292563138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:13.158 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:03:13.158 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:03:13.159 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:03:13.159 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:03:13.159 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:03:13.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:13.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:13.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:13.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:13.322 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:13.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:13.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:13.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:13.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:13.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:13.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:13.575 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:13.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:13.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:13.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:13.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:13.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:13.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:13.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:13.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:13.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:13.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:13.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:14.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:14.078 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:14.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:14.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:13 smithi114 ceph-mon[103029]: pgmap v511: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:14.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:13 smithi114 ceph-mon[106969]: pgmap v511: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:13 smithi160 ceph-mon[88407]: pgmap v511: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:14.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:14.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:14.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:14.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-14T04:03:14.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:14.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:14.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:14.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:14.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:14.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:14.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:14.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:14.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:14.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:14.703 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:14.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:14.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:14.808 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:14.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:14.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:14.843 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:14.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:14.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:14.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:15.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:15.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:03:15.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:15.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:15.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:15.207 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:15.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:15.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:15.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:15.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:15.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:15.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:15.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:15.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:15.557 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:15.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:15.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:15.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:15.710 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:15.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:15.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:15.751 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-14T04:03:15.812 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:15.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:15.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:15.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:15.960 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:15.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:16.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:16.084 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:16.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:16.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:15 smithi114 ceph-mon[106969]: pgmap v512: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:16.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3008126049' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:03:16.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:15 smithi114 ceph-mon[103029]: pgmap v512: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:16.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3008126049' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:03:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:15 smithi160 ceph-mon[88407]: pgmap v512: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3008126049' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:03:16.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:16.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:16.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:16.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:16.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:16.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:16.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:16.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:16.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:16.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:16.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:16.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:16.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:16.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:16.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:16.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:16.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:16.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:16 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3008126049' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:03:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:16 smithi114 ceph-mon[106969]: osdmap e155: 8 total, 8 up, 8 in 2024-09-14T04:03:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:16 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3008126049' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:03:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:16 smithi114 ceph-mon[103029]: osdmap e155: 8 total, 8 up, 8 in 2024-09-14T04:03:16.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:16.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:16.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:17.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:17.079 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:17.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:17.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:16 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3008126049' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:03:17.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:16 smithi160 ceph-mon[88407]: osdmap e155: 8 total, 8 up, 8 in 2024-09-14T04:03:17.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:17.206 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:17.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:17.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:17.329 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:17.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:17.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:17.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:17.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:17.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:17.575 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:17.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:17.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:17.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:17.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:17.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:17.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:17.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:17.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:17.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:17 smithi114 ceph-mon[106969]: pgmap v514: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:17.924 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:17 smithi114 ceph-mon[106969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:03:17.925 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:17 smithi114 ceph-mon[103029]: pgmap v514: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:17.925 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:17 smithi114 ceph-mon[103029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:03:17.925 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:17.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:17.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:18.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:18.078 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:18.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:17 smithi160 ceph-mon[88407]: pgmap v514: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:17 smithi160 ceph-mon[88407]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:03:18.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:18.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:18.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:18.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:18.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:18.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:18.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:18.432 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:18.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:18.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:18.534 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:03:18.534 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:03:18.534 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:03:18.534 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:03:18.534 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:66447 7.9 7:935178b5:::benchmark_data_smithi114_118301_object66446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:53:48.066838+0000", 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 570.46471905099997, 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.035576831, 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 66447 2024-09-14T04:03:18.535 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.066838+0000", 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.066838+0000", 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.536 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.066839+0000", 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3399999999999999e-07 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.066858+0000", 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9247999999999999e-05 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.066865+0000", 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1350000000000003e-06 2024-09-14T04:03:18.537 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.538 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.538 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.538 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.066870+0000", 2024-09-14T04:03:18.539 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.276e-06 2024-09-14T04:03:18.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.539 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.539 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.539 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.066955+0000", 2024-09-14T04:03:18.539 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4427999999999999e-05 2024-09-14T04:03:18.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.539 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.539 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.067186+0000", 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000231499 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.067365+0000", 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017885199999999999 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.075270+0000", 2024-09-14T04:03:18.540 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0079045390000000004 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.075497+0000", 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000227402 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.086859+0000", 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011362156 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.541 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.086871+0000", 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1885999999999999e-05 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.102332+0000", 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0154605539999999 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.102415+0000", 2024-09-14T04:03:18.542 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3321999999999999e-05 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:66463 7.9 7:9be8c393:::benchmark_data_smithi114_118301_object66462:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:53:48.079784+0000", 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 570.45177301900003, 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0317069400000001, 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:03:18.543 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 66463 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.079784+0000", 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.544 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.079784+0000", 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.079785+0000", 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5099999999999999e-07 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.079804+0000", 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9378999999999999e-05 2024-09-14T04:03:18.545 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.079825+0000", 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0820000000000001e-05 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.079832+0000", 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8469999999999998e-06 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.546 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.079887+0000", 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5269999999999998e-05 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.079999+0000", 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011133699999999999 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.080117+0000", 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000118384 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.547 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.086904+0000", 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0067873969999999997 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.086912+0000", 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1820000000000006e-06 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.102715+0000", 2024-09-14T04:03:18.548 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.015802423 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.103374+0000", 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00065893199999999999 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.111404+0000", 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0080302189999999999 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.549 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.111491+0000", 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6998999999999998e-05 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:66472 7.9 7:9df3dd94:::benchmark_data_smithi114_118301_object66471:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:53:48.090451+0000", 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 570.44110638300003, 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0214144329999999, 2024-09-14T04:03:18.550 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 66472 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.090451+0000", 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.090451+0000", 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.090451+0000", 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0200000000000002e-07 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.552 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.090472+0000", 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0988e-05 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.090480+0000", 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5469999999999999e-06 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.090485+0000", 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0409999999999997e-06 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.553 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.090873+0000", 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00038770699999999999 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:48.090878+0000", 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8599999999999998e-06 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.106103+0000", 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0152242520000001 2024-09-14T04:03:18.554 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.106226+0000", 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012331399999999999 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4", 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.106381+0000", 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000154881 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.111067+0000", 2024-09-14T04:03:18.555 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004686422 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.111718+0000", 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00065050800000000005 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.111770+0000", 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2556999999999998e-05 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.556 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:53:49.111865+0000", 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4853999999999997e-05 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:91990 7.9 7:961c4d8e:::benchmark_data_smithi114_118301_object91989:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:54:08.566196+0000", 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 549.96536064899999, 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8960660279999999, 2024-09-14T04:03:18.557 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 91990 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.566196+0000", 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.558 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.566196+0000", 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.566198+0000", 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.638e-06 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.559 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.566225+0000", 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6851000000000001e-05 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.566238+0000", 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2577e-05 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.566248+0000", 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0183999999999999e-05 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.560 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.567123+0000", 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00087518999999999997 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.567251+0000", 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012853499999999999 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4", 2024-09-14T04:03:18.561 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.567401+0000", 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014959399999999999 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.570970+0000", 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035688299999999998 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.586529+0000", 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015559312 2024-09-14T04:03:18.562 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.563 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.563 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.563 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.586540+0000", 2024-09-14T04:03:18.563 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0457e-05 2024-09-14T04:03:18.563 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.563 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.563 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:09.070513+0000", 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.48397303600000002 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:09.070529+0000", 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5945999999999999e-05 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.435289+0000", 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.364760178 2024-09-14T04:03:18.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.565 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.565 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:03:18.565 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.436075+0000", 2024-09-14T04:03:18.565 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00078626499999999997 2024-09-14T04:03:18.565 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.565 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.565 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.565 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.462055+0000", 2024-09-14T04:03:18.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:18.567 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.025980328 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.462263+0000", 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020710699999999999 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:91991 7.4 7:2fe02e41:::benchmark_data_smithi114_118301_object91990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-14T04:03:18.568 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:54:08.570975+0000", 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 549.96058226100001, 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8654673370000001, 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 91991 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.569 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.570975+0000", 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.570975+0000", 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.570977+0000", 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9120000000000001e-06 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.570 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.571002+0000", 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5273999999999999e-05 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.571016+0000", 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3672999999999999e-05 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.571026+0000", 2024-09-14T04:03:18.571 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0380999999999999e-05 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.571093+0000", 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7377000000000002e-05 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.571255+0000", 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016177300000000001 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.572 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4", 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.571406+0000", 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000150913 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.584022+0000", 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.01261543 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.584035+0000", 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.329e-05 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:09.072075+0000", 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.48804006300000002 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:09.072086+0000", 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1328000000000001e-05 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.417396+0000", 2024-09-14T04:03:18.574 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3453101199999999 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.418195+0000", 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00079845899999999995 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.436302+0000", 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.018107435000000002 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.575 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.576 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.436442+0000", 2024-09-14T04:03:18.576 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013990899999999999 2024-09-14T04:03:18.576 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.576 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.576 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.576 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.576 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:91993 7.4 7:2c737d20:::benchmark_data_smithi114_118301_object91992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-14T04:03:18.576 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:54:08.577049+0000", 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 549.95450819799998, 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8604945769999999, 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 91993 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.577 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577049+0000", 2024-09-14T04:03:18.578 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.578 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.578 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.578 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.578 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577049+0000", 2024-09-14T04:03:18.578 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.578 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.578 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.578 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577051+0000", 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7570000000000001e-06 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577079+0000", 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8694e-05 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577092+0000", 2024-09-14T04:03:18.579 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2247e-05 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577102+0000", 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0575e-05 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.584537+0000", 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.007434497 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.584545+0000", 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1710000000000002e-06 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:09.072121+0000", 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.487576379 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:09.072129+0000", 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9699999999999999e-06 2024-09-14T04:03:18.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.419257+0000", 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.347128061 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.419416+0000", 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000158534 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.419595+0000", 2024-09-14T04:03:18.582 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017910800000000001 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.424063+0000", 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044681800000000004 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.426289+0000", 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022255719999999999 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.583 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.436482+0000", 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010193562 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.436631+0000", 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014920299999999999 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.437544+0000", 2024-09-14T04:03:18.584 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00091206700000000002 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:91996 7.9 7:9ff403fc:::benchmark_data_smithi114_118301_object91995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:54:08.577700+0000", 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 549.95385674099998, 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8907095759999999, 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.585 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 91996 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577700+0000", 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577700+0000", 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577702+0000", 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1480000000000001e-06 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577739+0000", 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7775999999999999e-05 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577750+0000", 2024-09-14T04:03:18.587 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0393e-05 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.577757+0000", 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3209999999999999e-06 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.586595+0000", 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0088384580000000004 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.588 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:08.586602+0000", 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2589999999999997e-06 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:09.070557+0000", 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.48395480099999999 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:09.070564+0000", 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4460000000000004e-06 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.589 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.437408+0000", 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.366844446 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.437566+0000", 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015754500000000001 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.437769+0000", 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020257000000000001 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.590 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.442275+0000", 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0045066680000000001 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.451931+0000", 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0096556469999999998 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.465921+0000", 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013990585999999999 2024-09-14T04:03:18.591 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.467151+0000", 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001229857 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:54:10.468410+0000", 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001258655 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:174757 7.e 7:78ac3b8a:::benchmark_data_smithi114_118301_object53892:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e110)", 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:58.240749+0000", 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 440.29080809599998, 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2291268790000001, 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174757 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.593 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240749+0000", 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240749+0000", 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.594 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240751+0000", 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3240000000000001e-06 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240760+0000", 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1310000000000006e-06 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240783+0000", 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3448999999999999e-05 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.595 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240793+0000", 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0349e-05 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.241613+0000", 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00082006899999999998 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.241623+0000", 2024-09-14T04:03:18.596 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2399999999999996e-06 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.247561+0000", 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059381080000000001 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.247572+0000", 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1528e-05 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.597 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.437007+0000", 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.189434678 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for missing object", 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.437122+0000", 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115493 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.469785+0000", 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.032662627 2024-09-14T04:03:18.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.599 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.599 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.469876+0000", 2024-09-14T04:03:18.599 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0883000000000004e-05 2024-09-14T04:03:18.599 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.599 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.599 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:174771 7.e 7:7ce505e3:::benchmark_data_smithi114_118301_object53906:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e110)", 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:58.240828+0000", 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 440.29072915199998, 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.229410154, 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.600 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174771 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240828+0000", 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240828+0000", 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.601 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240829+0000", 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7400000000000003e-07 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240832+0000", 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.96e-06 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240840+0000", 2024-09-14T04:03:18.602 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9929999999999995e-06 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240846+0000", 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1909999999999997e-06 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.241729+0000", 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00088277899999999999 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:03:18.603 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.241735+0000", 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2659999999999998e-06 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.247600+0000", 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0058655030000000002 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.247608+0000", 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8180000000000004e-06 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.604 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.437333+0000", 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1897249240000001 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for missing object", 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.437432+0000", 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8554000000000005e-05 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.470168+0000", 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.032736225000000001 2024-09-14T04:03:18.605 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.470238+0000", 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0266999999999997e-05 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:174797 7.e 7:7a5cd54b:::benchmark_data_smithi114_118301_object53932:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e110)", 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:58.240856+0000", 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 440.29070122600001, 2024-09-14T04:03:18.606 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2323480149999999, 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174797 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240856+0000", 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240856+0000", 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240856+0000", 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2399999999999999e-07 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.608 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240859+0000", 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.446e-06 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240865+0000", 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4520000000000002e-06 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.240871+0000", 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4859999999999997e-06 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.609 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.241761+0000", 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00088994400000000004 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.241766+0000", 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5679999999999999e-06 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.247635+0000", 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0058690399999999998 2024-09-14T04:03:18.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.247643+0000", 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.328e-06 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.437457+0000", 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1898139889999999 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.611 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.441642+0000", 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041858599999999996 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.442117+0000", 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00047493900000000001 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.458729+0000", 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016612056 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.459181+0000", 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000451481 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.472994+0000", 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013813372000000001 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.473048+0000", 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3400999999999999e-05 2024-09-14T04:03:18.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:59.473204+0000", 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000156229 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:211967 7.9 7:92402baf:::benchmark_data_smithi114_118301_object91102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:51.054154+0000", 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 387.47740267099999, 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0496998660000001, 2024-09-14T04:03:18.614 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 211967 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.054154+0000", 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.615 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.054154+0000", 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.054156+0000", 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6750000000000001e-06 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.054162+0000", 2024-09-14T04:03:18.616 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9209999999999997e-06 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.054175+0000", 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2673e-05 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.054182+0000", 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0199999999999997e-06 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.617 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.057476+0000", 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032944409999999999 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.057784+0000", 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030827399999999998 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.058137+0000", 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00035301899999999999 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.618 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.062986+0000", 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0048482530000000003 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.063901+0000", 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00091502399999999996 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.083239+0000", 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.019338663999999998 2024-09-14T04:03:18.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.083249+0000", 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7370000000000001e-06 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.103779+0000", 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.020529673 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.103854+0000", 2024-09-14T04:03:18.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5492000000000002e-05 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:211986 7.9 7:95f42357:::benchmark_data_smithi114_118301_object91121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:51.087786+0000", 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 387.44377157299999, 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.037672124, 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.621 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 211986 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.087786+0000", 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.087786+0000", 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.087787+0000", 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.739e-06 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.087794+0000", 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5889999999999999e-06 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.087807+0000", 2024-09-14T04:03:18.623 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3461e-05 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.087821+0000", 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4024e-05 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.087891+0000", 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9535000000000004e-05 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.624 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.087901+0000", 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0457e-05 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.104330+0000", 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0164285200000001 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.104444+0000", 2024-09-14T04:03:18.625 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011448 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.104565+0000", 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000120894 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.114659+0000", 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010093758 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.626 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.114671+0000", 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1608e-05 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.125335+0000", 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010664442999999999 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.125410+0000", 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5349e-05 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.627 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.125458+0000", 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7267e-05 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:211991 7.9 7:90f0c4b1:::benchmark_data_smithi114_118301_object91126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:51.088494+0000", 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 387.44306304200001, 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0371575209999999, 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.628 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 211991 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.088494+0000", 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.088494+0000", 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.088495+0000", 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4900000000000005e-07 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.088499+0000", 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2939999999999999e-06 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.088510+0000", 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0966e-05 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.088517+0000", 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9519999999999997e-06 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.088571+0000", 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3449000000000003e-05 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.088579+0000", 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7520000000000002e-06 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.109952+0000", 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0213728660000001 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.632 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.110079+0000", 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000126756 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.110193+0000", 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113749 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.114675+0000", 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044825730000000001 2024-09-14T04:03:18.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.125498+0000", 2024-09-14T04:03:18.634 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010822379 2024-09-14T04:03:18.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.125530+0000", 2024-09-14T04:03:18.634 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2039999999999998e-05 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.125616+0000", 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6650000000000006e-05 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.125652+0000", 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5246000000000001e-05 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.635 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:212002 7.9 7:94266fd7:::benchmark_data_smithi114_118301_object91137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:51.105794+0000", 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 387.425762926, 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0433045430000001, 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:03:18.636 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 212002 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.105794+0000", 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.105794+0000", 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.105795+0000", 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8000000000000004e-07 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.105799+0000", 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1130000000000003e-06 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.638 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.105819+0000", 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9374000000000001e-05 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.105825+0000", 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8920000000000003e-06 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.105892+0000", 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6687999999999996e-05 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.639 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:51.105901+0000", 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5229999999999995e-06 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.114697+0000", 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.008796241 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.640 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.114805+0000", 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000108004 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.114928+0000", 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012295000000000001 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.125697+0000", 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010769577000000001 2024-09-14T04:03:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.139749+0000", 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014051159000000001 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.148727+0000", 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0089783640000000008 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:03:18.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.149052+0000", 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00032535799999999998 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:52.149099+0000", 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6419999999999999e-05 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:70102 8.c 8:3b00d0c9:::benchmark_data_smithi114_135745_object70101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-14T04:03:18.643 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:24.321931+0000", 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 294.20962611300001, 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.033535809, 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 70102 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.644 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.321931+0000", 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.321931+0000", 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.321932+0000", 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.283e-06 2024-09-14T04:03:18.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322039+0000", 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000106449 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322052+0000", 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3379e-05 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322061+0000", 2024-09-14T04:03:18.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1430000000000004e-06 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322175+0000", 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113303 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322409+0000", 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023447600000000001 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.647 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322546+0000", 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013709700000000001 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.331065+0000", 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0085183780000000001 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.331079+0000", 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4260000000000001e-05 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.355395+0000", 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0243157780000001 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.355467+0000", 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2262999999999996e-05 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.649 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:70103 8.c 8:3ddcc4db:::benchmark_data_smithi114_135745_object70102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:24.322087+0000", 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 294.20947055300002, 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0439978780000001, 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:03:18.650 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 70103 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322087+0000", 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322087+0000", 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322087+0000", 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7000000000000004e-07 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322109+0000", 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2047000000000001e-05 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322118+0000", 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0690000000000008e-06 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.322125+0000", 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4860000000000002e-06 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.325425+0000", 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032997669999999999 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.325544+0000", 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011972399999999999 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.325684+0000", 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013988800000000001 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.331110+0000", 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054253529999999999 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.654 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.331134+0000", 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4888999999999998e-05 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.355682+0000", 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.024547536 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.356468+0000", 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00078565299999999998 2024-09-14T04:03:18.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.365994+0000", 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0095259109999999998 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.366084+0000", 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0884999999999995e-05 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:70104 8.c 8:3f16e0b2:::benchmark_data_smithi114_135745_object70103:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:24.323473+0000", 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 294.20808419700001, 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.047376898, 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 70104 2024-09-14T04:03:18.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.323473+0000", 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.323473+0000", 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.323474+0000", 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.344e-06 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.323496+0000", 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1804999999999999e-05 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.323507+0000", 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1287000000000001e-05 2024-09-14T04:03:18.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.323515+0000", 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3830000000000003e-06 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.331159+0000", 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0076443270000000002 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.331164+0000", 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3029999999999999e-06 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.356561+0000", 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.025396516 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.356675+0000", 2024-09-14T04:03:18.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0001142 2024-09-14T04:03:18.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 7", 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.356838+0000", 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000163208 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.359361+0000", 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025222589999999998 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.370614+0000", 2024-09-14T04:03:18.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011253734 2024-09-14T04:03:18.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:03:18.665 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.370759+0000", 2024-09-14T04:03:18.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000144726 2024-09-14T04:03:18.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.370850+0000", 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0805999999999998e-05 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96290 8.c 8:3a19b7ba:::benchmark_data_smithi114_135745_object96289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:44.775710+0000", 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 273.75584723499998, 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.261377945, 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.666 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96290 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.775710+0000", 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.775710+0000", 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.775712+0000", 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1789999999999998e-06 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.775738+0000", 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5622999999999998e-05 2024-09-14T04:03:18.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.775755+0000", 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6979000000000001e-05 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.775763+0000", 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7069999999999998e-06 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.775841+0000", 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7770999999999998e-05 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.775992+0000", 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015106399999999999 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 7", 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.776244+0000", 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025175099999999998 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.786391+0000", 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010146753 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.786410+0000", 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8946000000000001e-05 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.778486+0000", 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99207674099999998 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.778500+0000", 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3903999999999999e-05 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.036990+0000", 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2584895220000001 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.037088+0000", 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8004999999999994e-05 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:175855 8.c 8:33239e74:::benchmark_data_smithi114_135745_object49883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:00:16.637908+0000", 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 181.89364928399999, 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.023321272, 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:03:18.674 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 175855 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:16.637908+0000", 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:16.637908+0000", 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:16.637910+0000", 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.153e-06 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:16.637917+0000", 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7279999999999998e-06 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:16.637930+0000", 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.348e-05 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:16.637944+0000", 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3658e-05 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:16.638403+0000", 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00045869299999999998 2024-09-14T04:03:18.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:16.638412+0000", 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1120000000000002e-06 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:17.650301+0000", 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.011889053 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:17.650422+0000", 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000121425 2024-09-14T04:03:18.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:17.650555+0000", 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013287300000000001 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:17.654416+0000", 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003860732 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:17.657910+0000", 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034946370000000001 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:17.661161+0000", 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032505289999999998 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:17.661197+0000", 2024-09-14T04:03:18.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6229999999999997e-05 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:00:17.661229+0000", 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1968999999999998e-05 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:03:18.681 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:03:18.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:18.682 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:18.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:18.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:18.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:18.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:18.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:18.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:18.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:18.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:19.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:19.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:19.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:19.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:19.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:19.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:19.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:19.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:19.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:19.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:19.351 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-14T04:03:19.352 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-09-14T04:03:19.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:03:19.354 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-14T04:03:19.355 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-14T04:03:19.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:19.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:19.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:19.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:19.542 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-14T04:03:19.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-14T04:03:19.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_153893 2024-09-14T04:03:19.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:03:19.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-14T04:03:19.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:19.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:19.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:19.706 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:19.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:19.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-14T04:03:19.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:19.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:19.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:19.930 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:03:19.938 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:19.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:19.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:20.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:20.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:20.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:20.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:19 smithi114 ceph-mon[106969]: pgmap v515: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:20.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/606050864' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:03:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:19 smithi114 ceph-mon[103029]: pgmap v515: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:19 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/606050864' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:03:20.134 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:19 smithi160 ceph-mon[88407]: pgmap v515: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:19 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/606050864' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:03:20.199 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:20.210 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:20.232 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:20.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:20.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:20.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:20.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:20.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:20.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:20.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:20.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 3510 3494 218.36 218.375 0.00332007 0.00456642 2024-09-14T04:03:20.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:20.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:20.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:03:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:03:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:03:20.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:20.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:20.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:20.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:20.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:20.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:21.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:21.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:21.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:21.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:20 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:03:21.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:20 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/606050864' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:03:21.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:20 smithi114 ceph-mon[103029]: osdmap e156: 8 total, 8 up, 8 in 2024-09-14T04:03:21.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:20 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:03:21.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:20 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/606050864' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:03:21.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:20 smithi114 ceph-mon[106969]: osdmap e156: 8 total, 8 up, 8 in 2024-09-14T04:03:21.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:21.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:20 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:03:21.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:20 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/606050864' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:03:21.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:20 smithi160 ceph-mon[88407]: osdmap e156: 8 total, 8 up, 8 in 2024-09-14T04:03:21.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:21.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:21.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:21.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:21.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:21.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:21.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:21.464 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:21.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:21.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 7290 7274 227.295 236.25 0.00397817 0.00439198 2024-09-14T04:03:21.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:21.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:21.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:21.644 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:03:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:03:21] "GET /metrics HTTP/1.1" 200 36290 "" "Prometheus/2.51.0" 2024-09-14T04:03:21.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:21.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:21.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:21.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:21.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:21.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:21.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:21.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:21.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:21 smithi114 ceph-mon[106969]: pgmap v517: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:21.970 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:21 smithi114 ceph-mon[103029]: pgmap v517: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:21.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:22.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:22.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:22.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:22.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:22.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:22.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:21 smithi160 ceph-mon[88407]: pgmap v517: 89 pgs: 89 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-14T04:03:22.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:22.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:22.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:22.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:22.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:22.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:22.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:22.534 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:22.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 10876 10860 226.232 224.125 0.00407046 0.00441262 2024-09-14T04:03:22.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:22.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:22.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:22.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:22.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:22.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:22.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:22.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:22.838 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:22.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:22.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:22.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:22.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:22.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:22.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:23.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:23.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:23.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:23.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:23.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:23.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:23.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:23.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:23.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:23.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:03:23.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:23.529 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:23.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:23.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 14068 14052 219.505 199.5 0.00342687 0.0045498 2024-09-14T04:03:23.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:23.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:23.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:23.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:23.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:23 smithi160 ceph-mon[88407]: pgmap v518: 89 pgs: 89 active+clean; 122 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s wr, 242 op/s 2024-09-14T04:03:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:23.909 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:23 smithi114 ceph-mon[106969]: pgmap v518: 89 pgs: 89 active+clean; 122 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s wr, 242 op/s 2024-09-14T04:03:23.909 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:23.910 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:23.910 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:23 smithi114 ceph-mon[103029]: pgmap v518: 89 pgs: 89 active+clean; 122 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 15 MiB/s wr, 242 op/s 2024-09-14T04:03:23.910 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:23.910 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:23.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:23.994 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:23.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:24.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:24.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:24.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:24.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:24.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:24.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:24.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:24.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:24.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:24.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:24.519 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:24.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:24.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 16875 16859 210.69 175.438 0.00680924 0.00474063 2024-09-14T04:03:24.620 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:24.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:24.653 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:24.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:24.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:24.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:24.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:24.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:24.939 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:24.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:25.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:25.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:25.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:25.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:25.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:25.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:25.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:25.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:25.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:25.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:25.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:25.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:25.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 15 19644 19629 204.427 173.125 0.00226572 0.00488337 2024-09-14T04:03:25.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:25.600 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:25.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:25.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:25.738 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:25.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:25.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:25.872 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:25.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:25.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:25 smithi160 ceph-mon[88407]: pgmap v519: 89 pgs: 89 active+clean; 865 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-14T04:03:25.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:26.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:26.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:26.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:26.032 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:25 smithi114 ceph-mon[103029]: pgmap v519: 89 pgs: 89 active+clean; 865 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-14T04:03:26.032 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:25 smithi114 ceph-mon[106969]: pgmap v519: 89 pgs: 89 active+clean; 865 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-14T04:03:26.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:26.144 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:26.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:26.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:26.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:26.274 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:26.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:26.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:26.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:26.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:26.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:26.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:26.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:26.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:26.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 21559 21543 192.305 119.625 0.00452242 0.00519331 2024-09-14T04:03:26.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:26.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:26.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:26.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:26.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:26.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:26.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:26.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:26 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1012597887' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:26.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:26 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1012597887' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:26 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1012597887' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:26.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:26.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:26.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:27.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:27.098 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:27.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:27.152 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:03:27.152 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:03:27.152 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:03:27.152 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:03:27.152 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:03:27.152 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:03:27.152 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:03:27.152 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:03:27.152 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:03:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:03:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:03:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:03:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:03:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:03:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:03:27.153 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:03:27.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:27.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:27.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:27.323 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:27.356 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:27.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:27.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:27.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:27.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 23619 23603 184.36 128.75 0.00535961 0.00541867 2024-09-14T04:03:27.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:27.618 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:27.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:27.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:27.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:27.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:27.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:27.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:27.886 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:27.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:27.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:27 smithi160 ceph-mon[88407]: pgmap v520: 89 pgs: 89 active+clean; 865 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-14T04:03:27.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:28.020 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:28.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:28.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:27 smithi114 ceph-mon[103029]: pgmap v520: 89 pgs: 89 active+clean; 865 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-14T04:03:28.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:27 smithi114 ceph-mon[106969]: pgmap v520: 89 pgs: 89 active+clean; 865 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-14T04:03:28.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:28.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:28.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:28.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:28.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:28.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:28.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:28.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:28.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:28.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:28.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:28.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 26270 26254 182.284 165.688 0.00507666 0.00548033 2024-09-14T04:03:28.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:28.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:28.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:28.698 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:28.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:28.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:28.837 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:28.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:28.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:28.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:28.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:29.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:29.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:29.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:29.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:29.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:29.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:29.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:29.371 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:29.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:29.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:29.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:29.509 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:29.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:29.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 28523 28507 178.136 140.812 0.00451369 0.00560882 2024-09-14T04:03:29.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:29.643 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:29.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:29.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:29.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:29.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:29.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:29.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:29.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:29.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:30.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:30.051 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:30.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:30.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:30.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:30.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:30.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:30.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:30.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:30.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:30.466 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:30.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:30.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 30783 30767 174.782 141.25 0.00427759 0.00571647 2024-09-14T04:03:30.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:30.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:30.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:30.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:30.609 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:03:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:03:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:03:30.609 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:30 smithi160 ceph-mon[88407]: pgmap v521: 89 pgs: 89 active+clean; 1017 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-14T04:03:30.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:30 smithi114 ceph-mon[103029]: pgmap v521: 89 pgs: 89 active+clean; 1017 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-14T04:03:30.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:30 smithi114 ceph-mon[106969]: pgmap v521: 89 pgs: 89 active+clean; 1017 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-14T04:03:30.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:30.730 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:30.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:30.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:30.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:30.867 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:30.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:30.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:30.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:31.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:31.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:31.139 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:31.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:31.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:31.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:31.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:31.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:31.394 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:03:31.396 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:03:31.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:31.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:31.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:31.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:31.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:31.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:31.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 33016 33000 171.846 139.562 0.0100223 0.00581388 2024-09-14T04:03:31.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:31.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:31.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:31.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:31.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:31.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:31.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:03:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:03:31] "GET /metrics HTTP/1.1" 200 36414 "" "Prometheus/2.51.0" 2024-09-14T04:03:31.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:31.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:31.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:31.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:32.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:32.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:32.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:32.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:32.219 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:32.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:32.321 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:32.360 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:32.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:32.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:32.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:32.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:32.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 35166 35150 168.963 134.375 0.0055694 0.00591406 2024-09-14T04:03:32.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:32.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:32 smithi114 ceph-mon[103029]: pgmap v522: 89 pgs: 89 active+clean; 1.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-09-14T04:03:32.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:32 smithi114 ceph-mon[103029]: from='client.35688 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:03:32.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:32 smithi114 ceph-mon[106969]: pgmap v522: 89 pgs: 89 active+clean; 1.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-09-14T04:03:32.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:32 smithi114 ceph-mon[106969]: from='client.35688 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:03:32.636 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:32.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:32 smithi160 ceph-mon[88407]: pgmap v522: 89 pgs: 89 active+clean; 1.6 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-09-14T04:03:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:32 smithi160 ceph-mon[88407]: from='client.35688 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:03:32.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:32.767 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:32.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:32.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:32.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:32.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:32.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:33.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:33.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:33.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:33.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:33.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:33.245 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-14T04:03:30.605991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1716787149,"num_objects":26213,"num_object_clones":0,"num_object_copies":78639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26213,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":26399,"num_write_kb":1677627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26345,"ondisk_log_size":26345,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8393148,"kb_used_data":4771452,"kb_used_omap":29,"kb_used_meta":3621474,"kb_avail":741568068,"statfs":{"total":767960285184,"available":759365701632,"internally_reserved":0,"allocated":4885966848,"data_stored":4879926026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30187,"internal_metadata":3708389909},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1716191232,"num_objects":26187,"num_object_clones":0,"num_object_copies":78561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26187,"num_write_kb":1675968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.790210"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":117,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.827595+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:03:19.827595+0000","last_peered":"2024-09-14T04:03:19.827595+0000","last_clean":"2024-09-14T04:03:19.827595+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:03:19.827595+0000","last_undegraded":"2024-09-14T04:03:19.827595+0000","last_fullsized":"2024-09-14T04:03:19.827595+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":123,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.780949+0000","last_change":"2024-09-14T03:55:58.245884+0000","last_active":"2024-09-14T04:03:16.780949+0000","last_peered":"2024-09-14T04:03:16.780949+0000","last_clean":"2024-09-14T04:03:16.780949+0000","last_became_active":"2024-09-14T03:55:58.245547+0000","last_became_peered":"2024-09-14T03:55:58.245547+0000","last_unstale":"2024-09-14T04:03:16.780949+0000","last_undegraded":"2024-09-14T04:03:16.780949+0000","last_fullsized":"2024-09-14T04:03:16.780949+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:30:10.818639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":121,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.754423+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:03:15.754423+0000","last_peered":"2024-09-14T04:03:15.754423+0000","last_clean":"2024-09-14T04:03:15.754423+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:03:15.754423+0000","last_undegraded":"2024-09-14T04:03:15.754423+0000","last_fullsized":"2024-09-14T04:03:15.754423+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":187,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.819531+0000","last_change":"2024-09-14T03:58:45.794738+0000","last_active":"2024-09-14T04:03:19.819531+0000","last_peered":"2024-09-14T04:03:19.819531+0000","last_clean":"2024-09-14T04:03:19.819531+0000","last_became_active":"2024-09-14T03:58:45.794111+0000","last_became_peered":"2024-09-14T03:58:45.794111+0000","last_unstale":"2024-09-14T04:03:19.819531+0000","last_undegraded":"2024-09-14T04:03:19.819531+0000","last_fullsized":"2024-09-14T04:03:19.819531+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:12:49.686765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":187,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.820600+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:03:19.820600+0000","last_peered":"2024-09-14T04:03:19.820600+0000","last_clean":"2024-09-14T04:03:19.820600+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:03:19.820600+0000","last_undegraded":"2024-09-14T04:03:19.820600+0000","last_fullsized":"2024-09-14T04:03:19.820600+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":104,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.780792+0000","last_change":"2024-09-14T03:43:11.269452+0000","last_active":"2024-09-14T04:03:16.780792+0000","last_peered":"2024-09-14T04:03:16.780792+0000","last_clean":"2024-09-14T04:03:16.780792+0000","last_became_active":"2024-09-14T03:43:11.269185+0000","last_became_peered":"2024-09-14T03:43:11.269185+0000","last_unstale":"2024-09-14T04:03:16.780792+0000","last_undegraded":"2024-09-14T04:03:16.780792+0000","last_fullsized":"2024-09-14T04:03:16.780792+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:53:12.602272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":116,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.827697+0000","last_change":"2024-09-14T03:58:45.819617+0000","last_active":"2024-09-14T04:03:19.827697+0000","last_peered":"2024-09-14T04:03:19.827697+0000","last_clean":"2024-09-14T04:03:19.827697+0000","last_became_active":"2024-09-14T03:58:45.819034+0000","last_became_peered":"2024-09-14T03:58:45.819034+0000","last_unstale":"2024-09-14T04:03:19.827697+0000","last_undegraded":"2024-09-14T04:03:19.827697+0000","last_fullsized":"2024-09-14T04:03:19.827697+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:12.478061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":130,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.214202+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:03:16.214202+0000","last_peered":"2024-09-14T04:03:16.214202+0000","last_clean":"2024-09-14T04:03:16.214202+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:03:16.214202+0000","last_undegraded":"2024-09-14T04:03:16.214202+0000","last_fullsized":"2024-09-14T04:03:16.214202+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"156'1462","reported_seq":1481,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:27.390098+0000","last_change":"2024-09-14T04:03:07.728802+0000","last_active":"2024-09-14T04:03:27.390098+0000","last_peered":"2024-09-14T04:03:27.390098+0000","last_clean":"2024-09-14T04:03:27.390098+0000","last_became_active":"2024-09-14T04:03:07.728436+0000","last_became_peered":"2024-09-14T04:03:07.728436+0000","last_unstale":"2024-09-14T04:03:27.390098+0000","last_undegraded":"2024-09-14T04:03:27.390098+0000","last_fullsized":"2024-09-14T04:03:27.390098+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:23:27.102215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95813632,"num_objects":1462,"num_object_clones":0,"num_object_copies":4386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1462,"num_write_kb":93568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":122,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.780538+0000","last_change":"2024-09-14T03:52:28.189069+0000","last_active":"2024-09-14T04:03:16.780538+0000","last_peered":"2024-09-14T04:03:16.780538+0000","last_clean":"2024-09-14T04:03:16.780538+0000","last_became_active":"2024-09-14T03:52:28.188421+0000","last_became_peered":"2024-09-14T03:52:28.188421+0000","last_unstale":"2024-09-14T04:03:16.780538+0000","last_undegraded":"2024-09-14T04:03:16.780538+0000","last_fullsized":"2024-09-14T04:03:16.780538+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:20:09.951934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":104,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.754135+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:03:15.754135+0000","last_peered":"2024-09-14T04:03:15.754135+0000","last_clean":"2024-09-14T04:03:15.754135+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:03:15.754135+0000","last_undegraded":"2024-09-14T04:03:15.754135+0000","last_fullsized":"2024-09-14T04:03:15.754135+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"156'1708","reported_seq":1727,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.782114+0000","last_change":"2024-09-14T04:03:07.807300+0000","last_active":"2024-09-14T04:03:28.782114+0000","last_peered":"2024-09-14T04:03:28.782114+0000","last_clean":"2024-09-14T04:03:28.782114+0000","last_became_active":"2024-09-14T04:03:07.806981+0000","last_became_peered":"2024-09-14T04:03:07.806981+0000","last_unstale":"2024-09-14T04:03:28.782114+0000","last_undegraded":"2024-09-14T04:03:28.782114+0000","last_fullsized":"2024-09-14T04:03:28.782114+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1708,"log_dups_size":0,"ondisk_log_size":1708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:40:24.589535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111935488,"num_objects":1708,"num_object_clones":0,"num_object_copies":5124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1708,"num_write_kb":109312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":123,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.781317+0000","last_change":"2024-09-14T03:54:09.079772+0000","last_active":"2024-09-14T04:03:16.781317+0000","last_peered":"2024-09-14T04:03:16.781317+0000","last_clean":"2024-09-14T04:03:16.781317+0000","last_became_active":"2024-09-14T03:54:09.079323+0000","last_became_peered":"2024-09-14T03:54:09.079323+0000","last_unstale":"2024-09-14T04:03:16.781317+0000","last_undegraded":"2024-09-14T04:03:16.781317+0000","last_fullsized":"2024-09-14T04:03:16.781317+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:59:58.328220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":137,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.806635+0000","last_change":"2024-09-14T03:58:45.793252+0000","last_active":"2024-09-14T04:03:15.806635+0000","last_peered":"2024-09-14T04:03:15.806635+0000","last_clean":"2024-09-14T04:03:15.806635+0000","last_became_active":"2024-09-14T03:58:45.792917+0000","last_became_peered":"2024-09-14T03:58:45.792917+0000","last_unstale":"2024-09-14T04:03:15.806635+0000","last_undegraded":"2024-09-14T04:03:15.806635+0000","last_fullsized":"2024-09-14T04:03:15.806635+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:11.399204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"156'1779","reported_seq":1791,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.839299+0000","last_change":"2024-09-14T04:03:07.732057+0000","last_active":"2024-09-14T04:03:28.839299+0000","last_peered":"2024-09-14T04:03:28.839299+0000","last_clean":"2024-09-14T04:03:28.839299+0000","last_became_active":"2024-09-14T04:03:07.731692+0000","last_became_peered":"2024-09-14T04:03:07.731692+0000","last_unstale":"2024-09-14T04:03:28.839299+0000","last_undegraded":"2024-09-14T04:03:28.839299+0000","last_fullsized":"2024-09-14T04:03:28.839299+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:14.053800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116588544,"num_objects":1779,"num_object_clones":0,"num_object_copies":5337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1779,"num_write_kb":113856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":105,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.781220+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:03:16.781220+0000","last_peered":"2024-09-14T04:03:16.781220+0000","last_clean":"2024-09-14T04:03:16.781220+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:03:16.781220+0000","last_undegraded":"2024-09-14T04:03:16.781220+0000","last_fullsized":"2024-09-14T04:03:16.781220+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":131,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.214071+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:03:16.214071+0000","last_peered":"2024-09-14T04:03:16.214071+0000","last_clean":"2024-09-14T04:03:16.214071+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:03:16.214071+0000","last_undegraded":"2024-09-14T04:03:16.214071+0000","last_fullsized":"2024-09-14T04:03:16.214071+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"156'1663","reported_seq":1675,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.828019+0000","last_change":"2024-09-14T04:03:07.729216+0000","last_active":"2024-09-14T04:03:28.828019+0000","last_peered":"2024-09-14T04:03:28.828019+0000","last_clean":"2024-09-14T04:03:28.828019+0000","last_became_active":"2024-09-14T04:03:07.728694+0000","last_became_peered":"2024-09-14T04:03:07.728694+0000","last_unstale":"2024-09-14T04:03:28.828019+0000","last_undegraded":"2024-09-14T04:03:28.828019+0000","last_fullsized":"2024-09-14T04:03:28.828019+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:43:01.789990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108986368,"num_objects":1663,"num_object_clones":0,"num_object_copies":4989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1663,"num_write_kb":106432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":122,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.754329+0000","last_change":"2024-09-14T03:58:45.810163+0000","last_active":"2024-09-14T04:03:15.754329+0000","last_peered":"2024-09-14T04:03:15.754329+0000","last_clean":"2024-09-14T04:03:15.754329+0000","last_became_active":"2024-09-14T03:58:45.809802+0000","last_became_peered":"2024-09-14T03:58:45.809802+0000","last_unstale":"2024-09-14T04:03:15.754329+0000","last_undegraded":"2024-09-14T04:03:15.754329+0000","last_fullsized":"2024-09-14T04:03:15.754329+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:40:33.911609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":187,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.819843+0000","last_change":"2024-09-14T03:54:09.079995+0000","last_active":"2024-09-14T04:03:19.819843+0000","last_peered":"2024-09-14T04:03:19.819843+0000","last_clean":"2024-09-14T04:03:19.819843+0000","last_became_active":"2024-09-14T03:54:09.079686+0000","last_became_peered":"2024-09-14T03:54:09.079686+0000","last_unstale":"2024-09-14T04:03:19.819843+0000","last_undegraded":"2024-09-14T04:03:19.819843+0000","last_fullsized":"2024-09-14T04:03:19.819843+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:15:31.274947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"156'1664","reported_seq":1683,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.700587+0000","last_change":"2024-09-14T04:03:07.877792+0000","last_active":"2024-09-14T04:03:28.700587+0000","last_peered":"2024-09-14T04:03:28.700587+0000","last_clean":"2024-09-14T04:03:28.700587+0000","last_became_active":"2024-09-14T04:03:07.877437+0000","last_became_peered":"2024-09-14T04:03:07.877437+0000","last_unstale":"2024-09-14T04:03:28.700587+0000","last_undegraded":"2024-09-14T04:03:28.700587+0000","last_fullsized":"2024-09-14T04:03:28.700587+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1664,"log_dups_size":0,"ondisk_log_size":1664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:00:25.962905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":1664,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1664,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":145,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.826079+0000","last_change":"2024-09-14T03:58:45.788412+0000","last_active":"2024-09-14T04:03:19.826079+0000","last_peered":"2024-09-14T04:03:19.826079+0000","last_clean":"2024-09-14T04:03:19.826079+0000","last_became_active":"2024-09-14T03:58:45.788078+0000","last_became_peered":"2024-09-14T03:58:45.788078+0000","last_unstale":"2024-09-14T04:03:19.826079+0000","last_undegraded":"2024-09-14T04:03:19.826079+0000","last_fullsized":"2024-09-14T04:03:19.826079+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:49:09.289398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":120,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.806982+0000","last_change":"2024-09-14T03:58:45.812662+0000","last_active":"2024-09-14T04:03:15.806982+0000","last_peered":"2024-09-14T04:03:15.806982+0000","last_clean":"2024-09-14T04:03:15.806982+0000","last_became_active":"2024-09-14T03:58:45.812338+0000","last_became_peered":"2024-09-14T03:58:45.812338+0000","last_unstale":"2024-09-14T04:03:15.806982+0000","last_undegraded":"2024-09-14T04:03:15.806982+0000","last_fullsized":"2024-09-14T04:03:15.806982+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:35:35.258171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"156'1747","reported_seq":1766,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.759246+0000","last_change":"2024-09-14T04:03:07.877491+0000","last_active":"2024-09-14T04:03:28.759246+0000","last_peered":"2024-09-14T04:03:28.759246+0000","last_clean":"2024-09-14T04:03:28.759246+0000","last_became_active":"2024-09-14T04:03:07.877148+0000","last_became_peered":"2024-09-14T04:03:07.877148+0000","last_unstale":"2024-09-14T04:03:28.759246+0000","last_undegraded":"2024-09-14T04:03:28.759246+0000","last_fullsized":"2024-09-14T04:03:28.759246+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1747,"log_dups_size":0,"ondisk_log_size":1747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:53:00.271784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114491392,"num_objects":1747,"num_object_clones":0,"num_object_copies":5241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1747,"num_write_kb":111808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":122,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.214203+0000","last_change":"2024-09-14T03:54:09.089856+0000","last_active":"2024-09-14T04:03:16.214203+0000","last_peered":"2024-09-14T04:03:16.214203+0000","last_clean":"2024-09-14T04:03:16.214203+0000","last_became_active":"2024-09-14T03:54:09.089559+0000","last_became_peered":"2024-09-14T03:54:09.089559+0000","last_unstale":"2024-09-14T04:03:16.214203+0000","last_undegraded":"2024-09-14T04:03:16.214203+0000","last_fullsized":"2024-09-14T04:03:16.214203+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:11:49.651236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":137,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.807130+0000","last_change":"2024-09-14T03:55:58.249703+0000","last_active":"2024-09-14T04:03:15.807130+0000","last_peered":"2024-09-14T04:03:15.807130+0000","last_clean":"2024-09-14T04:03:15.807130+0000","last_became_active":"2024-09-14T03:55:58.249465+0000","last_became_peered":"2024-09-14T03:55:58.249465+0000","last_unstale":"2024-09-14T04:03:15.807130+0000","last_undegraded":"2024-09-14T04:03:15.807130+0000","last_fullsized":"2024-09-14T04:03:15.807130+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:03.743246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"156'1606","reported_seq":1617,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.666563+0000","last_change":"2024-09-14T04:03:07.731910+0000","last_active":"2024-09-14T04:03:28.666563+0000","last_peered":"2024-09-14T04:03:28.666563+0000","last_clean":"2024-09-14T04:03:28.666563+0000","last_became_active":"2024-09-14T04:03:07.731579+0000","last_became_peered":"2024-09-14T04:03:07.731579+0000","last_unstale":"2024-09-14T04:03:28.666563+0000","last_undegraded":"2024-09-14T04:03:28.666563+0000","last_fullsized":"2024-09-14T04:03:28.666563+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1606,"log_dups_size":0,"ondisk_log_size":1606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:14.476528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105250816,"num_objects":1606,"num_object_clones":0,"num_object_copies":4818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1606,"num_write_kb":102784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":121,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.806747+0000","last_change":"2024-09-14T03:55:58.246315+0000","last_active":"2024-09-14T04:03:15.806747+0000","last_peered":"2024-09-14T04:03:15.806747+0000","last_clean":"2024-09-14T04:03:15.806747+0000","last_became_active":"2024-09-14T03:55:58.245997+0000","last_became_peered":"2024-09-14T03:55:58.245997+0000","last_unstale":"2024-09-14T04:03:15.806747+0000","last_undegraded":"2024-09-14T04:03:15.806747+0000","last_fullsized":"2024-09-14T04:03:15.806747+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:11:13.312397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":120,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.807150+0000","last_change":"2024-09-14T03:54:09.082688+0000","last_active":"2024-09-14T04:03:15.807150+0000","last_peered":"2024-09-14T04:03:15.807150+0000","last_clean":"2024-09-14T04:03:15.807150+0000","last_became_active":"2024-09-14T03:54:09.082402+0000","last_became_peered":"2024-09-14T03:54:09.082402+0000","last_unstale":"2024-09-14T04:03:15.807150+0000","last_undegraded":"2024-09-14T04:03:15.807150+0000","last_fullsized":"2024-09-14T04:03:15.807150+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:28:53.024974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"156'1655","reported_seq":1672,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.704103+0000","last_change":"2024-09-14T04:03:07.731782+0000","last_active":"2024-09-14T04:03:28.704103+0000","last_peered":"2024-09-14T04:03:28.704103+0000","last_clean":"2024-09-14T04:03:28.704103+0000","last_became_active":"2024-09-14T04:03:07.731476+0000","last_became_peered":"2024-09-14T04:03:07.731476+0000","last_unstale":"2024-09-14T04:03:28.704103+0000","last_undegraded":"2024-09-14T04:03:28.704103+0000","last_fullsized":"2024-09-14T04:03:28.704103+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1655,"log_dups_size":0,"ondisk_log_size":1655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:59:35.425011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108462080,"num_objects":1655,"num_object_clones":0,"num_object_copies":4965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1655,"num_write_kb":105920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":118,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.826235+0000","last_change":"2024-09-14T03:58:45.822672+0000","last_active":"2024-09-14T04:03:19.826235+0000","last_peered":"2024-09-14T04:03:19.826235+0000","last_clean":"2024-09-14T04:03:19.826235+0000","last_became_active":"2024-09-14T03:58:45.822047+0000","last_became_peered":"2024-09-14T03:58:45.822047+0000","last_unstale":"2024-09-14T04:03:19.826235+0000","last_undegraded":"2024-09-14T04:03:19.826235+0000","last_fullsized":"2024-09-14T04:03:19.826235+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:29:49.341047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":153,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.825517+0000","last_change":"2024-09-14T03:58:45.793497+0000","last_active":"2024-09-14T04:03:19.825517+0000","last_peered":"2024-09-14T04:03:19.825517+0000","last_clean":"2024-09-14T04:03:19.825517+0000","last_became_active":"2024-09-14T03:58:45.793169+0000","last_became_peered":"2024-09-14T03:58:45.793169+0000","last_unstale":"2024-09-14T04:03:19.825517+0000","last_undegraded":"2024-09-14T04:03:19.825517+0000","last_fullsized":"2024-09-14T04:03:19.825517+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:11:06.166164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"156'1701","reported_seq":1720,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.677467+0000","last_change":"2024-09-14T04:03:07.730843+0000","last_active":"2024-09-14T04:03:28.677467+0000","last_peered":"2024-09-14T04:03:28.677467+0000","last_clean":"2024-09-14T04:03:28.677467+0000","last_became_active":"2024-09-14T04:03:07.730528+0000","last_became_peered":"2024-09-14T04:03:07.730528+0000","last_unstale":"2024-09-14T04:03:28.677467+0000","last_undegraded":"2024-09-14T04:03:28.677467+0000","last_fullsized":"2024-09-14T04:03:28.677467+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1701,"log_dups_size":0,"ondisk_log_size":1701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:50.273059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111476736,"num_objects":1701,"num_object_clones":0,"num_object_copies":5103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1701,"num_write_kb":108864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":238,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.780486+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:03:16.780486+0000","last_peered":"2024-09-14T04:03:16.780486+0000","last_clean":"2024-09-14T04:03:16.780486+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:03:16.780486+0000","last_undegraded":"2024-09-14T04:03:16.780486+0000","last_fullsized":"2024-09-14T04:03:16.780486+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":135,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.754258+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:03:15.754258+0000","last_peered":"2024-09-14T04:03:15.754258+0000","last_clean":"2024-09-14T04:03:15.754258+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:03:15.754258+0000","last_undegraded":"2024-09-14T04:03:15.754258+0000","last_fullsized":"2024-09-14T04:03:15.754258+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":105,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.826836+0000","last_change":"2024-09-14T03:58:45.782514+0000","last_active":"2024-09-14T04:03:19.826836+0000","last_peered":"2024-09-14T04:03:19.826836+0000","last_clean":"2024-09-14T04:03:19.826836+0000","last_became_active":"2024-09-14T03:58:45.782196+0000","last_became_peered":"2024-09-14T03:58:45.782196+0000","last_unstale":"2024-09-14T04:03:19.826836+0000","last_undegraded":"2024-09-14T04:03:19.826836+0000","last_fullsized":"2024-09-14T04:03:19.826836+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:59:44.614261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":104,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.214417+0000","last_change":"2024-09-14T03:43:11.909945+0000","last_active":"2024-09-14T04:03:16.214417+0000","last_peered":"2024-09-14T04:03:16.214417+0000","last_clean":"2024-09-14T04:03:16.214417+0000","last_became_active":"2024-09-14T03:43:11.909666+0000","last_became_peered":"2024-09-14T03:43:11.909666+0000","last_unstale":"2024-09-14T04:03:16.214417+0000","last_undegraded":"2024-09-14T04:03:16.214417+0000","last_fullsized":"2024-09-14T04:03:16.214417+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:13:19.629641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"156'1449","reported_seq":1468,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:27.388904+0000","last_change":"2024-09-14T04:03:07.877655+0000","last_active":"2024-09-14T04:03:27.388904+0000","last_peered":"2024-09-14T04:03:27.388904+0000","last_clean":"2024-09-14T04:03:27.388904+0000","last_became_active":"2024-09-14T04:03:07.877341+0000","last_became_peered":"2024-09-14T04:03:07.877341+0000","last_unstale":"2024-09-14T04:03:27.388904+0000","last_undegraded":"2024-09-14T04:03:27.388904+0000","last_fullsized":"2024-09-14T04:03:27.388904+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:53:04.733141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94961664,"num_objects":1449,"num_object_clones":0,"num_object_copies":4347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1449,"num_write_kb":92736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":150,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.780696+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:03:16.780696+0000","last_peered":"2024-09-14T04:03:16.780696+0000","last_clean":"2024-09-14T04:03:16.780696+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:03:16.780696+0000","last_undegraded":"2024-09-14T04:03:16.780696+0000","last_fullsized":"2024-09-14T04:03:16.780696+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":188,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.819322+0000","last_change":"2024-09-14T03:58:45.795127+0000","last_active":"2024-09-14T04:03:19.819322+0000","last_peered":"2024-09-14T04:03:19.819322+0000","last_clean":"2024-09-14T04:03:19.819322+0000","last_became_active":"2024-09-14T03:58:45.794831+0000","last_became_peered":"2024-09-14T03:58:45.794831+0000","last_unstale":"2024-09-14T04:03:19.819322+0000","last_undegraded":"2024-09-14T04:03:19.819322+0000","last_fullsized":"2024-09-14T04:03:19.819322+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:22:18.983609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":199,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.819593+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:03:19.819593+0000","last_peered":"2024-09-14T04:03:19.819593+0000","last_clean":"2024-09-14T04:03:19.819593+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:03:19.819593+0000","last_undegraded":"2024-09-14T04:03:19.819593+0000","last_fullsized":"2024-09-14T04:03:19.819593+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"156'1704","reported_seq":1716,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.829641+0000","last_change":"2024-09-14T04:03:07.729330+0000","last_active":"2024-09-14T04:03:28.829641+0000","last_peered":"2024-09-14T04:03:28.829641+0000","last_clean":"2024-09-14T04:03:28.829641+0000","last_became_active":"2024-09-14T04:03:07.728759+0000","last_became_peered":"2024-09-14T04:03:07.728759+0000","last_unstale":"2024-09-14T04:03:28.829641+0000","last_undegraded":"2024-09-14T04:03:28.829641+0000","last_fullsized":"2024-09-14T04:03:28.829641+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1704,"log_dups_size":0,"ondisk_log_size":1704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:32:16.962466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111673344,"num_objects":1704,"num_object_clones":0,"num_object_copies":5112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1704,"num_write_kb":109056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":178,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.819147+0000","last_change":"2024-09-14T03:58:45.791394+0000","last_active":"2024-09-14T04:03:19.819147+0000","last_peered":"2024-09-14T04:03:19.819147+0000","last_clean":"2024-09-14T04:03:19.819147+0000","last_became_active":"2024-09-14T03:58:45.790907+0000","last_became_peered":"2024-09-14T03:58:45.790907+0000","last_unstale":"2024-09-14T04:03:19.819147+0000","last_undegraded":"2024-09-14T04:03:19.819147+0000","last_fullsized":"2024-09-14T04:03:19.819147+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:07.434141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":153,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.824267+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:03:19.824267+0000","last_peered":"2024-09-14T04:03:19.824267+0000","last_clean":"2024-09-14T04:03:19.824267+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:03:19.824267+0000","last_undegraded":"2024-09-14T04:03:19.824267+0000","last_fullsized":"2024-09-14T04:03:19.824267+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":179,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.823741+0000","last_change":"2024-09-14T03:54:09.087551+0000","last_active":"2024-09-14T04:03:19.823741+0000","last_peered":"2024-09-14T04:03:19.823741+0000","last_clean":"2024-09-14T04:03:19.823741+0000","last_became_active":"2024-09-14T03:54:09.086608+0000","last_became_peered":"2024-09-14T03:54:09.086608+0000","last_unstale":"2024-09-14T04:03:19.823741+0000","last_undegraded":"2024-09-14T04:03:19.823741+0000","last_fullsized":"2024-09-14T04:03:19.823741+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:38:14.920066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"156'1471","reported_seq":1490,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:27.363298+0000","last_change":"2024-09-14T04:03:07.727315+0000","last_active":"2024-09-14T04:03:27.363298+0000","last_peered":"2024-09-14T04:03:27.363298+0000","last_clean":"2024-09-14T04:03:27.363298+0000","last_became_active":"2024-09-14T04:03:07.727034+0000","last_became_peered":"2024-09-14T04:03:07.727034+0000","last_unstale":"2024-09-14T04:03:27.363298+0000","last_undegraded":"2024-09-14T04:03:27.363298+0000","last_fullsized":"2024-09-14T04:03:27.363298+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1471,"log_dups_size":0,"ondisk_log_size":1471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:15:03.295173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96403456,"num_objects":1471,"num_object_clones":0,"num_object_copies":4413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1471,"num_write_kb":94144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":136,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.754175+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:03:15.754175+0000","last_peered":"2024-09-14T04:03:15.754175+0000","last_clean":"2024-09-14T04:03:15.754175+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:03:15.754175+0000","last_undegraded":"2024-09-14T04:03:15.754175+0000","last_fullsized":"2024-09-14T04:03:15.754175+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":121,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.806955+0000","last_change":"2024-09-14T03:52:28.187042+0000","last_active":"2024-09-14T04:03:15.806955+0000","last_peered":"2024-09-14T04:03:15.806955+0000","last_clean":"2024-09-14T04:03:15.806955+0000","last_became_active":"2024-09-14T03:52:28.186714+0000","last_became_peered":"2024-09-14T03:52:28.186714+0000","last_unstale":"2024-09-14T04:03:15.806955+0000","last_undegraded":"2024-09-14T04:03:15.806955+0000","last_fullsized":"2024-09-14T04:03:15.806955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:50:57.007224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":164,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.824446+0000","last_change":"2024-09-14T03:58:45.812745+0000","last_active":"2024-09-14T04:03:19.824446+0000","last_peered":"2024-09-14T04:03:19.824446+0000","last_clean":"2024-09-14T04:03:19.824446+0000","last_became_active":"2024-09-14T03:58:45.812385+0000","last_became_peered":"2024-09-14T03:58:45.812385+0000","last_unstale":"2024-09-14T04:03:19.824446+0000","last_undegraded":"2024-09-14T04:03:19.824446+0000","last_fullsized":"2024-09-14T04:03:19.824446+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:18:40.445597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"156'1587","reported_seq":1598,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.849870+0000","last_change":"2024-09-14T04:03:07.732415+0000","last_active":"2024-09-14T04:03:28.849870+0000","last_peered":"2024-09-14T04:03:28.849870+0000","last_clean":"2024-09-14T04:03:28.849870+0000","last_became_active":"2024-09-14T04:03:07.732019+0000","last_became_peered":"2024-09-14T04:03:07.732019+0000","last_unstale":"2024-09-14T04:03:28.849870+0000","last_undegraded":"2024-09-14T04:03:28.849870+0000","last_fullsized":"2024-09-14T04:03:28.849870+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1587,"log_dups_size":0,"ondisk_log_size":1587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:03:34.216145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104005632,"num_objects":1587,"num_object_clones":0,"num_object_copies":4761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1587,"num_write_kb":101568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":139,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.754463+0000","last_change":"2024-09-14T03:58:45.788256+0000","last_active":"2024-09-14T04:03:15.754463+0000","last_peered":"2024-09-14T04:03:15.754463+0000","last_clean":"2024-09-14T04:03:15.754463+0000","last_became_active":"2024-09-14T03:58:45.788052+0000","last_became_peered":"2024-09-14T03:58:45.788052+0000","last_unstale":"2024-09-14T04:03:15.754463+0000","last_undegraded":"2024-09-14T04:03:15.754463+0000","last_fullsized":"2024-09-14T04:03:15.754463+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:38:19.891512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":192,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.824535+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:03:19.824535+0000","last_peered":"2024-09-14T04:03:19.824535+0000","last_clean":"2024-09-14T04:03:19.824535+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:03:19.824535+0000","last_undegraded":"2024-09-14T04:03:19.824535+0000","last_fullsized":"2024-09-14T04:03:19.824535+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":142,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.824088+0000","last_change":"2024-09-14T03:54:09.089016+0000","last_active":"2024-09-14T04:03:19.824088+0000","last_peered":"2024-09-14T04:03:19.824088+0000","last_clean":"2024-09-14T04:03:19.824088+0000","last_became_active":"2024-09-14T03:54:09.087653+0000","last_became_peered":"2024-09-14T03:54:09.087653+0000","last_unstale":"2024-09-14T04:03:19.824088+0000","last_undegraded":"2024-09-14T04:03:19.824088+0000","last_fullsized":"2024-09-14T04:03:19.824088+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:07:46.505921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"156'1703","reported_seq":1715,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.671046+0000","last_change":"2024-09-14T04:03:07.806198+0000","last_active":"2024-09-14T04:03:28.671046+0000","last_peered":"2024-09-14T04:03:28.671046+0000","last_clean":"2024-09-14T04:03:28.671046+0000","last_became_active":"2024-09-14T04:03:07.805672+0000","last_became_peered":"2024-09-14T04:03:07.805672+0000","last_unstale":"2024-09-14T04:03:28.671046+0000","last_undegraded":"2024-09-14T04:03:28.671046+0000","last_fullsized":"2024-09-14T04:03:28.671046+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1703,"log_dups_size":0,"ondisk_log_size":1703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:51:42.555640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111607808,"num_objects":1703,"num_object_clones":0,"num_object_copies":5109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1703,"num_write_kb":108992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":153,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.824669+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:03:19.824669+0000","last_peered":"2024-09-14T04:03:19.824669+0000","last_clean":"2024-09-14T04:03:19.824669+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:03:19.824669+0000","last_undegraded":"2024-09-14T04:03:19.824669+0000","last_fullsized":"2024-09-14T04:03:19.824669+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":133,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.780510+0000","last_change":"2024-09-14T03:52:28.190501+0000","last_active":"2024-09-14T04:03:16.780510+0000","last_peered":"2024-09-14T04:03:16.780510+0000","last_clean":"2024-09-14T04:03:16.780510+0000","last_became_active":"2024-09-14T03:52:28.190050+0000","last_became_peered":"2024-09-14T03:52:28.190050+0000","last_unstale":"2024-09-14T04:03:16.780510+0000","last_undegraded":"2024-09-14T04:03:16.780510+0000","last_fullsized":"2024-09-14T04:03:16.780510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:58:22.557305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":177,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.819539+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:03:19.819539+0000","last_peered":"2024-09-14T04:03:19.819539+0000","last_clean":"2024-09-14T04:03:19.819539+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:03:19.819539+0000","last_undegraded":"2024-09-14T04:03:19.819539+0000","last_fullsized":"2024-09-14T04:03:19.819539+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"156'1603","reported_seq":1615,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.651780+0000","last_change":"2024-09-14T04:03:07.730888+0000","last_active":"2024-09-14T04:03:28.651780+0000","last_peered":"2024-09-14T04:03:28.651780+0000","last_clean":"2024-09-14T04:03:28.651780+0000","last_became_active":"2024-09-14T04:03:07.730513+0000","last_became_peered":"2024-09-14T04:03:07.730513+0000","last_unstale":"2024-09-14T04:03:28.651780+0000","last_undegraded":"2024-09-14T04:03:28.651780+0000","last_fullsized":"2024-09-14T04:03:28.651780+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:35:53.967561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105054208,"num_objects":1603,"num_object_clones":0,"num_object_copies":4809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1603,"num_write_kb":102592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":145,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.827849+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:03:19.827849+0000","last_peered":"2024-09-14T04:03:19.827849+0000","last_clean":"2024-09-14T04:03:19.827849+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:03:19.827849+0000","last_undegraded":"2024-09-14T04:03:19.827849+0000","last_fullsized":"2024-09-14T04:03:19.827849+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":153,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.824920+0000","last_change":"2024-09-14T03:54:09.082573+0000","last_active":"2024-09-14T04:03:19.824920+0000","last_peered":"2024-09-14T04:03:19.824920+0000","last_clean":"2024-09-14T04:03:19.824920+0000","last_became_active":"2024-09-14T03:54:09.082266+0000","last_became_peered":"2024-09-14T03:54:09.082266+0000","last_unstale":"2024-09-14T04:03:19.824920+0000","last_undegraded":"2024-09-14T04:03:19.824920+0000","last_fullsized":"2024-09-14T04:03:19.824920+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:30:24.244999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":142,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.825970+0000","last_change":"2024-09-14T03:58:45.822348+0000","last_active":"2024-09-14T04:03:19.825970+0000","last_peered":"2024-09-14T04:03:19.825970+0000","last_clean":"2024-09-14T04:03:19.825970+0000","last_became_active":"2024-09-14T03:58:45.821621+0000","last_became_peered":"2024-09-14T03:58:45.821621+0000","last_unstale":"2024-09-14T04:03:19.825970+0000","last_undegraded":"2024-09-14T04:03:19.825970+0000","last_fullsized":"2024-09-14T04:03:19.825970+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:02:12.477742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"156'1685","reported_seq":1704,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:28.693895+0000","last_change":"2024-09-14T04:03:07.807356+0000","last_active":"2024-09-14T04:03:28.693895+0000","last_peered":"2024-09-14T04:03:28.693895+0000","last_clean":"2024-09-14T04:03:28.693895+0000","last_became_active":"2024-09-14T04:03:07.807066+0000","last_became_peered":"2024-09-14T04:03:07.807066+0000","last_unstale":"2024-09-14T04:03:28.693895+0000","last_undegraded":"2024-09-14T04:03:28.693895+0000","last_fullsized":"2024-09-14T04:03:28.693895+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:03:06.711698+0000","last_clean_scrub_stamp":"2024-09-14T04:03:06.711698+0000","objects_scrubbed":0,"log_size":1685,"log_dups_size":0,"ondisk_log_size":1685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:31:29.428879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110428160,"num_objects":1685,"num_object_clones":0,"num_object_copies":5055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1685,"num_write_kb":107840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":191,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.820671+0000","last_change":"2024-09-14T03:54:09.081554+0000","last_active":"2024-09-14T04:03:19.820671+0000","last_peered":"2024-09-14T04:03:19.820671+0000","last_clean":"2024-09-14T04:03:19.820671+0000","last_became_active":"2024-09-14T03:54:09.081158+0000","last_became_peered":"2024-09-14T03:54:09.081158+0000","last_unstale":"2024-09-14T04:03:19.820671+0000","last_undegraded":"2024-09-14T04:03:19.820671+0000","last_fullsized":"2024-09-14T04:03:19.820671+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:24:43.027300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":134,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.214556+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:03:16.214556+0000","last_peered":"2024-09-14T04:03:16.214556+0000","last_clean":"2024-09-14T04:03:16.214556+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:03:16.214556+0000","last_undegraded":"2024-09-14T04:03:16.214556+0000","last_fullsized":"2024-09-14T04:03:16.214556+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":103,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.806745+0000","last_change":"2024-09-14T03:43:11.285521+0000","last_active":"2024-09-14T04:03:15.806745+0000","last_peered":"2024-09-14T04:03:15.806745+0000","last_clean":"2024-09-14T04:03:15.806745+0000","last_became_active":"2024-09-14T03:43:11.284778+0000","last_became_peered":"2024-09-14T03:43:11.284778+0000","last_unstale":"2024-09-14T04:03:15.806745+0000","last_undegraded":"2024-09-14T04:03:15.806745+0000","last_fullsized":"2024-09-14T04:03:15.806745+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:51:32.744643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":113,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.806923+0000","last_change":"2024-09-14T03:43:52.490179+0000","last_active":"2024-09-14T04:03:15.806923+0000","last_peered":"2024-09-14T04:03:15.806923+0000","last_clean":"2024-09-14T04:03:15.806923+0000","last_became_active":"2024-09-14T03:43:52.489786+0000","last_became_peered":"2024-09-14T03:43:52.489786+0000","last_unstale":"2024-09-14T04:03:15.806923+0000","last_undegraded":"2024-09-14T04:03:15.806923+0000","last_fullsized":"2024-09-14T04:03:15.806923+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:55.590247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":126,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.807060+0000","last_change":"2024-09-14T03:43:11.274139+0000","last_active":"2024-09-14T04:03:15.807060+0000","last_peered":"2024-09-14T04:03:15.807060+0000","last_clean":"2024-09-14T04:03:15.807060+0000","last_became_active":"2024-09-14T03:43:11.271423+0000","last_became_peered":"2024-09-14T03:43:11.271423+0000","last_unstale":"2024-09-14T04:03:15.807060+0000","last_undegraded":"2024-09-14T04:03:15.807060+0000","last_fullsized":"2024-09-14T04:03:15.807060+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:52.045590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":153,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.823981+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:03:19.823981+0000","last_peered":"2024-09-14T04:03:19.823981+0000","last_clean":"2024-09-14T04:03:19.823981+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:03:19.823981+0000","last_undegraded":"2024-09-14T04:03:19.823981+0000","last_fullsized":"2024-09-14T04:03:19.823981+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":124,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.781161+0000","last_change":"2024-09-14T03:58:45.793790+0000","last_active":"2024-09-14T04:03:16.781161+0000","last_peered":"2024-09-14T04:03:16.781161+0000","last_clean":"2024-09-14T04:03:16.781161+0000","last_became_active":"2024-09-14T03:58:45.793253+0000","last_became_peered":"2024-09-14T03:58:45.793253+0000","last_unstale":"2024-09-14T04:03:16.781161+0000","last_undegraded":"2024-09-14T04:03:16.781161+0000","last_fullsized":"2024-09-14T04:03:16.781161+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:44.988345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":122,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.754398+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:03:15.754398+0000","last_peered":"2024-09-14T04:03:15.754398+0000","last_clean":"2024-09-14T04:03:15.754398+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:03:15.754398+0000","last_undegraded":"2024-09-14T04:03:15.754398+0000","last_fullsized":"2024-09-14T04:03:15.754398+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":105,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.826385+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:03:19.826385+0000","last_peered":"2024-09-14T04:03:19.826385+0000","last_clean":"2024-09-14T04:03:19.826385+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:03:19.826385+0000","last_undegraded":"2024-09-14T04:03:19.826385+0000","last_fullsized":"2024-09-14T04:03:19.826385+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":187,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.820004+0000","last_change":"2024-09-14T03:58:45.794557+0000","last_active":"2024-09-14T04:03:19.820004+0000","last_peered":"2024-09-14T04:03:19.820004+0000","last_clean":"2024-09-14T04:03:19.820004+0000","last_became_active":"2024-09-14T03:58:45.794173+0000","last_became_peered":"2024-09-14T03:58:45.794173+0000","last_unstale":"2024-09-14T04:03:19.820004+0000","last_undegraded":"2024-09-14T04:03:19.820004+0000","last_fullsized":"2024-09-14T04:03:19.820004+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:33:12.799351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":117,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.825923+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:03:19.825923+0000","last_peered":"2024-09-14T04:03:19.825923+0000","last_clean":"2024-09-14T04:03:19.825923+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:03:19.825923+0000","last_undegraded":"2024-09-14T04:03:19.825923+0000","last_fullsized":"2024-09-14T04:03:19.825923+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":131,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:16.214554+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:03:16.214554+0000","last_peered":"2024-09-14T04:03:16.214554+0000","last_clean":"2024-09-14T04:03:16.214554+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:03:16.214554+0000","last_undegraded":"2024-09-14T04:03:16.214554+0000","last_fullsized":"2024-09-14T04:03:16.214554+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":134,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.826152+0000","last_change":"2024-09-14T03:58:45.822679+0000","last_active":"2024-09-14T04:03:19.826152+0000","last_peered":"2024-09-14T04:03:19.826152+0000","last_clean":"2024-09-14T04:03:19.826152+0000","last_became_active":"2024-09-14T03:58:45.822051+0000","last_became_peered":"2024-09-14T03:58:45.822051+0000","last_unstale":"2024-09-14T04:03:19.826152+0000","last_undegraded":"2024-09-14T04:03:19.826152+0000","last_fullsized":"2024-09-14T04:03:19.826152+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:30:52.022607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":188,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.820523+0000","last_change":"2024-09-14T03:58:45.795656+0000","last_active":"2024-09-14T04:03:19.820523+0000","last_peered":"2024-09-14T04:03:19.820523+0000","last_clean":"2024-09-14T04:03:19.820523+0000","last_became_active":"2024-09-14T03:58:45.795369+0000","last_became_peered":"2024-09-14T03:58:45.795369+0000","last_unstale":"2024-09-14T04:03:19.820523+0000","last_undegraded":"2024-09-14T04:03:19.820523+0000","last_fullsized":"2024-09-14T04:03:19.820523+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:45.896015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":152,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.823702+0000","last_change":"2024-09-14T03:58:45.793634+0000","last_active":"2024-09-14T04:03:19.823702+0000","last_peered":"2024-09-14T04:03:19.823702+0000","last_clean":"2024-09-14T04:03:19.823702+0000","last_became_active":"2024-09-14T03:58:45.793229+0000","last_became_peered":"2024-09-14T03:58:45.793229+0000","last_unstale":"2024-09-14T04:03:19.823702+0000","last_undegraded":"2024-09-14T04:03:19.823702+0000","last_fullsized":"2024-09-14T04:03:19.823702+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:31.431481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":116,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.827455+0000","last_change":"2024-09-14T03:58:45.795790+0000","last_active":"2024-09-14T04:03:19.827455+0000","last_peered":"2024-09-14T04:03:19.827455+0000","last_clean":"2024-09-14T04:03:19.827455+0000","last_became_active":"2024-09-14T03:58:45.795468+0000","last_became_peered":"2024-09-14T03:58:45.795468+0000","last_unstale":"2024-09-14T04:03:19.827455+0000","last_undegraded":"2024-09-14T04:03:19.827455+0000","last_fullsized":"2024-09-14T04:03:19.827455+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:50:54.779571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":139,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.806774+0000","last_change":"2024-09-14T03:55:58.252985+0000","last_active":"2024-09-14T04:03:15.806774+0000","last_peered":"2024-09-14T04:03:15.806774+0000","last_clean":"2024-09-14T04:03:15.806774+0000","last_became_active":"2024-09-14T03:55:58.252479+0000","last_became_peered":"2024-09-14T03:55:58.252479+0000","last_unstale":"2024-09-14T04:03:15.806774+0000","last_undegraded":"2024-09-14T04:03:15.806774+0000","last_fullsized":"2024-09-14T04:03:15.806774+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:46:04.357446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":182,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.819376+0000","last_change":"2024-09-14T03:58:45.810289+0000","last_active":"2024-09-14T04:03:19.819376+0000","last_peered":"2024-09-14T04:03:19.819376+0000","last_clean":"2024-09-14T04:03:19.819376+0000","last_became_active":"2024-09-14T03:58:45.809938+0000","last_became_peered":"2024-09-14T03:58:45.809938+0000","last_unstale":"2024-09-14T04:03:19.819376+0000","last_undegraded":"2024-09-14T04:03:19.819376+0000","last_fullsized":"2024-09-14T04:03:19.819376+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:06:18.534582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":179,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.819629+0000","last_change":"2024-09-14T03:52:28.182347+0000","last_active":"2024-09-14T04:03:19.819629+0000","last_peered":"2024-09-14T04:03:19.819629+0000","last_clean":"2024-09-14T04:03:19.819629+0000","last_became_active":"2024-09-14T03:52:28.181624+0000","last_became_peered":"2024-09-14T03:52:28.181624+0000","last_unstale":"2024-09-14T04:03:19.819629+0000","last_undegraded":"2024-09-14T04:03:19.819629+0000","last_fullsized":"2024-09-14T04:03:19.819629+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:18.032619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":137,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.806862+0000","last_change":"2024-09-14T03:58:45.793294+0000","last_active":"2024-09-14T04:03:15.806862+0000","last_peered":"2024-09-14T04:03:15.806862+0000","last_clean":"2024-09-14T04:03:15.806862+0000","last_became_active":"2024-09-14T03:58:45.793009+0000","last_became_peered":"2024-09-14T03:58:45.793009+0000","last_unstale":"2024-09-14T04:03:15.806862+0000","last_undegraded":"2024-09-14T04:03:15.806862+0000","last_fullsized":"2024-09-14T04:03:15.806862+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:46:00.206467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":124,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.754200+0000","last_change":"2024-09-14T03:54:09.084402+0000","last_active":"2024-09-14T04:03:15.754200+0000","last_peered":"2024-09-14T04:03:15.754200+0000","last_clean":"2024-09-14T04:03:15.754200+0000","last_became_active":"2024-09-14T03:54:09.084076+0000","last_became_peered":"2024-09-14T03:54:09.084076+0000","last_unstale":"2024-09-14T04:03:15.754200+0000","last_undegraded":"2024-09-14T04:03:15.754200+0000","last_fullsized":"2024-09-14T04:03:15.754200+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:49:51.780807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":137,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.806737+0000","last_change":"2024-09-14T03:58:45.793059+0000","last_active":"2024-09-14T04:03:15.806737+0000","last_peered":"2024-09-14T04:03:15.806737+0000","last_clean":"2024-09-14T04:03:15.806737+0000","last_became_active":"2024-09-14T03:58:45.792703+0000","last_became_peered":"2024-09-14T03:58:45.792703+0000","last_unstale":"2024-09-14T04:03:15.806737+0000","last_undegraded":"2024-09-14T04:03:15.806737+0000","last_fullsized":"2024-09-14T04:03:15.806737+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:01:17.321430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":107,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.754395+0000","last_change":"2024-09-14T03:43:11.278982+0000","last_active":"2024-09-14T04:03:15.754395+0000","last_peered":"2024-09-14T04:03:15.754395+0000","last_clean":"2024-09-14T04:03:15.754395+0000","last_became_active":"2024-09-14T03:43:11.277577+0000","last_became_peered":"2024-09-14T03:43:11.277577+0000","last_unstale":"2024-09-14T04:03:15.754395+0000","last_undegraded":"2024-09-14T04:03:15.754395+0000","last_fullsized":"2024-09-14T04:03:15.754395+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:51.445273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":145,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.826632+0000","last_change":"2024-09-14T03:58:45.790089+0000","last_active":"2024-09-14T04:03:19.826632+0000","last_peered":"2024-09-14T04:03:19.826632+0000","last_clean":"2024-09-14T04:03:19.826632+0000","last_became_active":"2024-09-14T03:58:45.789707+0000","last_became_peered":"2024-09-14T03:58:45.789707+0000","last_unstale":"2024-09-14T04:03:19.826632+0000","last_undegraded":"2024-09-14T04:03:19.826632+0000","last_fullsized":"2024-09-14T04:03:19.826632+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:03:26.492725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":131,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.828078+0000","last_change":"2024-09-14T03:58:45.784441+0000","last_active":"2024-09-14T04:03:19.828078+0000","last_peered":"2024-09-14T04:03:19.828078+0000","last_clean":"2024-09-14T04:03:19.828078+0000","last_became_active":"2024-09-14T03:58:45.783913+0000","last_became_peered":"2024-09-14T03:58:45.783913+0000","last_unstale":"2024-09-14T04:03:19.828078+0000","last_undegraded":"2024-09-14T04:03:19.828078+0000","last_fullsized":"2024-09-14T04:03:19.828078+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:16:11.643108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":120,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-14T04:03:15.806536+0000","last_change":"2024-09-14T03:54:09.084713+0000","last_active":"2024-09-14T04:03:15.806536+0000","last_peered":"2024-09-14T04:03:15.806536+0000","last_clean":"2024-09-14T04:03:15.806536+0000","last_became_active":"2024-09-14T03:54:09.082369+0000","last_became_peered":"2024-09-14T03:54:09.082369+0000","last_unstale":"2024-09-14T04:03:15.806536+0000","last_undegraded":"2024-09-14T04:03:15.806536+0000","last_fullsized":"2024-09-14T04:03:15.806536+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:41:39.305100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":191,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-14T04:03:19.819730+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:03:19.819730+0000","last_peered":"2024-09-14T04:03:19.819730+0000","last_clean":"2024-09-14T04:03:19.819730+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:03:19.819730+0000","last_undegraded":"2024-09-14T04:03:19.819730+0000","last_fullsized":"2024-09-14T04:03:19.819730+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1716191232,"num_objects":26187,"num_object_clones":0,"num_object_copies":78561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26187,"num_write_kb":1675968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5145427968,"data_stored":5145427968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26187,"ondisk_log_size":26187,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430477,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1476236,"kb_used_data":882116,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":92268916,"statfs":{"total":95995035648,"available":94483369984,"internally_reserved":0,"allocated":903286784,"data_stored":902514276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626706,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":611840,"kb_used_data":310328,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93133312,"statfs":{"total":95995035648,"available":95368511488,"internally_reserved":0,"allocated":317775872,"data_stored":317015210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822934,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1814448,"kb_used_data":909096,"kb_used_omap":1,"kb_used_meta":905278,"kb_avail":91930704,"statfs":{"total":95995035648,"available":94137040896,"internally_reserved":0,"allocated":930914304,"data_stored":930164993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927005016},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019162,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903104,"kb_used_data":317368,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":92842048,"statfs":{"total":95995035648,"available":95070257152,"internally_reserved":0,"allocated":324984832,"data_stored":324223683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552315,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043568,"kb_used_data":803880,"kb_used_omap":4,"kb_used_meta":239675,"kb_avail":92701584,"statfs":{"total":95995035648,"available":94926422016,"internally_reserved":0,"allocated":823173120,"data_stored":822423890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":245427645},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435356,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":456748,"kb_used_data":245604,"kb_used_omap":6,"kb_used_meta":211129,"kb_avail":93288404,"statfs":{"total":95995035648,"available":95527325696,"internally_reserved":0,"allocated":251498496,"data_stored":250752414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":216196557},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827826,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1340272,"kb_used_data":950696,"kb_used_omap":5,"kb_used_meta":389562,"kb_avail":92404880,"statfs":{"total":95995035648,"available":94622597120,"internally_reserved":0,"allocated":973512704,"data_stored":972763403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":398911727},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383814,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":746932,"kb_used_data":352364,"kb_used_omap":5,"kb_used_meta":394554,"kb_avail":92998220,"statfs":{"total":95995035648,"available":95230177280,"internally_reserved":0,"allocated":360820736,"data_stored":360068157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":404023356},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385810432,"data_stored":385810432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":983826432,"data_stored":983826432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264896512,"data_stored":264896512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":873005056,"data_stored":873005056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":330432512,"data_stored":330432512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":993591296,"data_stored":993591296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":328925184,"data_stored":328925184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":984940544,"data_stored":984940544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:03:33.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-09-14T04:03:33.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:33.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:33.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:33.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:33.472 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:33.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:33.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 37085 37069 165.459 119.938 0.00803632 0.00603755 2024-09-14T04:03:33.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:33.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:33.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:33.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:33.751 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:33.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:33.783 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:33.826 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:03:33.826 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:03:33.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:33.886 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:33.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:33.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:34.020 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:34.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:34.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:03:34.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:34.180 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:34.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:34.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:34.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:34.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:34.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:34 smithi114 ceph-mon[103029]: pgmap v523: 89 pgs: 89 active+clean; 1.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-09-14T04:03:34.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:34 smithi114 ceph-mon[106969]: pgmap v523: 89 pgs: 89 active+clean; 1.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-09-14T04:03:34.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:34.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:34.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:34.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 38990 38974 162.366 119.062 0.011194 0.00615461 2024-09-14T04:03:34.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:34.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:34.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:34 smithi160 ceph-mon[88407]: pgmap v523: 89 pgs: 89 active+clean; 1.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 147 MiB/s wr, 2.35k op/s 2024-09-14T04:03:34.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:34.726 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:34.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:34.828 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:34.863 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:34.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:34.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:34.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:35.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:35.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:35.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:35.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:35.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:35.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:35.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:35.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:35.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:35.397 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:35.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:35.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:35.536 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:35.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:35.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 40754 40738 159.108 110.25 0.00551714 0.00628048 2024-09-14T04:03:35.638 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:35.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:35.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:35.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:35.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:35.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:35.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:35.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:35.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:35.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:36.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:36.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:36.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:36.171 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:36.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:36.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:36.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:36.336 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:36.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:36.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:36.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:36.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:36.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:36.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 42601 42585 156.539 115.438 0.00491304 0.00638344 2024-09-14T04:03:36.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:36.609 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:36.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:36.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:36 smithi114 ceph-mon[103029]: pgmap v524: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 182 MiB/s wr, 2.92k op/s 2024-09-14T04:03:36.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:36 smithi114 ceph-mon[106969]: pgmap v524: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 182 MiB/s wr, 2.92k op/s 2024-09-14T04:03:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:36 smithi160 ceph-mon[88407]: pgmap v524: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 182 MiB/s wr, 2.92k op/s 2024-09-14T04:03:36.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:36.733 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:36.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:36.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:36.857 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:36.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:36.958 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:36.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:36.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:37.082 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:37.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:37.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:37.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:37.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:37.251 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:37.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:37.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:37.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:37.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:37.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:37.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:37.522 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:37.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:37.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 43825 43809 152.092 76.5 0.00911299 0.00657002 2024-09-14T04:03:37.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:37.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:37.658 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:37.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:37.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:37.794 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:37.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:37.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:37.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:37.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:37.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:38.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:38.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-14T04:03:38.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:38.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:38.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:38.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:38.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:38.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:38.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:38.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:38.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:38 smithi114 ceph-mon[103029]: pgmap v525: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-14T04:03:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:38 smithi114 ceph-mon[106969]: pgmap v525: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-14T04:03:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:38.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:38.495 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:38.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:38.520 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:38.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 45662 45646 150.129 114.812 0.0153914 0.00665526 2024-09-14T04:03:38.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:38.633 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:38.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:38 smithi160 ceph-mon[88407]: pgmap v525: 89 pgs: 89 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-14T04:03:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:38.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:38.773 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:38.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:38.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:38.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:38.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:39.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:39.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:39.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:39.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:39.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:39.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:39.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:39.323 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:39.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:39.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:39.458 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:39.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:39.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:03:39.545898+0000 min lat: 0.00111329 max lat: 0.249216 avg lat: 0.00673853 2024-09-14T04:03:39.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:03:39.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 47465 47449 148.256 112.688 0.00547858 0.00673853 2024-09-14T04:03:39.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:39.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:39.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:39.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:39.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:39.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:39.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:39.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:39.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:39.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:39.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:40.015 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:40.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:40.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:40.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:40.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:40.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:40.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:40.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:40.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:40.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:40.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:40.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:40.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:40.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:40.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 49024 49008 145.836 97.4375 0.00525874 0.00685143 2024-09-14T04:03:40.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:40.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:40.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:40 smithi114 ceph-mon[103029]: pgmap v526: 89 pgs: 89 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-14T04:03:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:40 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:03:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:40 smithi114 ceph-mon[106969]: pgmap v526: 89 pgs: 89 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-14T04:03:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:40 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:03:40.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:40 smithi160 ceph-mon[88407]: pgmap v526: 89 pgs: 89 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-14T04:03:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:40 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:03:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:03:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:03:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:03:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:40.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:40.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:40.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:40.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:40.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:40.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:40.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:40.949 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:40.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:41.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:41.073 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:41.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:41.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:41.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:41.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:41.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:41.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:41.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:41.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:41.467 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:41.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:41.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 50230 50214 142.633 75.375 0.00481019 0.00700599 2024-09-14T04:03:41.555 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:03:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:03:41] "GET /metrics HTTP/1.1" 200 36423 "" "Prometheus/2.51.0" 2024-09-14T04:03:41.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:41.601 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:41.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:41.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:41.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:41.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:41.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:41.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:41.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:41.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:42.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:42.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:42.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:42.164 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:42.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:42.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:42.270 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:03:42.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:42.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:42.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:42.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:42.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:42.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:42 smithi114 ceph-mon[106969]: pgmap v527: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-09-14T04:03:42.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/970107292' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:03:42.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:42 smithi114 ceph-mon[103029]: pgmap v527: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-09-14T04:03:42.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/970107292' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:03:42.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 51968 51952 141.154 108.625 0.0135923 0.00707875 2024-09-14T04:03:42.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:42.597 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:42.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:42 smithi160 ceph-mon[88407]: pgmap v527: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-09-14T04:03:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/970107292' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:03:42.689 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:42.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:42.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:42.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:42.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:42.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:42.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:42.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:43.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:43.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:43.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:43.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:43.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:43.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:43.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:43.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:43.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:43 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:03:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:43 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/970107292' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:03:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:43 smithi114 ceph-mon[106969]: osdmap e157: 8 total, 8 up, 8 in 2024-09-14T04:03:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:43 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:03:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:43 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/970107292' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:03:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:43 smithi114 ceph-mon[103029]: osdmap e157: 8 total, 8 up, 8 in 2024-09-14T04:03:43.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:43 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:03:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:43 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/970107292' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:03:43.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:43 smithi160 ceph-mon[88407]: osdmap e157: 8 total, 8 up, 8 in 2024-09-14T04:03:43.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:43.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:43.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:43.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 53629 53613 139.598 103.812 0.00646727 0.00715711 2024-09-14T04:03:43.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:43.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:43.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:43.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:43.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:43.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:43.852 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:43.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:43.870 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:43.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:43.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:43.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:44.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:44.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:44.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:44.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:44.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:44.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 9 pgp_num to 16 2024-09-14T04:03:44.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:44.394 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:44.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:44.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:44.532 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:44.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:44.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 55285 55269 138.154 103.5 0.00670476 0.00723357 2024-09-14T04:03:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:44 smithi114 ceph-mon[103029]: pgmap v529: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-14T04:03:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:44 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1468752740' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-14T04:03:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:44 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-14T04:03:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:44 smithi114 ceph-mon[106969]: pgmap v529: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-14T04:03:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:44 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1468752740' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-14T04:03:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:44 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-14T04:03:44.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:44 smithi160 ceph-mon[88407]: pgmap v529: 89 pgs: 89 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-14T04:03:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:44 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1468752740' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-14T04:03:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:44 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"} : dispatch 2024-09-14T04:03:44.670 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:44.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:44.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:44.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:44.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:44.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:44.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:44.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:44.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:45.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:45.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:45.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:45.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:45.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:45.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:45.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:45.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:45.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:45.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:45.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:45.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:45.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 56609 56593 136.023 82.75 0.00264335 0.00729582 2024-09-14T04:03:45.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:45.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:45.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:45.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:45 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-14T04:03:45.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:45 smithi114 ceph-mon[103029]: osdmap e158: 8 total, 8 up, 8 in 2024-09-14T04:03:45.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:45 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-14T04:03:45.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:45 smithi114 ceph-mon[106969]: osdmap e158: 8 total, 8 up, 8 in 2024-09-14T04:03:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:45 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-09-14T04:03:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:45 smithi160 ceph-mon[88407]: osdmap e158: 8 total, 8 up, 8 in 2024-09-14T04:03:45.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:45.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:45.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:45.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:45.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:45.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:45.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:46.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:46.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:46.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:46.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:46.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:46.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:46.282 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:46.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:46.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:46.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:46.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:46.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:46.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:03:46.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:46.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 57890 57874 133.948 80.0625 0.0119159 0.00745966 2024-09-14T04:03:46.568 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:46.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:46.602 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:46 smithi114 ceph-mon[106969]: pgmap v531: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-09-14T04:03:46.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:46 smithi114 ceph-mon[103029]: pgmap v531: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-09-14T04:03:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:46 smithi160 ceph-mon[88407]: pgmap v531: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 130 MiB/s wr, 2.07k op/s 2024-09-14T04:03:46.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:46.727 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:46.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:46.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:46.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:46.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:46.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:46.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:46.992 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:46.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:47.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:47.115 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:47.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:47.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:47.257 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:47.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:47.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:47.398 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:47.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:47.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:47.522 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:47.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:47.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:47.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 59141 59125 131.956 78.1875 0.0328683 0.00756216 2024-09-14T04:03:47.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:47.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:47.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:47.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:47.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:47.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:47.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:47.960 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:47.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:47.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:48.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:48.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:48.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:48.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:48.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:48.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:48.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:48.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:48.371 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:48.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:48.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:48.504 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:48.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:48.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:48.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 60715 60699 130.796 98.375 0.00842146 0.00764098 2024-09-14T04:03:48.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:48.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:48 smithi114 ceph-mon[103029]: pgmap v532: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-14T04:03:48.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:48 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:48.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:48 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:48.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:48 smithi114 ceph-mon[106969]: pgmap v532: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-14T04:03:48.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:48 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:48.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:48 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:48.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:48.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:48.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:48 smithi160 ceph-mon[88407]: pgmap v532: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-14T04:03:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:48 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:48 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:03:48.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:48.778 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:48.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:48.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:48.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:48.918 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:48.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:49.020 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:49.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:49.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:49.129 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:03:49.129 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:03:49.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:49.166 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:03:49.166 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:03:49.181 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:03:49.181 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:03:49.183 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:49.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:49.200 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:03:49.201 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:03:49.235 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:03:49.235 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:03:49.268 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:03:49.268 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:03:49.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:49.299 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:03:49.299 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:03:49.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:49.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:49.330 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:03:49.330 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:03:49.364 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:03:49.364 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:03:49.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:49.455 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:49.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:49.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 62259 62243 129.652 96.5 0.0122735 0.00770796 2024-09-14T04:03:49.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:49.593 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:49.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:49.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:49 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2415651352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:49.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:49 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2415651352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:49 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2415651352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:03:49.694 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:49.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:49.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:49.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:49.868 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:49.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:49.970 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:50.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:50.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:50.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:50.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:50.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:50.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-14T04:03:50.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:50.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:50.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:50.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:50.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:50.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:50.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:50.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 63848 63832 128.671 99.3125 0.00820072 0.0077669 2024-09-14T04:03:50.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:50.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:50.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:50 smithi114 ceph-mon[103029]: pgmap v533: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T04:03:50.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:50 smithi114 ceph-mon[106969]: pgmap v533: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T04:03:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:50 smithi160 ceph-mon[88407]: pgmap v533: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T04:03:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:03:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:03:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:03:50.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:50.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:50.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:50.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:50.817 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:50.827 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-14T04:03:50.828 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:03:50.828 INFO:tasks.thrashosds.thrasher:Reweighting osd 6 to 0.8943603370542378 2024-09-14T04:03:50.828 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 6 0.8943603370542378 2024-09-14T04:03:50.863 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:50.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:50.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:50.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:50.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:51.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:51.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:51.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:51.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:51.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:51.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:51.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:51.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:51.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:51.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:51.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:51.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:51.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:51.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:51.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 65271 65255 127.429 88.9375 0.0040795 0.00784317 2024-09-14T04:03:51.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:03:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:03:51] "GET /metrics HTTP/1.1" 200 36423 "" "Prometheus/2.51.0" 2024-09-14T04:03:51.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:51.661 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:51.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:51.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:51.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:51.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:51.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:51.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:51.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:52.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:52.072 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:52.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:52.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:52.195 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:52.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:52.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:52.318 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:52.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:52.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:52.453 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:52.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:52.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 66426 66410 125.755 72.1875 0.00908429 0.00794472 2024-09-14T04:03:52.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:52.587 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:52.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:52.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:52 smithi114 ceph-mon[103029]: pgmap v534: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-14T04:03:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:52 smithi114 ceph-mon[106969]: pgmap v534: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-14T04:03:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:52 smithi160 ceph-mon[88407]: pgmap v534: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-14T04:03:52.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:52.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:52.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:03:52.814 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:52.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:52.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:52.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:52.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:52.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:52.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:53.070 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:53.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:53.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:53.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:53.238 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:53.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:53.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:53.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:53.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:53.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:53.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:53.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:53.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:53.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 67642 67626 124.292 76 0.0177945 0.00804079 2024-09-14T04:03:53.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:53.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:53.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:53.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:53.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:03:53.750 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:53.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:53.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:53.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:53.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:53.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:54.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:54.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:54.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:54.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:54.190 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:54.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:54.292 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:54.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:54.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:54.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:54.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:54.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:54.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:54.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 69035 69019 123.228 87.0625 0.00609917 0.00811075 2024-09-14T04:03:54.584 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:54.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:54.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:54 smithi114 ceph-mon[103029]: pgmap v535: 89 pgs: 89 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-14T04:03:54.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:54 smithi114 ceph-mon[106969]: pgmap v535: 89 pgs: 89 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-14T04:03:54.631 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:54 smithi160 ceph-mon[88407]: pgmap v535: 89 pgs: 89 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-14T04:03:54.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:54.726 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:54.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:54.802 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:03:54.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:54.856 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:54.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:54.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:54.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:54.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:55.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:55.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:55.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:55.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:55.258 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:55.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:55.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:55.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:55.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:55.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:55.505 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:55.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:55.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 70221 70205 121.864 74.125 0.0106239 0.00820094 2024-09-14T04:03:55.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:55.644 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:55.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:55.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:55.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:55.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:55.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:55.921 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:55.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:55.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:56.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:56.056 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:56.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:56.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:56.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:56.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:56.343 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:56.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:56.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:03:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:56.479 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:56.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:03:56.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 71705 71689 121.077 92.75 0.00563216 0.00825438 2024-09-14T04:03:56.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:56 smithi114 ceph-mon[103029]: pgmap v536: 89 pgs: 89 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T04:03:56.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:56 smithi114 ceph-mon[106969]: pgmap v536: 89 pgs: 89 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T04:03:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:56.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:56.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:56 smithi160 ceph-mon[88407]: pgmap v536: 89 pgs: 89 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T04:03:56.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:56.775 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:56.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:56.877 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:56.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:56.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:57.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:57.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:57.046 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:57.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:57.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:57.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:57.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:57.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:57.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:57.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:57.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:57.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:57.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:57.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 73134 73118 120.241 89.3125 0.0131271 0.00831199 2024-09-14T04:03:57.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:03:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:57.597 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:57.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:03:57.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:57.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:57.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:57.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:57.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:57.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:57.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:58.008 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:58.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:58.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:58.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:58.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:58.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:03:58.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:58.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:03:58.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:58.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:58.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:58.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:58.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 74582 74566 119.475 90.5 0.0196698 0.00836483 2024-09-14T04:03:58.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:58.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:58.575 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:58 smithi114 ceph-mon[103029]: pgmap v537: 89 pgs: 89 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-14T04:03:58.575 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:58 smithi114 ceph-mon[106969]: pgmap v537: 89 pgs: 89 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-14T04:03:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:58 smithi160 ceph-mon[88407]: pgmap v537: 89 pgs: 89 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-14T04:03:58.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:03:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:58.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:58.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:03:58.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:03:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:58.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:03:58.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:03:58.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:58.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:58.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:58.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:59.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:59.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:59.121 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:59.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:59.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:03:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:59.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:59.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:03:59.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:59.361 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:03:59.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:59.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:59.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:59.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:59.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:59.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:03:59.550179+0000 min lat: 0.00111329 max lat: 0.28871 avg lat: 0.00844106 2024-09-14T04:03:59.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:03:59.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 75808 75792 118.404 76.625 0.0120493 0.00844106 2024-09-14T04:03:59.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:59 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2155730457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:03:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:03:59 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:03:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:59 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2155730457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:03:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:03:59 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:03:59.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:03:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:59 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2155730457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:03:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:03:59 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:03:59.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:59.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:03:59.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:59.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:59.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:03:59.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:03:59.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:03:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:03:59.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:00.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:00.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:00.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:00.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:00.179 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:00.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:00.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:00.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:00.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:00.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:00.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:00.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:00.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:00.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 76867 76851 117.13 66.1875 0.00782607 0.00853266 2024-09-14T04:04:00.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:00.558 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:00 smithi160 ceph-mon[88407]: pgmap v538: 89 pgs: 89 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-14T04:04:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:00 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:04:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:00 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:04:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:00 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:04:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:00 smithi160 ceph-mon[88407]: osdmap e159: 8 total, 8 up, 8 in 2024-09-14T04:04:00.559 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:04:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:04:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:04:00.576 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:00 smithi114 ceph-mon[103029]: pgmap v538: 89 pgs: 89 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-14T04:04:00.576 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:00 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:04:00.576 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:00 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:04:00.576 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:00 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:04:00.576 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:00 smithi114 ceph-mon[103029]: osdmap e159: 8 total, 8 up, 8 in 2024-09-14T04:04:00.577 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:00 smithi114 ceph-mon[106969]: pgmap v538: 89 pgs: 89 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-14T04:04:00.577 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:00 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:04:00.577 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:00 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:04:00.577 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:00 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:04:00.577 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:00 smithi114 ceph-mon[106969]: osdmap e159: 8 total, 8 up, 8 in 2024-09-14T04:04:00.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:00.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:00.692 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:00.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:00.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:00.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:00.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:00.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:00.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:00.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:01.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:01.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:01.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:01.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:01.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:01.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:01.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:01.286 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:01.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:01.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:01.433 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:01.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:01.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:01.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 78230 78214 116.369 85.1875 0.011211 0.00858858 2024-09-14T04:04:01.582 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:01.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:01.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:01.695 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:04:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:04:01] "GET /metrics HTTP/1.1" 200 36430 "" "Prometheus/2.51.0" 2024-09-14T04:04:01.725 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:01.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:01.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:01.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:01.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:01.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:01.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:02.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:02.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:02.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:02.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:02.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:02.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:02.282 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:02.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:02.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:02.406 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:02.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:02.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:02.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:02.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 79175 79159 115.036 59.0625 0.00682051 0.00864349 2024-09-14T04:04:02.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:02.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:02.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:02 smithi114 ceph-mon[103029]: pgmap v540: 89 pgs: 89 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-14T04:04:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:02 smithi114 ceph-mon[106969]: pgmap v540: 89 pgs: 89 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-14T04:04:02.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:02 smithi160 ceph-mon[88407]: pgmap v540: 89 pgs: 89 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-14T04:04:02.662 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:04:02.662 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:04:02.662 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:04:02.662 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:04:02.662 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.662 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171088 7.b 7:da6a697d:::benchmark_data_smithi114_118301_object50223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.663 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.720093+0000", 2024-09-14T04:04:02.663 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.93704413900002, 2024-09-14T04:04:02.663 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97501888199999998, 2024-09-14T04:04:02.663 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.663 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.663 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.663 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171088 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.720093+0000", 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.720093+0000", 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.720095+0000", 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0360000000000001e-06 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.720102+0000", 2024-09-14T04:04:02.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9600000000000003e-06 2024-09-14T04:04:02.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.720113+0000", 2024-09-14T04:04:02.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1005000000000001e-05 2024-09-14T04:04:02.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.720118+0000", 2024-09-14T04:04:02.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2000000000000002e-06 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.720184+0000", 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5576999999999999e-05 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.720196+0000", 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2252000000000001e-05 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.663294+0000", 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.943097564 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.663395+0000", 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010149099999999999 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.663938+0000", 2024-09-14T04:04:02.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00054309200000000001 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.680578+0000", 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016639352 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.680594+0000", 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6092000000000001e-05 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695007+0000", 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014413086 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695062+0000", 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4851000000000001e-05 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695112+0000", 2024-09-14T04:04:02.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0324000000000001e-05 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171090 7.b 7:d0993bc1:::benchmark_data_smithi114_118301_object50225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.722350+0000", 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.93478786200001, 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.972938466, 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.671 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171090 2024-09-14T04:04:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.722350+0000", 2024-09-14T04:04:02.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.722350+0000", 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.722351+0000", 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.846e-06 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.722359+0000", 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1790000000000002e-06 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.722372+0000", 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3902e-05 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.722383+0000", 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0118999999999999e-05 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.722449+0000", 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6903000000000006e-05 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.722459+0000", 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9680000000000001e-06 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.675472+0000", 2024-09-14T04:04:02.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.95301215900000003 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.675602+0000", 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000130266 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.675734+0000", 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013178699999999999 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.682347+0000", 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0066133210000000001 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.686375+0000", 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0040282540000000002 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695196+0000", 2024-09-14T04:04:02.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.008821186 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695249+0000", 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2380999999999999e-05 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695288+0000", 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9195e-05 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.678 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171095 7.b 7:dfb6be34:::benchmark_data_smithi114_118301_object50230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.729081+0000", 2024-09-14T04:04:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.92805601800001, 2024-09-14T04:04:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97228804700000004, 2024-09-14T04:04:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.679 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171095 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.729081+0000", 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.729081+0000", 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.729083+0000", 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5460000000000001e-06 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.729088+0000", 2024-09-14T04:04:02.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9570000000000001e-06 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.729103+0000", 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4691999999999999e-05 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.729110+0000", 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4209999999999998e-06 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.729165+0000", 2024-09-14T04:04:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5303999999999998e-05 2024-09-14T04:04:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.729175+0000", 2024-09-14T04:04:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0205e-05 2024-09-14T04:04:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.678348+0000", 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.949172766 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.678430+0000", 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1581000000000004e-05 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.678511+0000", 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1050000000000005e-05 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.686380+0000", 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0078693559999999992 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.686424+0000", 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3526000000000003e-05 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701244+0000", 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014820364000000001 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701311+0000", 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7348999999999995e-05 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701369+0000", 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7930000000000003e-05 2024-09-14T04:04:02.686 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171097 7.b 7:d9bcb3b9:::benchmark_data_smithi114_118301_object50232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.731123+0000", 2024-09-14T04:04:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.92601391300002, 2024-09-14T04:04:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.970458457, 2024-09-14T04:04:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.687 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171097 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.731123+0000", 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.731123+0000", 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.731125+0000", 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.389e-06 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.731131+0000", 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.229e-06 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.731146+0000", 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5101e-05 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.731155+0000", 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4300000000000006e-06 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.731214+0000", 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9190000000000001e-05 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.731224+0000", 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0618000000000001e-05 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.680610+0000", 2024-09-14T04:04:02.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94938591900000002 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.680684+0000", 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3608000000000004e-05 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.680752+0000", 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7570999999999994e-05 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.686437+0000", 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0056856340000000002 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.686498+0000", 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0433999999999998e-05 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701495+0000", 2024-09-14T04:04:02.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014996948 2024-09-14T04:04:02.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:02.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701544+0000", 2024-09-14T04:04:02.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9465999999999997e-05 2024-09-14T04:04:02.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701582+0000", 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7920000000000003e-05 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171147 7.b 7:df3cc2bd:::benchmark_data_smithi114_118301_object50282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.796603+0000", 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.860534196, 2024-09-14T04:04:02.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.91143669299999996, 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171147 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796603+0000", 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796603+0000", 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796605+0000", 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.638e-06 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796612+0000", 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6919999999999996e-06 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796625+0000", 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3380999999999999e-05 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796634+0000", 2024-09-14T04:04:02.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2010000000000005e-06 2024-09-14T04:04:02.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.700 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.700 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796740+0000", 2024-09-14T04:04:02.701 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010597799999999999 2024-09-14T04:04:02.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.701 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796751+0000", 2024-09-14T04:04:02.701 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0730000000000001e-05 2024-09-14T04:04:02.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.682439+0000", 2024-09-14T04:04:02.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.88568843900000005 2024-09-14T04:04:02.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.682539+0000", 2024-09-14T04:04:02.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9543000000000006e-05 2024-09-14T04:04:02.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.682610+0000", 2024-09-14T04:04:02.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1075999999999995e-05 2024-09-14T04:04:02.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.686443+0000", 2024-09-14T04:04:02.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003833066 2024-09-14T04:04:02.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.705 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.690486+0000", 2024-09-14T04:04:02.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0040426180000000004 2024-09-14T04:04:02.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.707937+0000", 2024-09-14T04:04:02.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017451273999999999 2024-09-14T04:04:02.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.707996+0000", 2024-09-14T04:04:02.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8987999999999998e-05 2024-09-14T04:04:02.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.708040+0000", 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4069000000000001e-05 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171148 7.b 7:d382558e:::benchmark_data_smithi114_118301_object50283:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.796670+0000", 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.86046766700002, 2024-09-14T04:04:02.708 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.91155257300000003, 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171148 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796670+0000", 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796670+0000", 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796670+0000", 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0800000000000004e-07 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796673+0000", 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8849999999999999e-06 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796681+0000", 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8790000000000003e-06 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796687+0000", 2024-09-14T04:04:02.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9170000000000003e-06 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796775+0000", 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7800000000000006e-05 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.796782+0000", 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0949999999999998e-06 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.684313+0000", 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.88753079400000001 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.684385+0000", 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1864000000000002e-05 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.684450+0000", 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5642000000000006e-05 2024-09-14T04:04:02.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.690500+0000", 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0060500600000000003 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.690524+0000", 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3325000000000001e-05 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.708134+0000", 2024-09-14T04:04:02.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017610436 2024-09-14T04:04:02.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.708187+0000", 2024-09-14T04:04:02.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2884999999999999e-05 2024-09-14T04:04:02.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.708222+0000", 2024-09-14T04:04:02.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5383000000000003e-05 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171158 7.b 7:ded28d32:::benchmark_data_smithi114_118301_object50293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.813545+0000", 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.843592489, 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.89479220000000004, 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.716 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171158 2024-09-14T04:04:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.813545+0000", 2024-09-14T04:04:02.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.813545+0000", 2024-09-14T04:04:02.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.813547+0000", 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9530000000000002e-06 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.813555+0000", 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0020000000000006e-06 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.813572+0000", 2024-09-14T04:04:02.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6790999999999999e-05 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.813581+0000", 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0289999999999994e-06 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.813651+0000", 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0099999999999996e-05 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.813662+0000", 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0805000000000001e-05 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.686541+0000", 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87287943599999995 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.686653+0000", 2024-09-14T04:04:02.721 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000112419 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.686768+0000", 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000114524 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.692766+0000", 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059981560000000001 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.722 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695151+0000", 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023850220000000001 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.708278+0000", 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013127196000000001 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.708304+0000", 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5598000000000001e-05 2024-09-14T04:04:02.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.708337+0000", 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3169e-05 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171162 7.b 7:d055593b:::benchmark_data_smithi114_118301_object50297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.820872+0000", 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.836265545, 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.89511037900000001, 2024-09-14T04:04:02.724 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.725 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.725 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.725 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.725 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.725 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.725 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171162 2024-09-14T04:04:02.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.726 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.726 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.820872+0000", 2024-09-14T04:04:02.726 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.726 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.820872+0000", 2024-09-14T04:04:02.727 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.727 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.727 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.727 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.820874+0000", 2024-09-14T04:04:02.727 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.978e-06 2024-09-14T04:04:02.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.727 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.727 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.727 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.820882+0000", 2024-09-14T04:04:02.728 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4710000000000007e-06 2024-09-14T04:04:02.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.728 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.728 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.728 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.820897+0000", 2024-09-14T04:04:02.728 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4654000000000001e-05 2024-09-14T04:04:02.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.728 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.728 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.728 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.820905+0000", 2024-09-14T04:04:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9330000000000001e-06 2024-09-14T04:04:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.820968+0000", 2024-09-14T04:04:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2967999999999997e-05 2024-09-14T04:04:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.729 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.820977+0000", 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3139999999999993e-06 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.688554+0000", 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.86757695700000004 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.688637+0000", 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2534000000000006e-05 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.731 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.731 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.688713+0000", 2024-09-14T04:04:02.731 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6516999999999999e-05 2024-09-14T04:04:02.731 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.731 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.731 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.731 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695124+0000", 2024-09-14T04:04:02.731 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0064104360000000003 2024-09-14T04:04:02.731 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.731 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695337+0000", 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000213451 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.715876+0000", 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.020539266 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.732 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.715932+0000", 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5345e-05 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.715982+0000", 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0555e-05 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171166 7.b 7:d10e2d55:::benchmark_data_smithi114_118301_object50301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.733 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.824805+0000", 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.832332753, 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.89132656099999996, 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171166 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.734 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.824805+0000", 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.824805+0000", 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.824807+0000", 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1409999999999999e-06 2024-09-14T04:04:02.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.824821+0000", 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4005000000000001e-05 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.824839+0000", 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8301999999999999e-05 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.736 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.737 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.824848+0000", 2024-09-14T04:04:02.737 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5189999999999992e-06 2024-09-14T04:04:02.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.737 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.737 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.824933+0000", 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5809000000000005e-05 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.824953+0000", 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9281999999999999e-05 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.690557+0000", 2024-09-14T04:04:02.738 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.86560421499999995 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.690660+0000", 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010303599999999999 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.690776+0000", 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011592 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695298+0000", 2024-09-14T04:04:02.739 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00452243 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701152+0000", 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0058532180000000003 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.716047+0000", 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.01489533 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.740 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.716092+0000", 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5425e-05 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.716131+0000", 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8928999999999999e-05 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171177 7.b 7:d770aeb4:::benchmark_data_smithi114_118301_object50312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.741 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.844545+0000", 2024-09-14T04:04:02.742 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.812592655, 2024-09-14T04:04:02.742 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87983096400000005, 2024-09-14T04:04:02.742 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.742 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.742 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.742 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.742 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.743 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.743 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171177 2024-09-14T04:04:02.743 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.743 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.743 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.743 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.743 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.844545+0000", 2024-09-14T04:04:02.744 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.744 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.744 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.744 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.744 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.844545+0000", 2024-09-14T04:04:02.744 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.745 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.745 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.745 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.844547+0000", 2024-09-14T04:04:02.745 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8300000000000001e-06 2024-09-14T04:04:02.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.745 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.745 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.746 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.844555+0000", 2024-09-14T04:04:02.746 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5250000000000005e-06 2024-09-14T04:04:02.746 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.746 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.746 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.746 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.844580+0000", 2024-09-14T04:04:02.746 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4553999999999999e-05 2024-09-14T04:04:02.746 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.747 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.747 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.747 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.844588+0000", 2024-09-14T04:04:02.747 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7250000000000003e-06 2024-09-14T04:04:02.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.747 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.747 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.748 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.844652+0000", 2024-09-14T04:04:02.748 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3952e-05 2024-09-14T04:04:02.748 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.748 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.748 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.748 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.844663+0000", 2024-09-14T04:04:02.748 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0501e-05 2024-09-14T04:04:02.748 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.692797+0000", 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.8481341 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.692913+0000", 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115623 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.749 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.692985+0000", 2024-09-14T04:04:02.750 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2513000000000002e-05 2024-09-14T04:04:02.750 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.750 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.750 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.750 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695305+0000", 2024-09-14T04:04:02.750 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023196079999999999 2024-09-14T04:04:02.750 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.750 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:02.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:02.752 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.752 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701432+0000", 2024-09-14T04:04:02.752 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0061276029999999997 2024-09-14T04:04:02.752 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.752 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.752 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.752 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.724226+0000", 2024-09-14T04:04:02.752 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.022793971999999999 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.724324+0000", 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8113000000000001e-05 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.724376+0000", 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1344999999999997e-05 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.753 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171181 7.b 7:d9ba4f90:::benchmark_data_smithi114_118301_object50316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.848919+0000", 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.80821827900002, 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.88288272899999998, 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.754 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171181 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.848919+0000", 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.755 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.848919+0000", 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.848921+0000", 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0470000000000001e-06 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.848929+0000", 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4919999999999996e-06 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.756 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.848949+0000", 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9856000000000001e-05 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.848959+0000", 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.023e-05 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.849064+0000", 2024-09-14T04:04:02.757 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000105507 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.849074+0000", 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0229e-05 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695389+0000", 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.84631498599999999 2024-09-14T04:04:02.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695501+0000", 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011136700000000001 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.695616+0000", 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115271 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.759 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701593+0000", 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059766000000000003 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.707853+0000", 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062602339999999999 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.731699+0000", 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.023845756999999999 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.760 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.731747+0000", 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8773999999999998e-05 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.731802+0000", 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4379000000000002e-05 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.761 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171182 7.b 7:dddf5bf0:::benchmark_data_smithi114_118301_object50317:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.849194+0000", 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.80794333, 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.88282311499999999, 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171182 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.762 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.849194+0000", 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.849194+0000", 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.763 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.849195+0000", 2024-09-14T04:04:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6000000000000003e-07 2024-09-14T04:04:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.764 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.849198+0000", 2024-09-14T04:04:02.765 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5480000000000002e-06 2024-09-14T04:04:02.765 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.765 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.765 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.765 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.849209+0000", 2024-09-14T04:04:02.765 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0550999999999999e-05 2024-09-14T04:04:02.766 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.766 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.766 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.766 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.849216+0000", 2024-09-14T04:04:02.766 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7429999999999996e-06 2024-09-14T04:04:02.766 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.849272+0000", 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6521999999999999e-05 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.849279+0000", 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2350000000000002e-06 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.767 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.768 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.768 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.698465+0000", 2024-09-14T04:04:02.768 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.84918525700000003 2024-09-14T04:04:02.768 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.768 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.768 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.768 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.698575+0000", 2024-09-14T04:04:02.768 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011027900000000001 2024-09-14T04:04:02.768 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.769 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.769 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.769 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.698651+0000", 2024-09-14T04:04:02.769 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6354999999999997e-05 2024-09-14T04:04:02.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.769 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.769 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.769 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701598+0000", 2024-09-14T04:04:02.769 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029471570000000002 2024-09-14T04:04:02.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.707892+0000", 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.006293407 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.731928+0000", 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024036571 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.731976+0000", 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7902000000000003e-05 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.732017+0000", 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0828000000000003e-05 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.771 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:171225 7.b 7:ded45cc0:::benchmark_data_smithi114_118301_object50360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:51.926853+0000", 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 490.73028468000001, 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.80537582200000002, 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:02.772 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171225 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.926853+0000", 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.926853+0000", 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.774 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.774 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.926855+0000", 2024-09-14T04:04:02.774 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0949999999999998e-06 2024-09-14T04:04:02.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.775 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.775 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.926860+0000", 2024-09-14T04:04:02.775 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.485e-06 2024-09-14T04:04:02.775 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.775 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.775 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.776 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.926873+0000", 2024-09-14T04:04:02.776 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2446000000000001e-05 2024-09-14T04:04:02.776 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.776 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.776 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.926880+0000", 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0360000000000001e-06 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.926940+0000", 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0469e-05 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:51.926949+0000", 2024-09-14T04:04:02.777 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5220000000000007e-06 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701629+0000", 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.77468036699999998 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701718+0000", 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8724e-05 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.778 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.701839+0000", 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000121512 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.708082+0000", 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062426679999999998 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.708097+0000", 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5084e-05 2024-09-14T04:04:02.779 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.732133+0000", 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024036161 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.732187+0000", 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3601999999999999e-05 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.780 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:52.732229+0000", 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1650999999999998e-05 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:70092 8.7 8:e1007481:::benchmark_data_smithi114_135745_object70091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:24.314244+0000", 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 338.34289338299999, 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.037345476, 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.781 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 70092 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.314244+0000", 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.314244+0000", 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.314245+0000", 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1799999999999995e-07 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.314264+0000", 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9187000000000001e-05 2024-09-14T04:04:02.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.314275+0000", 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0781999999999999e-05 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.314280+0000", 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.728e-06 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.784 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.314324+0000", 2024-09-14T04:04:02.785 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3578000000000002e-05 2024-09-14T04:04:02.785 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.785 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.785 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.785 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.314431+0000", 2024-09-14T04:04:02.786 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010670000000000001 2024-09-14T04:04:02.786 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.786 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.786 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,6", 2024-09-14T04:04:02.786 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.314717+0000", 2024-09-14T04:04:02.786 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00028637500000000001 2024-09-14T04:04:02.787 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.787 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.787 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.787 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.331243+0000", 2024-09-14T04:04:02.787 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016526348 2024-09-14T04:04:02.787 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.788 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.788 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.788 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.335095+0000", 2024-09-14T04:04:02.788 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0038512379999999999 2024-09-14T04:04:02.788 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.789 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.789 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.789 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.335106+0000", 2024-09-14T04:04:02.789 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1623999999999999e-05 2024-09-14T04:04:02.789 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.789 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.789 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.789 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.351432+0000", 2024-09-14T04:04:02.789 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.016326107 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.351589+0000", 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015719099999999999 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:70097 8.7 8:eb1360ad:::benchmark_data_smithi114_135745_object70096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:24.319094+0000", 2024-09-14T04:04:02.790 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 338.33804323800001, 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.041128813, 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 70097 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.791 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.319094+0000", 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.319094+0000", 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.319096+0000", 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4100000000000001e-06 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.792 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.319120+0000", 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4065000000000001e-05 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.319130+0000", 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0514e-05 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.319137+0000", 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.277e-06 2024-09-14T04:04:02.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.319504+0000", 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00036614500000000001 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.319616+0000", 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011251699999999999 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.794 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,6", 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.319737+0000", 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012065000000000001 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.335149+0000", 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015412212 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.335158+0000", 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.568e-06 2024-09-14T04:04:02.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.351971+0000", 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0168131359999999 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.353494+0000", 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015235769999999999 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.796 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.360159+0000", 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0066649389999999999 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.360223+0000", 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3802999999999999e-05 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96282 8.f 8:f215a1ca:::benchmark_data_smithi114_135745_object96281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:45.775472+0000", 2024-09-14T04:04:02.797 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 316.88166578300002, 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2884918240000001, 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96282 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775472+0000", 2024-09-14T04:04:02.798 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775472+0000", 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775474+0000", 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0770000000000002e-06 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.799 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775532+0000", 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8221999999999999e-05 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775555+0000", 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3578999999999999e-05 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775566+0000", 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0643e-05 2024-09-14T04:04:02.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775648+0000", 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1833000000000001e-05 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775657+0000", 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2089999999999994e-06 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.789165+0000", 2024-09-14T04:04:02.801 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013507408 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.789175+0000", 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0295000000000001e-05 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.061696+0000", 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2725213900000001 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.802 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.063963+0000", 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002267168 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96299 8.f 8:f22c1256:::benchmark_data_smithi114_135745_object96298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:45.775587+0000", 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 316.88155060700001, 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.300857588, 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.803 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96299 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775587+0000", 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.804 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775587+0000", 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775587+0000", 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5000000000000003e-07 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775629+0000", 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2001e-05 2024-09-14T04:04:02.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775637+0000", 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5360000000000004e-06 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775642+0000", 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5419999999999996e-06 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.806 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775717+0000", 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4459e-05 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775723+0000", 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3810000000000004e-06 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.789215+0000", 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013491281000000001 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.807 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.789222+0000", 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3069999999999997e-06 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.063994+0000", 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.274772378 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.064704+0000", 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00070931900000000003 2024-09-14T04:04:02.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3", 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.064879+0000", 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017514900000000001 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.075378+0000", 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010499474 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.809 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.075423+0000", 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4369999999999997e-05 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.075477+0000", 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4388999999999997e-05 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.075549+0000", 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1897999999999995e-05 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.810 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.076444+0000", 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00089555400000000001 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96306 8.b 8:d2f20d6e:::benchmark_data_smithi114_135745_object96305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:45.775651+0000", 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 316.88148672400001, 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2857586510000001, 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.811 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96306 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775651+0000", 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.812 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775651+0000", 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775651+0000", 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1399999999999997e-07 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775689+0000", 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8365999999999998e-05 2024-09-14T04:04:02.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775696+0000", 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9670000000000004e-06 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775702+0000", 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2399999999999998e-06 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775760+0000", 2024-09-14T04:04:02.814 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8729e-05 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775767+0000", 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6560000000000003e-06 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.784615+0000", 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0088479219999999994 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.784626+0000", 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1063000000000001e-05 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.048802+0000", 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2641763290000001 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.051565+0000", 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027626059999999999 2024-09-14T04:04:02.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,4", 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.051760+0000", 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019490999999999999 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.059827+0000", 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0080676050000000003 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.817 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.059891+0000", 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3995e-05 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.059905+0000", 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3671e-05 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.059967+0000", 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1433999999999995e-05 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.819 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.819 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.061409+0000", 2024-09-14T04:04:02.819 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0014427439999999999 2024-09-14T04:04:02.819 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.819 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.819 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.819 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.819 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96308 8.f 8:f406a679:::benchmark_data_smithi114_135745_object96307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-14T04:04:02.820 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:45.775716+0000", 2024-09-14T04:04:02.820 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 316.88142174699999, 2024-09-14T04:04:02.820 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.301798966, 2024-09-14T04:04:02.820 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.820 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.820 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.820 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.821 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:04:02.821 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:04:02.821 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96308 2024-09-14T04:04:02.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.821 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.821 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.821 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775716+0000", 2024-09-14T04:04:02.822 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.822 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775716+0000", 2024-09-14T04:04:02.822 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.823 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.823 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775716+0000", 2024-09-14T04:04:02.823 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1699999999999999e-07 2024-09-14T04:04:02.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.824 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.824 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775755+0000", 2024-09-14T04:04:02.824 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8600999999999999e-05 2024-09-14T04:04:02.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.824 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.825 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.825 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775762+0000", 2024-09-14T04:04:02.825 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1770000000000001e-06 2024-09-14T04:04:02.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.825 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.825 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.825 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775768+0000", 2024-09-14T04:04:02.826 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1029999999999998e-06 2024-09-14T04:04:02.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.826 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.826 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775875+0000", 2024-09-14T04:04:02.826 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107559 2024-09-14T04:04:02.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.827 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:04:02.827 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775881+0000", 2024-09-14T04:04:02.827 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9850000000000003e-06 2024-09-14T04:04:02.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.827 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.828 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.789244+0000", 2024-09-14T04:04:02.828 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013362364 2024-09-14T04:04:02.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.828 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.828 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.789251+0000", 2024-09-14T04:04:02.828 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6930000000000001e-06 2024-09-14T04:04:02.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.829 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.829 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.068013+0000", 2024-09-14T04:04:02.829 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.278762776 2024-09-14T04:04:02.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.068153+0000", 2024-09-14T04:04:02.830 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000139703 2024-09-14T04:04:02.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.831 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,3", 2024-09-14T04:04:02.831 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.068314+0000", 2024-09-14T04:04:02.831 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016110699999999999 2024-09-14T04:04:02.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.832 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.832 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.075435+0000", 2024-09-14T04:04:02.832 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0071212599999999999 2024-09-14T04:04:02.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.832 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.832 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.076568+0000", 2024-09-14T04:04:02.833 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001133057 2024-09-14T04:04:02.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.076656+0000", 2024-09-14T04:04:02.834 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7977000000000001e-05 2024-09-14T04:04:02.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.834 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.834 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.834 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.076732+0000", 2024-09-14T04:04:02.835 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5764000000000002e-05 2024-09-14T04:04:02.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.835 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.835 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.077515+0000", 2024-09-14T04:04:02.835 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00078242299999999997 2024-09-14T04:04:02.836 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.836 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.836 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.837 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96314 8.b 8:df6260a7:::benchmark_data_smithi114_135745_object96313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-14T04:04:02.837 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:45.775782+0000", 2024-09-14T04:04:02.837 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 316.88135542999999, 2024-09-14T04:04:02.837 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.287436963, 2024-09-14T04:04:02.837 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:02.837 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:02.838 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:02.838 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:02.838 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:04:02.838 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:04:02.838 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96314 2024-09-14T04:04:02.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.839 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:02.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.839 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:02.839 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775782+0000", 2024-09-14T04:04:02.839 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.840 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.840 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:02.840 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775782+0000", 2024-09-14T04:04:02.840 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:02.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.840 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.841 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:02.841 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775782+0000", 2024-09-14T04:04:02.841 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0900000000000002e-07 2024-09-14T04:04:02.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775867+0000", 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4222999999999996e-05 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775875+0000", 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9840000000000001e-06 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.775880+0000", 2024-09-14T04:04:02.842 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8850000000000004e-06 2024-09-14T04:04:02.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.776179+0000", 2024-09-14T04:04:02.843 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00029871900000000002 2024-09-14T04:04:02.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:04:02.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.776186+0000", 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.742e-06 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.784651+0000", 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0084646770000000003 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.784659+0000", 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3799999999999994e-06 2024-09-14T04:04:02.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.056135+0000", 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2714761649999999 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.056303+0000", 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016755000000000001 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,4", 2024-09-14T04:04:02.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.056495+0000", 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000192251 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.061475+0000", 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0049795819999999998 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.061488+0000", 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3358e-05 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.061991+0000", 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00050322599999999996 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.062049+0000", 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7717999999999999e-05 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:47.063219+0000", 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001170094 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.847 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.848 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.848 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:02.848 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:02.848 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:04:02.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:02.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:02.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:02.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:03.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:03.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:03.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:03.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:03.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:03.255 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:03.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:03.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:03.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:03.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:03.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:03.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:03.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 80228 80212 113.917 65.8125 0.0127953 0.00877382 2024-09-14T04:04:03.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:03.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:03.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:03.709 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:03.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:03.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:03.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:03.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:03.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:03.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:03.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:03.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:03.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:04.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:04.125 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:04.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:04.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:04.262 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:04.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:04.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:04.378 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 0.89436 (e4f4) 2024-09-14T04:04:04.410 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:04.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:04.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:04.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:04.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 81466 81450 113.105 77.375 0.0173761 0.0088364 2024-09-14T04:04:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:04.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:04.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:04 smithi114 ceph-mon[106969]: pgmap v541: 89 pgs: 89 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T04:04:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:04 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3506640693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8943603370542378} : dispatch 2024-09-14T04:04:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:04 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8943603370542378} : dispatch 2024-09-14T04:04:04.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:04 smithi114 ceph-mon[103029]: pgmap v541: 89 pgs: 89 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T04:04:04.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:04 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3506640693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8943603370542378} : dispatch 2024-09-14T04:04:04.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:04 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8943603370542378} : dispatch 2024-09-14T04:04:04.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:04 smithi160 ceph-mon[88407]: pgmap v541: 89 pgs: 89 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T04:04:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:04 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3506640693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8943603370542378} : dispatch 2024-09-14T04:04:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:04 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.8943603370542378} : dispatch 2024-09-14T04:04:04.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:04.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:04.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:04.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:04.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-14T04:04:04.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:04.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:04.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:04.985 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:04.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:05.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:05.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:05.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:05.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:05.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:05.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:05.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:05.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:05.402 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:05.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:05.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:05.539 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:05.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:05.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 82714 82698 112.342 78 0.0170169 0.008897 2024-09-14T04:04:05.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:05 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.8943603370542378}]': finished 2024-09-14T04:04:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:05 smithi160 ceph-mon[88407]: osdmap e160: 8 total, 8 up, 8 in 2024-09-14T04:04:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:05 smithi160 ceph-mon[88407]: pgmap v543: 89 pgs: 89 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-14T04:04:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:05.680 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:05.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:05.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:05.798 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:05 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.8943603370542378}]': finished 2024-09-14T04:04:05.799 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:05 smithi114 ceph-mon[106969]: osdmap e160: 8 total, 8 up, 8 in 2024-09-14T04:04:05.799 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:05 smithi114 ceph-mon[106969]: pgmap v543: 89 pgs: 89 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-14T04:04:05.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:05 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.8943603370542378}]': finished 2024-09-14T04:04:05.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:05 smithi114 ceph-mon[103029]: osdmap e160: 8 total, 8 up, 8 in 2024-09-14T04:04:05.799 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:05 smithi114 ceph-mon[103029]: pgmap v543: 89 pgs: 89 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-14T04:04:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:05.821 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:05.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:05.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:05.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:05.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:06.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:06.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:06.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:06.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:06.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:06.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:06.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:06.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:06.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:06.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:06.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:06.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:06.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 84041 84025 111.716 82.9375 0.014714 0.00894629 2024-09-14T04:04:06.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:06.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:06.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:06.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:06.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:06.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:06.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:06.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:06.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:07.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:07.124 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:07.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:07.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:07.257 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:07.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:07.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:07.397 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:07.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:07.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:07.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:07.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:07.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:07.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 85281 85265 111.003 77.5 0.0131423 0.00900414 2024-09-14T04:04:07.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:07.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:07.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:07.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:07.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:07.808 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:07.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:07.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:07.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:07.931 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:07.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:08.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:08.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:08.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:08.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:07 smithi114 ceph-mon[103029]: pgmap v544: 89 pgs: 89 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-14T04:04:08.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:08.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:07 smithi114 ceph-mon[106969]: pgmap v544: 89 pgs: 89 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-14T04:04:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:08.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:07 smithi160 ceph-mon[88407]: pgmap v544: 89 pgs: 89 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-14T04:04:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:08.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:08.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:08.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:08.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:08.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:08.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:08.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:08.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:08.462 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:08.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:08.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 86272 86256 110.001 61.9375 0.00345188 0.00907641 2024-09-14T04:04:08.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:08.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:08.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:08.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:08.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:04:08.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:08.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:08.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:08.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:08.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:08.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:08.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:08.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:08.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:08.991 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:08.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:09.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:09.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:09.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:09.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:09.236 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:09.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:09.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:09.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:09.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:09.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:09.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:09.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:09.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:09.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:09.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:09.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 87362 87346 109.162 68.125 0.0177162 0.00915579 2024-09-14T04:04:09.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:09.708 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:09.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:09.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:09.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:09.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:09.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:09 smithi114 ceph-mon[106969]: pgmap v545: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 46 MiB/s wr, 740 op/s 2024-09-14T04:04:09.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:09 smithi114 ceph-mon[103029]: pgmap v545: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 46 MiB/s wr, 740 op/s 2024-09-14T04:04:09.932 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:09.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:09.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:09.982 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:09.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:10.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:10.133 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:10.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:09 smithi160 ceph-mon[88407]: pgmap v545: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 46 MiB/s wr, 740 op/s 2024-09-14T04:04:10.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:10.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:10.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:10.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:10.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:10.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:10.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:10.553 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 88588 88572 108.523 76.625 0.00705332 0.00920894 2024-09-14T04:04:10.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:10.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:10.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:10.660 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:04:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:04:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:04:10.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:10.716 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:10.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:10.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:10.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:10.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:10.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:10.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:10.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:10.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:11.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:11.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:11.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:11.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:11.281 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:11.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:11.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:11.420 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:11.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:11.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:11.521 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-14T04:04:11.522 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:04:11.522 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-14T04:04:11.522 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:04:11.522 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:04:11.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-14T04:04:11.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:11.554 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 89797 89781 107.886 75.5625 0.0121506 0.0092637 2024-09-14T04:04:11.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:11.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:11.557 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:04:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:04:11] "GET /metrics HTTP/1.1" 200 36443 "" "Prometheus/2.51.0" 2024-09-14T04:04:11.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:11.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:11.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:11.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:11.823 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:11.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:11.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:11.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:11.924 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:04:11.924 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:04:11.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:11.964 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:04:11.965 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:04:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:11.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:11.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:11.996 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:04:11.997 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:04:12.031 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:04:12.032 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:04:12.061 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:04:12.061 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:04:12.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:12.098 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:04:12.099 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:04:12.123 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:12.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:12.127 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:04:12.127 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:04:12.157 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:04:12.157 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:04:12.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:12.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:12.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:12.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:12.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[106969]: pgmap v546: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-14T04:04:12.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2230953445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[103029]: pgmap v546: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2230953445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:04:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:04:12.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:12.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:12.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:12.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:12.537 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:12.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:12.554 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 15 90891 90876 107.142 68.4375 0.0119252 0.00932873 2024-09-14T04:04:12.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:12 smithi160 ceph-mon[88407]: pgmap v546: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-14T04:04:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:12 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2230953445' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:04:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:04:12.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:12.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:12.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:12.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:12.778 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:12.813 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:12.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:12.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:12.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:12.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:13.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:13.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:13.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:13.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:13.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:13.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:13.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:13.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:13.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:13.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:13.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:13.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:13.506 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:13.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:13.555 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 92148 92132 106.612 78.5 0.0142392 0.00937518 2024-09-14T04:04:13.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:13.645 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:13.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:13.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:13.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:13.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:13.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:13.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:13.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:13.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:14.020 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:14.041 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:14.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:14.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:14.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:14.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:14.196 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:14.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:14.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:14.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:14.334 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:14.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:14.335 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:04:14.335 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:04:14.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:14 smithi114 ceph-mon[103029]: pgmap v547: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-14T04:04:14.420 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:14 smithi114 ceph-mon[106969]: pgmap v547: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-14T04:04:14.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:14.471 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:14 smithi160 ceph-mon[88407]: pgmap v547: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-14T04:04:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:14.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:14.556 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 93370 93354 106.06 76.375 0.0189445 0.00942432 2024-09-14T04:04:14.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:14.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:14.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:14.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:14.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:14.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:14.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:14.847 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:14.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:14.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:14.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:04:14.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:15.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:15.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:15.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:15.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:15.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:15.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:15.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:15.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:15.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:15.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:15.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:15.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:15.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:15.556 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 94513 94497 105.441 71.4375 0.0262648 0.00947905 2024-09-14T04:04:15.565 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:15.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:15.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:15.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:15.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:15.808 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:15.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:15.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:15.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:15.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:15.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:15.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:16.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:16.119 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:16.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:16.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:16.260 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:16.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:16.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:16.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:16.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:16.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:16.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:16.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:16.556 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 15 95738 95723 104.936 76.625 0.0230578 0.00952517 2024-09-14T04:04:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:16 smithi114 ceph-mon[103029]: pgmap v548: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-14T04:04:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:16 smithi114 ceph-mon[106969]: pgmap v548: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-14T04:04:16.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:16 smithi160 ceph-mon[88407]: pgmap v548: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-14T04:04:16.668 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:16.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:16.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-14T04:04:16.771 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:16.812 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:16.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:16.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:16.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:16.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:16.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:17.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:17.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:17.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:17.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:17.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:17.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:17.285 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:17.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:17.382 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:17.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:17.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:17.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:17.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:17.556 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 96871 96855 104.346 70.75 0.02197 0.00957864 2024-09-14T04:04:17.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:17.666 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:17.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:17.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:17.803 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:17.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:17.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:17.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:17.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:17.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:18.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:18.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:18.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:18.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:18.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:18.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:18.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:18.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:18.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:18.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:18.362 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:04:18.362 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-14T04:04:18.362 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:04:18.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:18 smithi114 ceph-mon[106969]: pgmap v549: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s 2024-09-14T04:04:18.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:18 smithi114 ceph-mon[103029]: pgmap v549: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s 2024-09-14T04:04:18.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:18.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:18.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:18.556 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 98019 98003 103.794 71.75 0.0132103 0.0096301 2024-09-14T04:04:18.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:18.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:18.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:18 smithi160 ceph-mon[88407]: pgmap v549: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s 2024-09-14T04:04:18.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:18.768 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:18.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:18.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:18.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:18.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:18.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:19.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:19.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:19.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:19.168 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:19.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:19.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:19.270 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:19.308 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:19.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:19.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:19.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:19.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:19.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:19.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:19.556 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:04:19.556083+0000 min lat: 0.00111329 max lat: 0.336723 avg lat: 0.00967435 2024-09-14T04:04:19.556 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:04:19.556 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 99222 99206 103.317 75.1875 0.0111008 0.00967435 2024-09-14T04:04:19.591 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:19.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:19.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:19.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:19.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:19.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:19.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:19.739 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:19.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:19.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:19.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:19.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:20.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:20.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:20.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:20.142 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:20.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:20.211 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:04:20.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:20.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:20.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:20.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:20.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:20.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:20.556 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:20.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:20.557 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 100350 100334 102.778 70.5 0.011836 0.00972464 2024-09-14T04:04:20.604 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:20 smithi114 ceph-mon[106969]: pgmap v550: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-14T04:04:20.604 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:20 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1723270171' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:04:20.604 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:20 smithi114 ceph-mon[103029]: pgmap v550: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-14T04:04:20.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:20 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1723270171' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:04:20.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:04:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:04:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:04:20.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:20 smithi160 ceph-mon[88407]: pgmap v550: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-14T04:04:20.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:20 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1723270171' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:04:20.696 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:20.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:20.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:20.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:20.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:20.937 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:20.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:20.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:21.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:21.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:21.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:21.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:21.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:21.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:21.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:21.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:21.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:21.379 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:21.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:21.453 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:04:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:04:21] "GET /metrics HTTP/1.1" 200 36443 "" "Prometheus/2.51.0" 2024-09-14T04:04:21.453 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:21 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:04:21.453 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1723270171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:04:21.453 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:21 smithi114 ceph-mon[106969]: osdmap e161: 8 total, 8 up, 8 in 2024-09-14T04:04:21.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:21 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:04:21.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1723270171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:04:21.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:21 smithi114 ceph-mon[103029]: osdmap e161: 8 total, 8 up, 8 in 2024-09-14T04:04:21.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:21.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:21.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:21.557 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 101399 101383 102.178 65.5625 0.0149315 0.00978143 2024-09-14T04:04:21.601 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:04:21.601 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:04:21.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:21.656 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:21.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:21 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:04:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1723270171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:04:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:21 smithi160 ceph-mon[88407]: osdmap e161: 8 total, 8 up, 8 in 2024-09-14T04:04:21.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:21.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:21.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:21.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:21.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:21.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:22.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:22.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:22.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:22.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:22.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:22.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:22.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:22.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:22.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:22.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:22.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:22 smithi114 ceph-mon[103029]: pgmap v552: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-14T04:04:22.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:22 smithi114 ceph-mon[106969]: pgmap v552: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-14T04:04:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:22.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:22.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:22.560 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 102576 102560 101.724 73.5625 0.0115973 0.00982594 2024-09-14T04:04:22.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:22.644 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:22.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:22 smithi160 ceph-mon[88407]: pgmap v552: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-14T04:04:22.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:22.787 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:22.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:22.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:22.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:22.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:23.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:23.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:23.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:23.166 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:23.205 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:23.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:23.307 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:23.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:23.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:23.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:23.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:23.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:23.557 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 103739 103723 101.27 72.6875 0.00726743 0.00986933 2024-09-14T04:04:23.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:23.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-14T04:04:23.627 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:23.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:23.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:23.770 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:23.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:23.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:23 smithi114 ceph-mon[106969]: pgmap v553: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T04:04:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:23 smithi114 ceph-mon[103029]: pgmap v553: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T04:04:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:23.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:23 smithi160 ceph-mon[88407]: pgmap v553: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T04:04:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:04:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:23.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:23.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:24.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:24.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:24.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:24.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:24.184 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:24.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:24.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:24.321 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:24.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:24.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:24.463 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:24.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:24.557 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 104808 104792 100.74 66.8125 0.0336029 0.0099217 2024-09-14T04:04:24.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:24.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:24.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:24.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:24.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:24.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:24.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:24.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:24.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:24.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:24.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:24.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:04:24.976 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:25.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:25.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:25.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:25.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:25.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:25.180 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-14T04:04:25.181 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:04:25.181 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-14T04:04:25.181 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-14T04:04:25.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:25.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:25.316 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:25.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:25.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:25.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:25.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:25.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:25.557 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 105791 105775 100.144 61.4375 0.00979269 0.00998087 2024-09-14T04:04:25.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:25.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:25.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:25.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:25.729 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:25.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:25.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:25.868 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:25.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:25 smithi160 ceph-mon[88407]: pgmap v554: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T04:04:25.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:26.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:26.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:26.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:26.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:26.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:25 smithi114 ceph-mon[106969]: pgmap v554: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T04:04:26.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:25 smithi114 ceph-mon[103029]: pgmap v554: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T04:04:26.140 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:26.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:26.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:26.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:26.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:26.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:26.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:26.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:26.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:26.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:26.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:26.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:26.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:26.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:26.557 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 15 106829 106814 99.6188 64.9375 0.083824 0.0100314 2024-09-14T04:04:26.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:26.673 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:26.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:26.698 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:26 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3302587039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:26.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:26 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3302587039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:26.775 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:26.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:26.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:26 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3302587039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:26.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:26.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:26.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:27.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:27.090 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:27.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:27.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:27.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:27.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:27.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:27.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:27.367 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:27.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:27.394 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:04:27.394 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:04:27.395 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:04:27.395 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:04:27.395 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:04:27.395 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:04:27.395 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:04:27.395 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:04:27.395 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:04:27.395 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:04:27.396 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:04:27.396 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:04:27.396 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:04:27.396 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:04:27.396 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:04:27.396 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:04:27.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:27.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:27.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:27.557 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 107945 107929 99.1786 69.6875 0.00780683 0.0100787 2024-09-14T04:04:27.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:27.648 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:27.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:27.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:27.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:27.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:27.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:27.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:27.909 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:27.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:28.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:28.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:28.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:28.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:27 smithi114 ceph-mon[103029]: pgmap v555: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T04:04:28.052 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:27 smithi114 ceph-mon[106969]: pgmap v555: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T04:04:28.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:27 smithi160 ceph-mon[88407]: pgmap v555: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-14T04:04:28.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:28.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:28.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:28.318 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:28.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:28.408 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:28.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:28.465 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:28.490 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:28.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:28.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:28.559 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 109005 108989 98.6986 66.25 0.0152349 0.0101262 2024-09-14T04:04:28.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:28.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:28.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:28.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:28.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:28.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:28.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:28.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:28.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:28.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:28.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:28.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:29.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:29.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:29.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:29.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:29.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:29.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:29.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:29.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:29.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:29.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:29.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:29.404 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:29.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:29.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:29.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:29.559 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 110097 110081 98.2636 68.25 0.00886059 0.0101725 2024-09-14T04:04:29.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:29.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:29.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:29.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:29.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:29.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:29.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:29.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:29 smithi114 ceph-mon[103029]: pgmap v556: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-14T04:04:29.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:29 smithi114 ceph-mon[106969]: pgmap v556: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-14T04:04:29.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:29.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:29.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:30.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:30.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:30.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:30.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:30.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:30.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:30.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:29 smithi160 ceph-mon[88407]: pgmap v556: 89 pgs: 89 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-14T04:04:30.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:30.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:30.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:30.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:30.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:04:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:30.439 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:30.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:30.541 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:30.559 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 111160 111144 97.8153 66.4375 0.0120539 0.0102184 2024-09-14T04:04:30.574 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:30.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:04:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:04:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:04:30.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:30.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:30.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:30.852 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:30.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:30.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:30.983 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:30.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:31.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:31.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:31.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:31.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:31.256 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:31.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:31.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:31.381 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:31.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:31.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:31.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:31.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:31.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:31.560 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 111982 111966 97.1703 51.375 0.0126356 0.0102865 2024-09-14T04:04:31.605 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:04:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:04:31] "GET /metrics HTTP/1.1" 200 36437 "" "Prometheus/2.51.0" 2024-09-14T04:04:31.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:31.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:31.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:31.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:31.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:31.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:31 smithi114 ceph-mon[106969]: pgmap v557: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-14T04:04:31.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:31 smithi114 ceph-mon[103029]: pgmap v557: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-14T04:04:31.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:31.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:31.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:32.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:32.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:32.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:31 smithi160 ceph-mon[88407]: pgmap v557: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-14T04:04:32.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:32.215 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:32.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:32.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:32.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:32.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:32.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:32.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:32.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:32.560 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 112956 112940 96.6731 60.875 0.0122713 0.0103396 2024-09-14T04:04:32.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:32.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:32.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:32.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:32.785 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:32.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:32.886 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:32.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:32.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:33.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:33.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:33.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:33.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:33.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:33.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:33.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:33.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:33.332 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:33.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:33.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:33.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:33.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:33.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:33.560 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 113961 113945 96.2156 62.8125 0.0315833 0.0103882 2024-09-14T04:04:33.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:33.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:33.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:33.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:33.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:33.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:33.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:33.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:33 smithi114 ceph-mon[103029]: pgmap v558: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 992 op/s 2024-09-14T04:04:33.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:33 smithi114 ceph-mon[106969]: pgmap v558: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 992 op/s 2024-09-14T04:04:33.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:33.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:33.965 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:04:33.965 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:04:33.965 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:04:33.965 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:174757 7.e 7.51dc351e (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e109)", 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:57.242785+0000", 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 516.71492887600004, 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99647101599999999, 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:33.966 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174757 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242785+0000", 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242785+0000", 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.968 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:33.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242786+0000", 2024-09-14T04:04:33.968 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3540000000000001e-06 2024-09-14T04:04:33.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.968 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:33.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242791+0000", 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9960000000000001e-06 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242799+0000", 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6320000000000008e-06 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242808+0000", 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6719999999999993e-06 2024-09-14T04:04:33.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242883+0000", 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4781000000000001e-05 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242890+0000", 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0149999999999998e-06 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260933+0000", 2024-09-14T04:04:33.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.018043551000000001 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260942+0000", 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1279999999999997e-06 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.239245+0000", 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97830248799999997 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.239256+0000", 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1399e-05 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:174771 7.e 7.c7a0a73e (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e109)", 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:57.242837+0000", 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 516.71487669600003, 2024-09-14T04:04:33.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99646744499999995, 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174771 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242837+0000", 2024-09-14T04:04:33.973 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242837+0000", 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242838+0000", 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9499999999999998e-07 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242840+0000", 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8059999999999999e-06 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242847+0000", 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1739999999999997e-06 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242852+0000", 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5550000000000002e-06 2024-09-14T04:04:33.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242940+0000", 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7332999999999999e-05 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242944+0000", 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7779999999999998e-06 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.976 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260967+0000", 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.018022765999999999 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260974+0000", 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9940000000000003e-06 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.239295+0000", 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97832120099999997 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.977 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.239305+0000", 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4429999999999992e-06 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:174772 7.8 7.3ed11168 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e109)", 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:57.242907+0000", 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 516.71480654499999, 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.994613937, 2024-09-14T04:04:33.978 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174772 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242907+0000", 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242907+0000", 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242908+0000", 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.01e-07 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242910+0000", 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1610000000000001e-06 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242917+0000", 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6680000000000004e-06 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242922+0000", 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.648e-06 2024-09-14T04:04:33.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242970+0000", 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7324000000000003e-05 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.242975+0000", 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7300000000000002e-06 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.254802+0000", 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011826827 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.254819+0000", 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6659000000000001e-05 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.237510+0000", 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98269114400000002 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.983 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.237521+0000", 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1375e-05 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:174779 7.8 7.8b6ecc38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:57.249332+0000", 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 516.70838198900003, 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98822152299999999, 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:33.984 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174779 2024-09-14T04:04:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:33.985 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249332+0000", 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249332+0000", 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249334+0000", 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.669e-06 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:33.987 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249339+0000", 2024-09-14T04:04:33.987 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1429999999999997e-06 2024-09-14T04:04:33.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:33.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249351+0000", 2024-09-14T04:04:33.988 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1727e-05 2024-09-14T04:04:33.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:33.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249364+0000", 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3097e-05 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.254845+0000", 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054815829999999999 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.254852+0000", 2024-09-14T04:04:33.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3029999999999996e-06 2024-09-14T04:04:33.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.990 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.990 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.990 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.237545+0000", 2024-09-14T04:04:33.990 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98269336900000004 2024-09-14T04:04:33.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.990 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.990 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:33.990 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.237554+0000", 2024-09-14T04:04:33.990 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6319999999999997e-06 2024-09-14T04:04:33.991 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:33.991 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:33.991 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:33.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.991 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.991 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:174781 7.8 7.5b9ce798 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-14T04:04:33.991 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:57.249804+0000", 2024-09-14T04:04:33.991 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 516.70791027099995, 2024-09-14T04:04:33.991 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98777971200000003, 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174781 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.992 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249804+0000", 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249804+0000", 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:33.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249804+0000", 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5199999999999996e-07 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249807+0000", 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9079999999999999e-06 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249826+0000", 2024-09-14T04:04:33.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8981999999999999e-05 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.249834+0000", 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1459999999999999e-06 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.254872+0000", 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0050384119999999999 2024-09-14T04:04:33.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.254878+0000", 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8409999999999996e-06 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.237575+0000", 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98269694200000002 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.237583+0000", 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7290000000000006e-06 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:174785 7.8 7.66fab1b8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:55:57.260426+0000", 2024-09-14T04:04:33.997 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 516.69728755799997, 2024-09-14T04:04:33.998 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97718810700000003, 2024-09-14T04:04:33.998 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:33.998 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:33.998 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:33.998 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:33.998 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:33.998 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:33.998 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174785 2024-09-14T04:04:33.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.998 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260426+0000", 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260426+0000", 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:33.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260428+0000", 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.792e-06 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260434+0000", 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8100000000000003e-06 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.000 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260447+0000", 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3311e-05 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260455+0000", 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8590000000000005e-06 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260566+0000", 2024-09-14T04:04:34.001 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110302 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:57.260576+0000", 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0906e-05 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.237607+0000", 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97703008999999996 2024-09-14T04:04:34.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:55:58.237615+0000", 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0369999999999995e-06 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:184477 7.8 7:1f777fe4:::benchmark_data_smithi114_118301_object63612:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:12.804981+0000", 2024-09-14T04:04:34.003 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 501.15273295700001, 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.025234347, 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 184477 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.804981+0000", 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.804981+0000", 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.804983+0000", 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5039999999999999e-06 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.804988+0000", 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4650000000000002e-06 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805002+0000", 2024-09-14T04:04:34.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4061e-05 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805011+0000", 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3700000000000001e-06 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805585+0000", 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00057357800000000004 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.007 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805598+0000", 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2856e-05 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.830136+0000", 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0245385119999999 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.830215+0000", 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9000999999999998e-05 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:184482 7.8 7:14335b58:::benchmark_data_smithi114_118301_object63617:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:12.805025+0000", 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 501.15268855099998, 2024-09-14T04:04:34.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.025688731, 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 184482 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805025+0000", 2024-09-14T04:04:34.011 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.011 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805025+0000", 2024-09-14T04:04:34.011 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.011 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805026+0000", 2024-09-14T04:04:34.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3799999999999998e-07 2024-09-14T04:04:34.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805028+0000", 2024-09-14T04:04:34.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2409999999999998e-06 2024-09-14T04:04:34.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805037+0000", 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7409999999999998e-06 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805042+0000", 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1279999999999999e-06 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805632+0000", 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00059015900000000004 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805641+0000", 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0890000000000006e-06 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.830652+0000", 2024-09-14T04:04:34.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0250111740000001 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.830714+0000", 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1760999999999994e-05 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:184494 7.8 7:135520d5:::benchmark_data_smithi114_118301_object63629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:04:34.015 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:12.805052+0000", 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 501.15266205900002, 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0509660430000001, 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 184494 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.016 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:34.018 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.018 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805052+0000", 2024-09-14T04:04:34.018 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805052+0000", 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805052+0000", 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.77e-07 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.020 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805054+0000", 2024-09-14T04:04:34.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.719e-06 2024-09-14T04:04:34.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.020 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.020 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805063+0000", 2024-09-14T04:04:34.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6149999999999997e-06 2024-09-14T04:04:34.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.020 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805068+0000", 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1760000000000001e-06 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805663+0000", 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00059564100000000005 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.805670+0000", 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3069999999999999e-06 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.830737+0000", 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.025066912 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.830840+0000", 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103059 2024-09-14T04:04:34.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.830950+0000", 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011069300000000001 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.840138+0000", 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0091879609999999997 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.840188+0000", 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9945999999999999e-05 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.855857+0000", 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015668453999999998 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.855960+0000", 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103451 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856018+0000", 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7732000000000002e-05 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:184513 7.8 7:10a55626:::benchmark_data_smithi114_118301_object63648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:12.837189+0000", 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 501.12052468299999, 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.019082311, 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.026 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 184513 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.837189+0000", 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.837189+0000", 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.837191+0000", 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.782e-06 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.837197+0000", 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6160000000000001e-06 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.837214+0000", 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6872e-05 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.837223+0000", 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1970000000000002e-06 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.839892+0000", 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026688409999999999 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.839916+0000", 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4508999999999999e-05 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.833972+0000", 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99405637499999999 2024-09-14T04:04:34.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.834078+0000", 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000105888 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.834170+0000", 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1739999999999999e-05 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:34.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.840199+0000", 2024-09-14T04:04:34.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0060288779999999997 2024-09-14T04:04:34.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.847954+0000", 2024-09-14T04:04:34.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0077550290000000001 2024-09-14T04:04:34.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856061+0000", 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0081067640000000007 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856232+0000", 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000171211 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856272+0000", 2024-09-14T04:04:34.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9609e-05 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:184519 7.8 7:1796d333:::benchmark_data_smithi114_118301_object63654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:12.852278+0000", 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 501.10543584800001, 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0041445630000001, 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.034 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 184519 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.852278+0000", 2024-09-14T04:04:34.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.852278+0000", 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.852280+0000", 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0700000000000001e-06 2024-09-14T04:04:34.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.852288+0000", 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1119999999999996e-06 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.852305+0000", 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7039999999999999e-05 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.852315+0000", 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3700000000000001e-06 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.852491+0000", 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017601199999999999 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.852518+0000", 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7370999999999999e-05 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.836579+0000", 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98406065300000001 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:34.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.836696+0000", 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011760399999999999 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.836816+0000", 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011940800000000001 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.840203+0000", 2024-09-14T04:04:34.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033874460000000001 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.855707+0000", 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015503566999999999 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856310+0000", 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00060347000000000005 2024-09-14T04:04:34.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856388+0000", 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7451000000000001e-05 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856423+0000", 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4989e-05 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.042 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:184525 7.8 7:185d559b:::benchmark_data_smithi114_118301_object63660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:12.860430+0000", 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 501.09728446299999, 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99613716200000002, 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.043 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 184525 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.860430+0000", 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.860430+0000", 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.860431+0000", 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6440000000000001e-06 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.860439+0000", 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.452e-06 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.860453+0000", 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4085e-05 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.860461+0000", 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9379999999999992e-06 2024-09-14T04:04:34.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.860541+0000", 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0678000000000002e-05 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.860553+0000", 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1698e-05 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.840227+0000", 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97967382199999997 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.840334+0000", 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000106809 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.840437+0000", 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103368 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.855744+0000", 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015307474 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.049 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856460+0000", 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00071504100000000003 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856478+0000", 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8597000000000002e-05 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856531+0000", 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2432000000000002e-05 2024-09-14T04:04:34.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856567+0000", 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6124000000000002e-05 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:184536 7.8 7:1317025d:::benchmark_data_smithi114_118301_object63671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:04:34.051 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:56:12.885424+0000", 2024-09-14T04:04:34.052 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 501.07229038700001, 2024-09-14T04:04:34.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97501704, 2024-09-14T04:04:34.052 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.052 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.052 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:34.052 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.052 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.052 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.052 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 184536 2024-09-14T04:04:34.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.885424+0000", 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.885424+0000", 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.885425+0000", 2024-09-14T04:04:34.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7910000000000001e-06 2024-09-14T04:04:34.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.885432+0000", 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7139999999999996e-06 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.885451+0000", 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9043e-05 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.885461+0000", 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5659999999999996e-06 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.885645+0000", 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000183831 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:12.885668+0000", 2024-09-14T04:04:34.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3825999999999999e-05 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.844324+0000", 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.95865532399999998 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.844418+0000", 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3889999999999997e-05 2024-09-14T04:04:34.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.844513+0000", 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5877000000000004e-05 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.855781+0000", 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011267452000000001 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.856581+0000", 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00079984100000000003 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.860304+0000", 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037235599999999999 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.860395+0000", 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0587000000000005e-05 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:56:13.860441+0000", 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5738000000000002e-05 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:223576 7.e 7:748854a7:::benchmark_data_smithi114_118301_object102711:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e118)", 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:57:08.309674+0000", 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 445.64804036200002, 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0403940890000001, 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.061 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 223576 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309674+0000", 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309674+0000", 2024-09-14T04:04:34.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309675+0000", 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.513e-06 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309679+0000", 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6600000000000001e-06 2024-09-14T04:04:34.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309698+0000", 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9485999999999999e-05 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309709+0000", 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0308e-05 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.310009+0000", 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030003799999999998 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.310028+0000", 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9454999999999999e-05 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.318879+0000", 2024-09-14T04:04:34.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0088511920000000008 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.318889+0000", 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9680000000000001e-06 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.349986+0000", 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0310968069999999 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.350068+0000", 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1662000000000005e-05 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:223578 7.e 7:7e045313:::benchmark_data_smithi114_118301_object102713:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e118)", 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:57:08.309736+0000", 2024-09-14T04:04:34.067 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 445.64797796599998, 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0627633030000001, 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 223578 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309736+0000", 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309736+0000", 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309737+0000", 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.44e-07 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309739+0000", 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8380000000000002e-06 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309747+0000", 2024-09-14T04:04:34.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9009999999999994e-06 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309754+0000", 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1959999999999996e-06 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.310539+0000", 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00078562200000000003 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.310557+0000", 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7685000000000001e-05 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.318913+0000", 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0083559359999999996 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.072 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.318922+0000", 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2159999999999995e-06 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.350465+0000", 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0315435319999999 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:34.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.350557+0000", 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1180000000000005e-05 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.350667+0000", 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110604 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.357689+0000", 2024-09-14T04:04:34.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0070213419999999999 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.357744+0000", 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5525000000000001e-05 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.372416+0000", 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014672286999999999 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.372452+0000", 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5074999999999999e-05 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.372499+0000", 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.782e-05 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.076 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:223581 7.e 7:76c2bd77:::benchmark_data_smithi114_118301_object102716:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e118)", 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:57:08.309761+0000", 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 445.64795272600003, 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.069464907, 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.077 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 223581 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309761+0000", 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309761+0000", 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309762+0000", 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5400000000000002e-07 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309764+0000", 2024-09-14T04:04:34.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9759999999999998e-06 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309769+0000", 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6500000000000001e-06 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.309775+0000", 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2850000000000002e-06 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.310693+0000", 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00091803000000000002 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.310709+0000", 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6328999999999999e-05 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.318942+0000", 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0082329210000000007 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.318952+0000", 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6690000000000001e-06 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.354437+0000", 2024-09-14T04:04:34.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.035485472 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.354565+0000", 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000127534 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.354690+0000", 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000125069 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.357773+0000", 2024-09-14T04:04:34.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030831610000000001 2024-09-14T04:04:34.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:34.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.361542+0000", 2024-09-14T04:04:34.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037695789999999999 2024-09-14T04:04:34.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.379128+0000", 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017585324999999999 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.379185+0000", 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6901000000000003e-05 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.379226+0000", 2024-09-14T04:04:34.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1551999999999998e-05 2024-09-14T04:04:34.086 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.086 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.086 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.086 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:04:34.087 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:04:34.088 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.44156.0:223633 7.e 7:7c95255c:::benchmark_data_smithi114_118301_object102768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-14T04:04:34.088 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:57:08.367801+0000", 2024-09-14T04:04:34.088 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 445.58991255000001, 2024-09-14T04:04:34.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.011552778, 2024-09-14T04:04:34.088 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.088 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.44156", 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 223633 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.367801+0000", 2024-09-14T04:04:34.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.367801+0000", 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.367803+0000", 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9870000000000002e-06 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.367816+0000", 2024-09-14T04:04:34.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2098e-05 2024-09-14T04:04:34.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.367899+0000", 2024-09-14T04:04:34.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3138000000000003e-05 2024-09-14T04:04:34.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.367907+0000", 2024-09-14T04:04:34.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1009999999999992e-06 2024-09-14T04:04:34.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.367975+0000", 2024-09-14T04:04:34.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7766000000000001e-05 2024-09-14T04:04:34.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:08.367984+0000", 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7540000000000001e-06 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.357807+0000", 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98982282399999999 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.357939+0000", 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013158700000000001 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,5", 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.358050+0000", 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110939 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.365002+0000", 2024-09-14T04:04:34.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0069521670000000004 2024-09-14T04:04:34.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:34.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.372380+0000", 2024-09-14T04:04:34.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.007378267 2024-09-14T04:04:34.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.379275+0000", 2024-09-14T04:04:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0068953679999999998 2024-09-14T04:04:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.379318+0000", 2024-09-14T04:04:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2471e-05 2024-09-14T04:04:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:57:09.379354+0000", 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6310999999999998e-05 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:70093 8.f 8:faf6a5d8:::benchmark_data_smithi114_135745_object70092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e133)", 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:24.324432+0000", 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 369.633282055, 2024-09-14T04:04:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0070298499999999, 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 70093 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.324432+0000", 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.324432+0000", 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.324433+0000", 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.548e-06 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.324481+0000", 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7531000000000003e-05 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.324490+0000", 2024-09-14T04:04:34.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2189999999999993e-06 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.324498+0000", 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0809999999999994e-06 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.325292+0000", 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00079358300000000005 2024-09-14T04:04:34.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.325301+0000", 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7320000000000004e-06 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.330194+0000", 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.004893547 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.102 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.331462+0000", 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0012676090000000001 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:70110 8.f 8:fa1d15f6:::benchmark_data_smithi114_135745_object70109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:24.335896+0000", 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 369.62181762099999, 2024-09-14T04:04:34.103 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0031520110000001, 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 70110 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.335896+0000", 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.335896+0000", 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.335898+0000", 2024-09-14T04:04:34.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.736e-06 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.335922+0000", 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3532000000000001e-05 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.335935+0000", 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3488e-05 2024-09-14T04:04:34.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.335943+0000", 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6070000000000002e-06 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.336003+0000", 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0010999999999999e-05 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.336010+0000", 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6310000000000003e-06 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.331786+0000", 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99577522500000004 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:34.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.332100+0000", 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00031440500000000002 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1", 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.332260+0000", 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015961099999999999 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.337010+0000", 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0047502940000000004 2024-09-14T04:04:34.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.338887+0000", 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001877578 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.338956+0000", 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8715999999999997e-05 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.339048+0000", 2024-09-14T04:04:34.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2176999999999995e-05 2024-09-14T04:04:34.111 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.111 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.111 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.111 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:70113 8.f 8:fdc1663b:::benchmark_data_smithi114_135745_object70112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-14T04:04:34.111 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:24.339283+0000", 2024-09-14T04:04:34.111 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 369.61843099100003, 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99996335800000002, 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 70113 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.339283+0000", 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.339283+0000", 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.339285+0000", 2024-09-14T04:04:34.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.843e-06 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.339312+0000", 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7135999999999999e-05 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.339326+0000", 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4063000000000001e-05 2024-09-14T04:04:34.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.339332+0000", 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9190000000000004e-06 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.339399+0000", 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7300999999999998e-05 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:24.339408+0000", 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0129999999999999e-06 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.334545+0000", 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99513665799999995 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.334680+0000", 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013459499999999999 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1", 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.334827+0000", 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014732600000000001 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:04:34.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.337017+0000", 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021903209999999998 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.339110+0000", 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020923489999999999 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.339163+0000", 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3529000000000002e-05 2024-09-14T04:04:34.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:04:34.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:04:34.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:04:34.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:25.339246+0000", 2024-09-14T04:04:34.119 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3305000000000003e-05 2024-09-14T04:04:34.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.119 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:04:34.119 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:04:34.119 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:04:34.158 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:33 smithi160 ceph-mon[88407]: pgmap v558: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 62 MiB/s wr, 992 op/s 2024-09-14T04:04:34.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:34.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:34.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:34.299 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:34.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:34.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:34.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:34.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:34.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:34.560 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 114945 114929 95.7527 61.5 0.00608844 0.010439 2024-09-14T04:04:34.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:34.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:34.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:34.720 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:34.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:34.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:34.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:34.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:34.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:34.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:34.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:34.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:35.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:35.127 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:35.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:35.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:35.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-14T04:04:35.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:35.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:35.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:35.400 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:35.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:35.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:35.541 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:35.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:35.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 115940 115924 95.31 62.1875 0.0125794 0.0104872 2024-09-14T04:04:35.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:35.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:35.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:35.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:35.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:35.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:35.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:35.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:35.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:35 smithi114 ceph-mon[103029]: pgmap v559: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-14T04:04:35.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:35 smithi114 ceph-mon[106969]: pgmap v559: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-14T04:04:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:04:35.874 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:04:35.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-14T04:04:35.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:35.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:35.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:35.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:36.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:36.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:36.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:35 smithi160 ceph-mon[88407]: pgmap v559: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-14T04:04:36.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:36.257 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:36.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:36.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:36.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:36.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:36.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:36.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:36.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:36.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:36.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 116921 116905 94.8684 61.3125 0.0141045 0.0105361 2024-09-14T04:04:36.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:36.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:36.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:36.754 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:36.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:36.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:36.879 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:36.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:36.927 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:36.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:36.965 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:37.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:37.095 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:37.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:37.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:37.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:37.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:37.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:37.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:37.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:37.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:37.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:37.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:37.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 117914 117898 94.4478 62.0625 0.0150304 0.0105826 2024-09-14T04:04:37.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:37.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:37.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:37.695 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:04:37.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:37.805 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:37.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:37.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:37.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:37.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:37.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:38.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:38.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:38.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:38.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:38.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:37 smithi114 ceph-mon[103029]: pgmap v560: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s 2024-09-14T04:04:38.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:37 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2181976413' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:04:38.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:38.097 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:37 smithi114 ceph-mon[106969]: pgmap v560: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s 2024-09-14T04:04:38.097 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:37 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2181976413' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:04:38.097 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:37 smithi160 ceph-mon[88407]: pgmap v560: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s 2024-09-14T04:04:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:37 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2181976413' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:04:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:38.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:38.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:38.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:38.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:38.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:38.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:38.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:38.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:38.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:38.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 118871 118855 94.0094 59.8125 0.00700937 0.0106326 2024-09-14T04:04:38.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:38.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:38.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:38.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:38.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:38.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:38 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:04:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2181976413' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:04:38.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:38 smithi114 ceph-mon[103029]: osdmap e162: 8 total, 8 up, 8 in 2024-09-14T04:04:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:38 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:04:38.775 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2181976413' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:04:38.776 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:38 smithi114 ceph-mon[106969]: osdmap e162: 8 total, 8 up, 8 in 2024-09-14T04:04:38.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:38.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:38.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:39.012 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:39.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:39.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:39.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:38 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:04:39.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2181976413' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:04:39.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:38 smithi160 ceph-mon[88407]: osdmap e162: 8 total, 8 up, 8 in 2024-09-14T04:04:39.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:39.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:39.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:39.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:39.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:39.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:39.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:39.468 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:39.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:39.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:04:39.561090+0000 min lat: 0.00111329 max lat: 0.336723 avg lat: 0.0106736 2024-09-14T04:04:39.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:04:39.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 119916 119900 93.6507 65.3125 0.0091334 0.0106736 2024-09-14T04:04:39.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:39.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:39.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:39.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:39.758 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:39.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:39.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:39.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:39.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:39.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:40.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:40.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:40.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:40.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:40.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:40.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:40.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:40.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:40.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:40.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:40 smithi114 ceph-mon[103029]: pgmap v562: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-14T04:04:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:40 smithi114 ceph-mon[106969]: pgmap v562: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-14T04:04:40.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:40.446 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:40.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:40.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:40.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 120920 120904 93.2692 62.75 0.0165492 0.0107171 2024-09-14T04:04:40.593 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:40.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:04:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:04:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:04:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:40 smithi160 ceph-mon[88407]: pgmap v562: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-14T04:04:40.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:40.727 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:40.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:40.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:04:40.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:40.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:40.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:41.005 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:41.035 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:41.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:41.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:41.185 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:41.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:41.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:41.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:41.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:41.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:41.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:41 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1732203803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:41.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:41 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1732203803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:41.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:41.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:41.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 121864 121848 92.8512 59 0.01693 0.0107652 2024-09-14T04:04:41.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:41.603 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:41.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:41 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1732203803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:41.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:41.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:41.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:41.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:41.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:04:41.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:04:41.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:04:41.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:04:41.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:04:41.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:04:41.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:04:41.751 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:04:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:04:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:04:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:04:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:04:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:04:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:04:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:04:41.752 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:04:41.752 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:04:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:04:41] "GET /metrics HTTP/1.1" 200 36438 "" "Prometheus/2.51.0" 2024-09-14T04:04:41.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:41.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:41.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:41.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:42.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:42.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:42.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:42.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:42.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:42.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:42.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:42.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:42.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:42.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:42.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:42 smithi114 ceph-mon[103029]: pgmap v563: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s 2024-09-14T04:04:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:42 smithi114 ceph-mon[106969]: pgmap v563: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s 2024-09-14T04:04:42.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:42.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:42.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:42.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:42.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:42.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:42.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 122864 122848 92.4855 62.5 0.00884419 0.0108079 2024-09-14T04:04:42.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:42 smithi160 ceph-mon[88407]: pgmap v563: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s 2024-09-14T04:04:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:42.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:42.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:42.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:42.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:42.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:42.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:42.959 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:42.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:43.006 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:04:43.006 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:04:43.006 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:04:43.006 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:04:43.006 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:04:43.006 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:04:43.006 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:04:43.006 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:04:43.007 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:04:43.007 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:04:43.007 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:04:43.007 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:04:43.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:43.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:43.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:43.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:43.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:43.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:43.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:43.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:43.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:43.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:43.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:43.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:43.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:43.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 123781 123765 92.0668 57.3125 0.0246845 0.010857 2024-09-14T04:04:43.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:43.632 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:43.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:43.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:43.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:43.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:43.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:43.912 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:43.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:44.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:44.051 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:44.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:44.153 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:44.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:44.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:44.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:44.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:44.320 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:44.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:44.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:44.446 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:44.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:44.547 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:44.561 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 124710 124694 91.6667 58.0625 0.0204914 0.0109043 2024-09-14T04:04:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:44.589 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:44.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:44.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:44 smithi114 ceph-mon[106969]: pgmap v564: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-14T04:04:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:44 smithi114 ceph-mon[103029]: pgmap v564: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-14T04:04:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:44 smithi160 ceph-mon[88407]: pgmap v564: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-14T04:04:44.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:44.711 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:44.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:44.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:44.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:44.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:44.941 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:44.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:44.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:45.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:45.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:45.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:45.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:45.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:45.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:45.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:45.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:45.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:45.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:45.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:45.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:45.562 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 125671 125655 91.2992 60.0625 0.00908656 0.0109484 2024-09-14T04:04:45.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:45.674 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:45.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:45.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:45.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:45.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:45.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:45.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:45.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:45.996 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:04:45.997 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:04:46.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:46.078 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:46.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:46.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:46.220 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:46.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:46.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:46.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:46.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:46.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:46.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:46.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:46.562 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 126661 126645 90.961 61.875 0.00774182 0.0109887 2024-09-14T04:04:46.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:46.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:46 smithi114 ceph-mon[103029]: pgmap v565: 89 pgs: 89 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 62 MiB/s wr, 986 op/s 2024-09-14T04:04:46.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:46 smithi114 ceph-mon[106969]: pgmap v565: 89 pgs: 89 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 62 MiB/s wr, 986 op/s 2024-09-14T04:04:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:46.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:46.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:46 smithi160 ceph-mon[88407]: pgmap v565: 89 pgs: 89 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 62 MiB/s wr, 986 op/s 2024-09-14T04:04:46.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-14T04:04:46.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:46.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:46.798 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:46.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:46.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:46.945 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:46.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:46.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:47.047 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:47.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:47.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:47.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:47.217 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:47.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:47.237 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:04:47.237 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:04:47.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-14T04:04:47.292 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:47.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:47.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:47.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:47.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:47.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:47.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:47.562 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 127591 127575 90.5879 58.125 0.0149804 0.0110339 2024-09-14T04:04:47.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:47.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:47.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:47.619 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:47.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:04:47.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:47.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:47.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:47.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:47.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:47.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:47.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:48.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:48.035 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:48.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:48.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:48.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:48.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:48.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:48.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:48 smithi114 ceph-mon[103029]: pgmap v566: 89 pgs: 89 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 62 MiB/s wr, 986 op/s 2024-09-14T04:04:48.312 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:48.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:48.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:48.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:48.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:48.544 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:48 smithi114 ceph-mon[106969]: pgmap v566: 89 pgs: 89 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 62 MiB/s wr, 986 op/s 2024-09-14T04:04:48.562 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 128501 128485 90.209 56.875 0.0158426 0.0110808 2024-09-14T04:04:48.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:48.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:48.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:48 smithi160 ceph-mon[88407]: pgmap v566: 89 pgs: 89 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 62 MiB/s wr, 986 op/s 2024-09-14T04:04:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:48.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:48.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:48.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:48.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:48.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:48.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:48.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:48.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:49.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:49.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:49.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:49.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:49.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:49.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:49.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:49.358 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:49.362 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:49.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:49.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:49.509 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:49.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:49.562 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0069 2024-09-14T04:04:49.562 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 129458 2024-09-14T04:04:49.562 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 89.8945 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 39.1013 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 236.25 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 51.375 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1438 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 625.621 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3780 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 822 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0111201 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0110349 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 0.336723 2024-09-14T04:04:49.563 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00111329 2024-09-14T04:04:49.569 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-14T04:04:49.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:49.647 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:49.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:49.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:49.788 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:49.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:49.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:49.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:49.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:50.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:50.057 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:50.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:50.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:50.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:50.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:50.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:50.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:50.252 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:50.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:50.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:50.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:50.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:50.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:50.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:50.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:50 smithi114 ceph-mon[106969]: pgmap v567: 89 pgs: 89 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s 2024-09-14T04:04:50.518 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:50 smithi114 ceph-mon[103029]: pgmap v567: 89 pgs: 89 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s 2024-09-14T04:04:50.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:50.599 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:50.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:50 smithi160 ceph-mon[88407]: pgmap v567: 89 pgs: 89 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s 2024-09-14T04:04:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:04:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:04:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:04:50.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:50.733 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:50.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:50.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:50.873 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:50.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:50.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:51.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:51.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:51.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:51.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:51.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:51.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:51.268 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:51.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:51.310 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:04:51.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:51.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:51.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:51.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:51.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:51.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:51.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:51.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:51 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/146224611' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:04:51.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:51 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:04:51.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:04:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:04:51] "GET /metrics HTTP/1.1" 200 36438 "" "Prometheus/2.51.0" 2024-09-14T04:04:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/146224611' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:04:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:51 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:04:51.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:51 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/146224611' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:04:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:51 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:04:51.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:51.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:51.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:51.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:51.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:51.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:51.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:51.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:51.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:51.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:52.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:52.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:52.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:52.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:52.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:52.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:52.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:52.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:52.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:52 smithi114 ceph-mon[106969]: pgmap v568: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T04:04:52.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:52 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:04:52.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:52 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:04:52.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:52 smithi114 ceph-mon[106969]: osdmap e163: 8 total, 8 up, 8 in 2024-09-14T04:04:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:52 smithi114 ceph-mon[103029]: pgmap v568: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T04:04:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:52 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:04:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:52 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:04:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:52 smithi114 ceph-mon[103029]: osdmap e163: 8 total, 8 up, 8 in 2024-09-14T04:04:52.379 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:52.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:52.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:52.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:52.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:52.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:52.653 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:52.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:52 smithi160 ceph-mon[88407]: pgmap v568: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T04:04:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:52 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:04:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:52 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:04:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:52 smithi160 ceph-mon[88407]: osdmap e163: 8 total, 8 up, 8 in 2024-09-14T04:04:52.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:52.790 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:52.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:52.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:52.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:52.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:52.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:53.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:53.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:53.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:53.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:53.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:53.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:53.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:53.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:53.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:53.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:53.364 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:53.364 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:53.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:53.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:53.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:53.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:53.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:53.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:04:53.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:53.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:53.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:53.773 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:04:53.773 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:04:53.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:53.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:53.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:53.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:53.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:54.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:54.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:54.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:54.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:54.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:54.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:54.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:54.305 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:54.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:54.334 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:54 smithi114 ceph-mon[106969]: pgmap v570: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 57 MiB/s wr, 963 op/s 2024-09-14T04:04:54.334 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:54 smithi114 ceph-mon[103029]: pgmap v570: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 57 MiB/s wr, 963 op/s 2024-09-14T04:04:54.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:54.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:54.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:54.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:54.580 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:54.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:54 smithi160 ceph-mon[88407]: pgmap v570: 89 pgs: 89 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 57 MiB/s wr, 963 op/s 2024-09-14T04:04:54.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:54.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:04:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:54.717 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:54.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:54.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:54.832 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:04:54.832 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:04:54.833 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:04:54.833 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:04:54.858 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:04:54.859 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:04:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:54.869 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:54.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:54.895 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:04:54.895 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:04:54.950 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:04:54.950 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:04:54.950 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:04:54.950 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:04:54.951 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:04:54.951 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:04:54.951 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:04:54.951 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:04:54.951 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:04:54.951 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:04:54.951 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:04:54.951 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:04:54.951 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:04:54.952 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:04:54.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:54.997 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:04:54.998 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:04:55.012 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:55.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:55.036 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:04:55.037 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:04:55.075 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:04:55.075 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:04:55.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:55.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:55.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:55.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:55.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:55.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:55.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:55.420 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:55.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:55.507 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:55 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3853058446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:55.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:55 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3853058446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:55.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:55.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:55.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:55.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:55 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3853058446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:04:55.701 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:55.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:55.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:04:55.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:55.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:04:55.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:55.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-14T04:04:55.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:55.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:56.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:56.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:56.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:56.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:56.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:56.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:56.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:56.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:56.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:56.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:56.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:56.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:56 smithi114 ceph-mon[103029]: pgmap v571: 89 pgs: 89 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 972 op/s 2024-09-14T04:04:56.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:56 smithi114 ceph-mon[106969]: pgmap v571: 89 pgs: 89 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 972 op/s 2024-09-14T04:04:56.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:56.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:56.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:56.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:56.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:56 smithi160 ceph-mon[88407]: pgmap v571: 89 pgs: 89 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 972 op/s 2024-09-14T04:04:56.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:56.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:56.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:04:56.827 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:04:56.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-14T04:04:56.866 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:56.882 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:56.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:56.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:56.968 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:56.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:57.070 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:57.127 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:57.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:57.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:57.281 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:57.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:57.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:57.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:57.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:57.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:57.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:57.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:57.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:57.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:57.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:57.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:57.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:57.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:57.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:57.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:57.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:57.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:57.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:58.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:58.115 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:58.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:58.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:58.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:58.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:58.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:04:58.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:58.359 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:04:58 smithi114 ceph-mon[106969]: pgmap v572: 89 pgs: 89 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 972 op/s 2024-09-14T04:04:58.360 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:04:58 smithi114 ceph-mon[103029]: pgmap v572: 89 pgs: 89 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 972 op/s 2024-09-14T04:04:58.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:58.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:58.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:58.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:58.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:58.643 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:04:58 smithi160 ceph-mon[88407]: pgmap v572: 89 pgs: 89 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 972 op/s 2024-09-14T04:04:58.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:58.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:58.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:58.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:58.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:58.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:04:58.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:04:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:58.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:58.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:04:59.054 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:59.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:59.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:59.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:04:59.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:59.230 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:59.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:04:59.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:04:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:59.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:59.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:04:59.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:04:59.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:59.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:04:59.506 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:59.541 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:04:59.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:04:59.644 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:59.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:04:59.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:04:59.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:04:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:59.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:04:59.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:04:59.917 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:04:59.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:04:59.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:00.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:00.061 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:00.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:00.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:00.196 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:00.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:00.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:00.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:00.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:00 smithi114 ceph-mon[103029]: pgmap v573: 89 pgs: 89 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 30 MiB/s wr, 973 op/s 2024-09-14T04:05:00.420 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:00 smithi114 ceph-mon[106969]: pgmap v573: 89 pgs: 89 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 30 MiB/s wr, 973 op/s 2024-09-14T04:05:00.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:00.467 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:00.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:00.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:00.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:00.602 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:00.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:00 smithi160 ceph-mon[88407]: pgmap v573: 89 pgs: 89 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 30 MiB/s wr, 973 op/s 2024-09-14T04:05:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:05:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:05:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:05:00.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:00.740 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:00.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:00.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:00.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:00.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:01.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:01.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:01.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:01.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:01.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:01.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:01.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:01.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:01.329 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:05:01.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:01.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:01.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:01.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:01.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:01.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:01.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:01 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2581563684' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:01.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:05:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:05:01] "GET /metrics HTTP/1.1" 200 36443 "" "Prometheus/2.51.0" 2024-09-14T04:05:01.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:01 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2581563684' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:01 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2581563684' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:01.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:01.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:01.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:01.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:01.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:01.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:01.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:01.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:01.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:02.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:02.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:02.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:02.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:02.243 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:02.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:02.344 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:02.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:02.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:02.477 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:02.506 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:02.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:02.608 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:02.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:02 smithi114 ceph-mon[106969]: pgmap v574: 89 pgs: 89 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 5.5 MiB/s wr, 977 op/s 2024-09-14T04:05:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:02 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:05:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:02 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:05:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:02 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2581563684' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:05:02.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:02 smithi114 ceph-mon[106969]: osdmap e164: 8 total, 8 up, 8 in 2024-09-14T04:05:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:02 smithi114 ceph-mon[103029]: pgmap v574: 89 pgs: 89 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 5.5 MiB/s wr, 977 op/s 2024-09-14T04:05:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:02 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:05:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:02 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:05:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:02 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2581563684' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:05:02.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:02 smithi114 ceph-mon[103029]: osdmap e164: 8 total, 8 up, 8 in 2024-09-14T04:05:02.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:02.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:02 smithi160 ceph-mon[88407]: pgmap v574: 89 pgs: 89 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 5.5 MiB/s wr, 977 op/s 2024-09-14T04:05:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:02 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:05:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:02 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:05:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:02 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2581563684' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:05:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:02 smithi160 ceph-mon[88407]: osdmap e164: 8 total, 8 up, 8 in 2024-09-14T04:05:02.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:02.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:02.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:02.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:02.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:02.909 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:02.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:02.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:03.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:03.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:03.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:03.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:03.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:03.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:03.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:03.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:03.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:03.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:03.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:03.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:03.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:03.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:03.557 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:03.604 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:03.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:03.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:03.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:03.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:03.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:03.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:03.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:04.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:04.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:04.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:04.171 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:04.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:04.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:04.306 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:04.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:04.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:04.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:04.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:04.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:04.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:04.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:04 smithi114 ceph-mon[103029]: pgmap v576: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 3.1 MiB/s wr, 974 op/s 2024-09-14T04:05:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:04 smithi114 ceph-mon[106969]: pgmap v576: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 3.1 MiB/s wr, 974 op/s 2024-09-14T04:05:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:04 smithi160 ceph-mon[88407]: pgmap v576: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 3.1 MiB/s wr, 974 op/s 2024-09-14T04:05:04.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:04.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:04.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:04.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:04.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:04.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:04.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:04.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:04.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:04.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:05.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:05.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:05.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:05.134 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:05:05.134 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:05:05.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:05:05.134 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96274 8.0 8.a661faf0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:44.788058+0000", 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 380.26628877600001, 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0146617819999999, 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.135 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96274 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788058+0000", 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788058+0000", 2024-09-14T04:05:05.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788060+0000", 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9319999999999999e-06 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788111+0000", 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0946999999999999e-05 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788120+0000", 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1630000000000002e-06 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788129+0000", 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6330000000000002e-06 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797008+0000", 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0088793980000000002 2024-09-14T04:05:05.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797016+0000", 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2539999999999992e-06 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.802707+0000", 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.005691025 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.802720+0000", 2024-09-14T04:05:05.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.243e-05 2024-09-14T04:05:05.140 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.140 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.140 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.140 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96276 8.0 8.f34e0690 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:05:05.140 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:44.788159+0000", 2024-09-14T04:05:05.140 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 380.26618802500002, 2024-09-14T04:05:05.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.016413411, 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96276 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788159+0000", 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788159+0000", 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788159+0000", 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.13e-07 2024-09-14T04:05:05.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788219+0000", 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9855999999999997e-05 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788233+0000", 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3543999999999999e-05 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788240+0000", 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0990000000000001e-06 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797039+0000", 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0087991800000000002 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797045+0000", 2024-09-14T04:05:05.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9730000000000002e-06 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.804559+0000", 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0075136769999999 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.804572+0000", 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3468999999999999e-05 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.145 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96277 8.0 8.99e4bea0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:44.788250+0000", 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 380.266096651, 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.016359896, 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.146 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96277 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788250+0000", 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788250+0000", 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788251+0000", 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3200000000000005e-07 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788298+0000", 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7247999999999999e-05 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788305+0000", 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2509999999999997e-06 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788311+0000", 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8799999999999996e-06 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797065+0000", 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0087544100000000007 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797071+0000", 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8100000000000003e-06 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.804602+0000", 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.007530391 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.804610+0000", 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3739999999999998e-06 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96280 8.0 8.d31b1580 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:44.788319+0000", 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 380.26602790099997, 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0163256110000001, 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96280 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.151 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788319+0000", 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788319+0000", 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788319+0000", 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2300000000000002e-07 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788364+0000", 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4215000000000003e-05 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788370+0000", 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9319999999999999e-06 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788384+0000", 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3932e-05 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797093+0000", 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0087083999999999998 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797099+0000", 2024-09-14T04:05:05.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8889999999999999e-06 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.804635+0000", 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0075360630000001 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.804644+0000", 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7569999999999999e-06 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.155 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96281 8.0 8.a15a8ff0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:44.788391+0000", 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 380.26595568599998, 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.016319403, 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.156 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96281 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788391+0000", 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788391+0000", 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788392+0000", 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5600000000000001e-07 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788438+0000", 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6755999999999998e-05 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788447+0000", 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2779999999999993e-06 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788452+0000", 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4140000000000002e-06 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797120+0000", 2024-09-14T04:05:05.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0086676270000000007 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797125+0000", 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.784e-06 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.804702+0000", 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0075765139999999 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.804710+0000", 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5739999999999996e-06 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96282 8.f 8.5385a84f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:44.788459+0000", 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 380.26588763299998, 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99694830899999998, 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.161 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96282 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788459+0000", 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788459+0000", 2024-09-14T04:05:05.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788460+0000", 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9900000000000001e-07 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788505+0000", 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5915999999999999e-05 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788512+0000", 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3450000000000002e-06 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.788517+0000", 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5990000000000001e-06 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.802207+0000", 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013689481 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.802216+0000", 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6209999999999993e-06 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.785397+0000", 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98318120600000003 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.785407+0000", 2024-09-14T04:05:05.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0742e-05 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96299 8.f 8.6a48344f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:44.797223+0000", 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 380.25712375299997, 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98822058800000001, 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.166 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96299 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797223+0000", 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797223+0000", 2024-09-14T04:05:05.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797225+0000", 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.782e-06 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797272+0000", 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7116000000000001e-05 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797282+0000", 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8320000000000001e-06 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.797290+0000", 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2789999999999998e-06 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.802237+0000", 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0049473249999999998 2024-09-14T04:05:05.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.802243+0000", 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6609999999999997e-06 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.785435+0000", 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98319237800000003 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.170 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.785444+0000", 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2150000000000001e-06 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96307 8.0 8.6193aeb0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:44.810884+0000", 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 380.24346247800003, 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99388354499999998, 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.171 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96307 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.810884+0000", 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.172 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.810884+0000", 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.810885+0000", 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7000000000000003e-07 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.810936+0000", 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0640999999999998e-05 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.173 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.810950+0000", 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3978000000000001e-05 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.810957+0000", 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2919999999999998e-06 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.811017+0000", 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0047999999999998e-05 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.811026+0000", 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6400000000000003e-06 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.804749+0000", 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99372327000000005 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.804768+0000", 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8706e-05 2024-09-14T04:05:05.175 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:96308 8.f 8.9e65602f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:58:44.812727+0000", 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 380.241619883, 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97274520499999995, 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.176 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 96308 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.812727+0000", 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.812727+0000", 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.177 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.812728+0000", 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3680000000000001e-06 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.812777+0000", 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9023000000000001e-05 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.812790+0000", 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3196999999999999e-05 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.812799+0000", 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3010000000000007e-06 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.812883+0000", 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4573999999999994e-05 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:44.812895+0000", 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2095e-05 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.785465+0000", 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97256920700000005 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:58:45.785472+0000", 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4399999999999999e-06 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:125244 8.0 8:0e768df1:::benchmark_data_smithi114_135745_object125243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e139)", 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:59:17.277075+0000", 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 347.777271488, 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.004360632, 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 125244 2024-09-14T04:05:05.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.277075+0000", 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.277075+0000", 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.277077+0000", 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6920000000000001e-06 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.277102+0000", 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4530999999999999e-05 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.277140+0000", 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8974999999999999e-05 2024-09-14T04:05:05.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.277149+0000", 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5399999999999996e-06 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.277417+0000", 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00026782800000000001 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-14T04:05:05.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.277436+0000", 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9218000000000001e-05 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.285958+0000", 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0085214690000000003 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.286010+0000", 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2126999999999997e-05 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.281350+0000", 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99534058599999997 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.281436+0000", 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5666000000000003e-05 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.186 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:125278 8.0 8:03233040:::benchmark_data_smithi114_135745_object125277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:59:17.305491+0000", 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 347.74885592499999, 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98693068699999997, 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 125278 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.305491+0000", 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.305491+0000", 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.305493+0000", 2024-09-14T04:05:05.188 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1299999999999999e-06 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.305519+0000", 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5874e-05 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.305554+0000", 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4814999999999998e-05 2024-09-14T04:05:05.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.305565+0000", 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1080000000000001e-05 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.305786+0000", 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000221566 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.305795+0000", 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0229999999999998e-06 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.281937+0000", 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97614147600000001 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.282047+0000", 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000109724 2024-09-14T04:05:05.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.282138+0000", 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1912999999999999e-05 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.292197+0000", 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010058047000000001 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.292262+0000", 2024-09-14T04:05:05.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5623999999999994e-05 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.292319+0000", 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6715000000000002e-05 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.292381+0000", 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1970000000000005e-05 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.193 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.292422+0000", 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0729999999999998e-05 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:125290 8.0 8:0fd2d41d:::benchmark_data_smithi114_135745_object125289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:59:17.318849+0000", 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 347.73549814500001, 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97735572100000001, 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.194 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 125290 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.318849+0000", 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.195 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.318849+0000", 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.318850+0000", 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7400000000000001e-06 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.318880+0000", 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9209000000000001e-05 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.318900+0000", 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9986000000000001e-05 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.318908+0000", 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8200000000000003e-06 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.318986+0000", 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7688000000000006e-05 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:17.318998+0000", 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2143e-05 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.285583+0000", 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96658474000000005 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.285708+0000", 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012473900000000001 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,5", 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.285860+0000", 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015191400000000001 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.292432+0000", 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0065726639999999998 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.292486+0000", 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3742000000000001e-05 2024-09-14T04:05:05.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.296126+0000", 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036396309999999999 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.296166+0000", 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0045e-05 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.200 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:18.296204+0000", 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8659999999999999e-05 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:151270 8.b 8:d2ac91ef:::benchmark_data_smithi114_135745_object25298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:59:46.675467+0000", 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 318.37887949999998, 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.002356952, 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:05.201 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 151270 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.675467+0000", 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.675467+0000", 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.675469+0000", 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6780000000000001e-06 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.675474+0000", 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.023e-06 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.203 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.675488+0000", 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3959e-05 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.675495+0000", 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1629999999999999e-06 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.675583+0000", 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8276999999999995e-05 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.204 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.675593+0000", 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5400000000000001e-06 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.669635+0000", 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99404226799999995 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.669751+0000", 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000116247 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.205 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.669836+0000", 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4511000000000005e-05 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.677549+0000", 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0077134170000000002 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.677590+0000", 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0197000000000002e-05 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.206 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.677711+0000", 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000121688 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.677775+0000", 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3517999999999997e-05 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.677824+0000", 2024-09-14T04:05:05.207 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9465999999999997e-05 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:151303 8.b 8:dba87101:::benchmark_data_smithi114_135745_object25331:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T03:59:46.710911+0000", 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 318.343435521, 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97200704900000001, 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.208 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 151303 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.710911+0000", 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.710911+0000", 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.710913+0000", 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.77e-06 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.710920+0000", 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6839999999999999e-06 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.210 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.710938+0000", 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8442999999999999e-05 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.710947+0000", 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2029999999999998e-06 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.711041+0000", 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4135000000000006e-05 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:46.711051+0000", 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4979999999999995e-06 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.672598+0000", 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.961547139 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.672714+0000", 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115643 2024-09-14T04:05:05.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.672836+0000", 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122481 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.677634+0000", 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0047977590000000004 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.677669+0000", 2024-09-14T04:05:05.213 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4774000000000003e-05 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.682846+0000", 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.005176791 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.682880+0000", 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4282000000000001e-05 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.214 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T03:59:47.682918+0000", 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8447e-05 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:233560 8.f 8:f61c9540:::benchmark_data_smithi114_135745_object107588:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e147)", 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:01:22.951893+0000", 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 222.10245418400001, 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.029155499, 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.215 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 233560 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951893+0000", 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.216 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951893+0000", 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951893+0000", 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2499999999999995e-07 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951895+0000", 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3250000000000002e-06 2024-09-14T04:05:05.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951902+0000", 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4899999999999997e-06 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951907+0000", 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8339999999999998e-06 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.952401+0000", 2024-09-14T04:05:05.218 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00049403900000000004 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.952407+0000", 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3570000000000003e-06 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.958806+0000", 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0063987410000000003 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.958821+0000", 2024-09-14T04:05:05.219 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5279000000000001e-05 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.980996+0000", 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.022174833 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.981048+0000", 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2076000000000001e-05 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.220 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:233573 8.f 8:fd13784c:::benchmark_data_smithi114_135745_object107601:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e147)", 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:01:22.951918+0000", 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 222.10242903299999, 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.029598571, 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 233573 2024-09-14T04:05:05.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951918+0000", 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951918+0000", 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.222 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951918+0000", 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9799999999999999e-07 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951920+0000", 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7540000000000001e-06 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951925+0000", 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1909999999999999e-06 2024-09-14T04:05:05.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951930+0000", 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4560000000000002e-06 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.952476+0000", 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000546734 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.952482+0000", 2024-09-14T04:05:05.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3149999999999999e-06 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.958980+0000", 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0064981090000000002 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.958987+0000", 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.798e-06 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.981451+0000", 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0224648030000001 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.981516+0000", 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5012999999999996e-05 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.226 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:233589 8.f 8:fbe96a1b:::benchmark_data_smithi114_135745_object107617:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:01:22.951984+0000", 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 222.102362597, 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0377738759999999, 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 233589 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951984+0000", 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951984+0000", 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951985+0000", 2024-09-14T04:05:05.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0299999999999999e-07 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951987+0000", 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5770000000000001e-06 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.951997+0000", 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4050000000000006e-06 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.952002+0000", 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9570000000000001e-06 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.952503+0000", 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000501239 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.952508+0000", 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6430000000000003e-06 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.959006+0000", 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0064987819999999998 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.959013+0000", 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2380000000000002e-06 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.981560+0000", 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0225472790000001 2024-09-14T04:05:05.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.981668+0000", 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010796799999999999 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.981791+0000", 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122991 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.989583+0000", 2024-09-14T04:05:05.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0077917790000000004 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.989651+0000", 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8408e-05 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.989665+0000", 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3747999999999999e-05 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.989716+0000", 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0998999999999997e-05 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.989758+0000", 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2360000000000001e-05 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:233608 8.f 8:f553ce5a:::benchmark_data_smithi114_135745_object107636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:01:22.979638+0000", 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 222.07470857199999, 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.021952693, 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 233608 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.235 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.979638+0000", 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.979638+0000", 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.979639+0000", 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1519999999999999e-06 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.979644+0000", 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4630000000000003e-06 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.979655+0000", 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0851e-05 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.979660+0000", 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2889999999999997e-06 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.979721+0000", 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1488999999999994e-05 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.979732+0000", 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.04e-05 2024-09-14T04:05:05.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.985926+0000", 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.006194037 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.986037+0000", 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111172 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.986145+0000", 2024-09-14T04:05:05.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000108077 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.993571+0000", 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0074263430000000002 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.993609+0000", 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7330999999999999e-05 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:05:05.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001489+0000", 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0078804989999999991 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001540+0000", 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0367000000000001e-05 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001591+0000", 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1223000000000001e-05 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:233617 8.f 8:f3c3e70a:::benchmark_data_smithi114_135745_object107645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:01:22.991283+0000", 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 222.06306381100001, 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.010590922, 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 233617 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.991283+0000", 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.991283+0000", 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.991284+0000", 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5009999999999999e-06 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.991289+0000", 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7650000000000001e-06 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.991302+0000", 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3063e-05 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.991308+0000", 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6459999999999998e-06 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.994103+0000", 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027946910000000002 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:22.994113+0000", 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0715e-05 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.989790+0000", 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99567646799999998 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.989908+0000", 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000117933 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.990025+0000", 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000117279 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001674+0000", 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011648680999999999 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001804+0000", 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013065099999999999 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001823+0000", 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9093000000000002e-05 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001848+0000", 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5028000000000002e-05 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001874+0000", 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5408000000000001e-05 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45086.0:233634 8.f 8:f6e40d75:::benchmark_data_smithi114_135745_object107662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:01:23.008294+0000", 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 222.04605253099999, 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.99375804899999998, 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:05:05.248 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45086", 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 233634 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.008294+0000", 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.008294+0000", 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.008296+0000", 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6419999999999999e-06 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:05:05.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.008301+0000", 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1139999999999997e-06 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.008316+0000", 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.501e-05 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.008323+0000", 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4789999999999999e-06 2024-09-14T04:05:05.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.009050+0000", 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072681899999999997 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.009061+0000", 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0841e-05 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.252 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.993633+0000", 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98457204600000003 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.993732+0000", 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8663e-05 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:23.993837+0000", 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010511800000000001 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001762+0000", 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0079248270000000006 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001886+0000", 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012404899999999999 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.001977+0000", 2024-09-14T04:05:05.254 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1475000000000001e-05 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.002021+0000", 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3801000000000003e-05 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:01:24.002052+0000", 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1164999999999997e-05 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.255 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:05:05.256 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:05:05.256 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:05:05.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:05.288 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:05.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:05.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:05.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:05.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:05.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:05.563 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:05.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:05.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:05.703 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:05.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:05.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:05.841 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:05.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:05.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:05.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:05.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:05.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:06.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:06.120 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:06.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:06.221 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:06.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:06.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:06.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:06.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:06.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:06.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:06.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:06.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:06.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:06.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:06 smithi114 ceph-mon[103029]: pgmap v577: 89 pgs: 89 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 953 op/s 2024-09-14T04:05:06.613 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:06 smithi114 ceph-mon[106969]: pgmap v577: 89 pgs: 89 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 953 op/s 2024-09-14T04:05:06.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:06 smithi160 ceph-mon[88407]: pgmap v577: 89 pgs: 89 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 953 op/s 2024-09-14T04:05:06.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:06.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:06.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:06.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:06.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:06.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:06.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:07.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:07.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:07.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:07.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:07.240 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:07.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:07.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:07.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:07.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:07.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:05:07.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:07.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:07.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:07.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:07.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:07.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:07.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:07.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:07.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:07.901 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:07.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:07.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:07.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:08.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:08.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:08.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:08.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:08.220 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:08.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:08.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:08.306 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:05:08.306 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:05:08.306 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:05:08.306 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:05:08.306 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:05:08.306 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:05:08.307 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:05:08.307 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:05:08.307 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:05:08.307 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:05:08.307 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:05:08.307 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:05:08.321 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:08.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:08.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:08.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:08 smithi114 ceph-mon[106969]: pgmap v578: 89 pgs: 89 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 953 op/s 2024-09-14T04:05:08.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:08.493 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:08.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:08 smithi114 ceph-mon[103029]: pgmap v578: 89 pgs: 89 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 953 op/s 2024-09-14T04:05:08.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:08.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:08.497 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:08.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:08.599 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:08.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:05:08.634 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:08.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:08 smithi160 ceph-mon[88407]: pgmap v578: 89 pgs: 89 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 953 op/s 2024-09-14T04:05:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:08.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:08.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:08.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:08.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:08.932 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:08.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:09.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:09.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:09.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:09.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:09.169 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:09.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:09.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:09.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:09.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:09.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:09.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:09.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:09.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:09.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:09.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:09.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:09.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:09.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:09.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:09.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:09.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:09.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:09.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:09.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:09.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:10.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:10.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:10.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:10.027 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:05:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:05:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-14T04:05:10.027 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:05:10.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:10.155 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:10.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:10.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:10.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:10.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:10.356 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:05:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:05:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:05:10.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:10.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:10.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:10.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:10.468 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:10 smithi114 ceph-mon[106969]: pgmap v579: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 951 op/s 2024-09-14T04:05:10.468 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:10 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2935207752' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:10.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:10 smithi114 ceph-mon[103029]: pgmap v579: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 951 op/s 2024-09-14T04:05:10.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:10 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2935207752' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:10.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:10.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:10.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:10.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:10 smithi160 ceph-mon[88407]: pgmap v579: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 951 op/s 2024-09-14T04:05:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:10 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2935207752' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:10.694 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:10.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:10.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-14T04:05:10.796 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:10.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:10.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:10.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:10.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:10.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:11.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:11.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:11.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:11.162 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:11.200 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:05:11.200 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:05:11.200 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:05:11.200 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:05:11.200 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:05:11.200 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:05:11.200 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:05:11.201 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:05:11.201 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:05:11.201 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:05:11.201 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:05:11.201 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:05:11.201 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:05:11.201 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:05:11.201 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:05:11.201 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:05:11.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:11.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:11.252 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:11.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:11.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:11.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:11.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:11.482 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:11.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:11.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:11.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:11.590 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:05:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:05:11] "GET /metrics HTTP/1.1" 200 36439 "" "Prometheus/2.51.0" 2024-09-14T04:05:11.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:11.685 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:11.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:11.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:11.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:11.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:11.956 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:11.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:12.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:12.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:12.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:12.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:12.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:12.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:12.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:12.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:12.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:05:12.419 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:05:12.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-14T04:05:12.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:12.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:12.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:12.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:12 smithi160 ceph-mon[88407]: pgmap v580: 89 pgs: 89 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 946 op/s 2024-09-14T04:05:12.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:05:12.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:12.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:12 smithi114 ceph-mon[103029]: pgmap v580: 89 pgs: 89 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 946 op/s 2024-09-14T04:05:12.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:05:12.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:12 smithi114 ceph-mon[106969]: pgmap v580: 89 pgs: 89 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 946 op/s 2024-09-14T04:05:12.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:05:12.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:12.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:12.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:12.805 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:12.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:12.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:12.943 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:12.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:13.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:13.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:13.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:13.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:13.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:13.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:13.240 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:05:13.240 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:05:13.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:13.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:13.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:13.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:13.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:13.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:13.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:13.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:13.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:13.627 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:13.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:13.725 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:13.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:13.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:13.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:13.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:13.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:13.887 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:13.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:13.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:05:13.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-14T04:05:13.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:14.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:14.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:14.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:14.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:14.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:14.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:14.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:14.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:14.476 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:14 smithi114 ceph-mon[106969]: pgmap v581: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-14T04:05:14.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:14 smithi114 ceph-mon[103029]: pgmap v581: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-14T04:05:14.502 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:14.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:14.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:14.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:14.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:14 smithi160 ceph-mon[88407]: pgmap v581: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-14T04:05:14.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:14.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:14.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:14.858 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:14.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:14.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:14.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:15.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:15.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:15.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:15.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:15.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:15.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:15.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:15.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:15.373 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:15.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:15.389 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:05:15.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:15.505 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:15.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:15.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:15.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3493033571' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:05:15.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3493033571' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:05:15.642 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:15.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:15.650 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3493033571' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:05:15.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:15.781 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:15.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:15.883 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:15.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:15.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:16.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:16.065 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:16.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:16.166 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:16.198 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:16.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:16.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:16.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:16.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:16.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:16.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:16.475 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:16.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:16.480 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:16 smithi160 ceph-mon[88407]: pgmap v582: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T04:05:16.480 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:16 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:16.480 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:16 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3493033571' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:05:16.480 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:16 smithi160 ceph-mon[88407]: osdmap e165: 8 total, 8 up, 8 in 2024-09-14T04:05:16.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:16.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:16.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:16.732 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:16.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:16.834 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:16.870 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:16.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:16 smithi114 ceph-mon[106969]: pgmap v582: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T04:05:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:16 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:16 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3493033571' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:05:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:16 smithi114 ceph-mon[106969]: osdmap e165: 8 total, 8 up, 8 in 2024-09-14T04:05:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:16 smithi114 ceph-mon[103029]: pgmap v582: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T04:05:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:16 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:16.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:16 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3493033571' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:05:16.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:16 smithi114 ceph-mon[103029]: osdmap e165: 8 total, 8 up, 8 in 2024-09-14T04:05:16.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:17.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:17.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:17.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:17.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:17.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:17.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:17.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:17.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:17.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:17.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:17.429 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:17.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:17.499 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:05:17.499 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:05:17.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:17.564 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:05:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:05:17.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:17.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:17.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:17.713 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:17.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:17.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:17.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:17.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:17.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:17.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:17.990 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:17.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:17.996 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:18.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:18.125 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:18.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:18.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:18.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:18.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:18.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:18.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:18.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:18.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:18.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:18.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:18.529 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:18.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:18.586 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:18 smithi114 ceph-mon[106969]: pgmap v584: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 919 op/s 2024-09-14T04:05:18.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:18 smithi114 ceph-mon[103029]: pgmap v584: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 919 op/s 2024-09-14T04:05:18.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:18.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-14T04:05:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:18 smithi160 ceph-mon[88407]: pgmap v584: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 919 op/s 2024-09-14T04:05:18.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:18.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:18.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:18.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:18.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:18.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:18.906 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:05:18.907 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:05:18.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-14T04:05:18.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:18.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:19.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:19.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:19.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:19.165 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:19.196 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:19.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:19.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:19.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:19.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:19.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:19.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:19.538 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:19.578 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:19.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:19.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:19.723 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:19.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:19.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:19.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:19.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:19.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:20.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:20.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:20.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:05:20.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:20.153 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:20.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:20.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:20.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:20.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:20.310 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:20.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:20.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:20.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:20.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:20.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:20.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:20 smithi160 ceph-mon[88407]: pgmap v585: 89 pgs: 89 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 919 op/s 2024-09-14T04:05:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:05:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:05:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:05:20.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:20.682 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:20 smithi114 ceph-mon[103029]: pgmap v585: 89 pgs: 89 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 919 op/s 2024-09-14T04:05:20.682 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:20 smithi114 ceph-mon[106969]: pgmap v585: 89 pgs: 89 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 919 op/s 2024-09-14T04:05:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:20.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:20.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:20.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:20.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:20.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:20.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:20.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:20.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:21.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:21.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:21.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:21.145 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:05:21.145 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:05:21.182 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:05:21.182 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:05:21.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:21.213 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:05:21.214 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:05:21.239 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:21.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:21.254 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:05:21.254 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:05:21.289 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:05:21.290 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:05:21.319 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:05:21.319 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:05:21.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:21.352 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:05:21.352 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:05:21.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:21.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:21.384 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:05:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:05:21.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:21.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:21.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:21.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:21.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:21 smithi114 ceph-mon[106969]: pgmap v586: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-14T04:05:21.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4081201451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:21.529 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:05:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:05:21] "GET /metrics HTTP/1.1" 200 36439 "" "Prometheus/2.51.0" 2024-09-14T04:05:21.530 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:21 smithi114 ceph-mon[103029]: pgmap v586: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-14T04:05:21.530 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4081201451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:21.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:21 smithi160 ceph-mon[88407]: pgmap v586: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-14T04:05:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4081201451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:21.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:21.661 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:21.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:21.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:21.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:21.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:21.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:21.932 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:21.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:22.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:22.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:22.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:22.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:22.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:22.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:22.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:22.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:22.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:22.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:22.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:22.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:22.540 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:05:22.540 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:05:22.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.540 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:160011 9.6 9:643a7c71:::benchmark_data_smithi114_153893_object30550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-14T04:05:22.540 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.408264+0000", 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.020236031000000002, 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020268666000000001, 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160011 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408264+0000", 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408264+0000", 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408263+0000", 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408268+0000", 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6659999999999996e-06 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408362+0000", 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3697999999999996e-05 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408367+0000", 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3009999999999997e-06 2024-09-14T04:05:22.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423897+0000", 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015529708 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.424248+0000", 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035095999999999999 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.424345+0000", 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7472000000000005e-05 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:160012 9.7 9:e796ffef:::benchmark_data_smithi114_153893_object30551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.408802+0000", 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.019697848, 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.019814426999999999, 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.545 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160012 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408802+0000", 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408802+0000", 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408801+0000", 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408807+0000", 2024-09-14T04:05:22.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5639999999999996e-06 2024-09-14T04:05:22.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408830+0000", 2024-09-14T04:05:22.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3674999999999999e-05 2024-09-14T04:05:22.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.408837+0000", 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3419999999999996e-06 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418033+0000", 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0091964109999999998 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418308+0000", 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000274861 2024-09-14T04:05:22.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418416+0000", 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108488 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423390+0000", 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049731640000000004 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.550 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.551 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.551 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:160016 9.7 9:e95225fa:::benchmark_data_smithi114_153893_object30555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-14T04:05:22.551 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.413819+0000", 2024-09-14T04:05:22.551 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.014680979, 2024-09-14T04:05:22.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014870990000000001, 2024-09-14T04:05:22.551 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160016 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.413819+0000", 2024-09-14T04:05:22.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.413819+0000", 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.413817+0000", 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:05:22.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.413823+0000", 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5910000000000003e-06 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.413836+0000", 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2649e-05 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.413841+0000", 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4720000000000003e-06 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423913+0000", 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010071495 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.424030+0000", 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117265 2024-09-14T04:05:22.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.424160+0000", 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039599999999999 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:160015 9.5 e165/152 9:a8c57f68:::benchmark_data_smithi114_153893_object30554:head v 165'10035, mlcod=165'10032)", 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.415071+0000", 2024-09-14T04:05:22.556 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.013429056999999999, 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013692437, 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.415071+0000", 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.415071+0000", 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.415070+0000", 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.415076+0000", 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3589999999999996e-06 2024-09-14T04:05:22.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.415088+0000", 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2446000000000001e-05 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.415093+0000", 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9710000000000003e-06 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.416044+0000", 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00095073800000000002 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.416085+0000", 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1337000000000001e-05 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:159997 9.7 9:e0065c07:::benchmark_data_smithi114_153893_object30536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.392267+0000", 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.036232825000000003, 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.036558913999999998, 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 159997 2024-09-14T04:05:22.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.392267+0000", 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.392267+0000", 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.392266+0000", 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.392270+0000", 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9589999999999997e-06 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.392277+0000", 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3509999999999996e-06 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.392281+0000", 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9480000000000001e-06 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.412510+0000", 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020228348 2024-09-14T04:05:22.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.412671+0000", 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161948 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.412805+0000", 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013319800000000001 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:22.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423377+0000", 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01057204 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423385+0000", 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3890000000000005e-06 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:160018 9.f 9:ff438955:::benchmark_data_smithi114_153893_object30557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-14T04:05:22.566 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.416844+0000", 2024-09-14T04:05:22.567 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.011656120000000001, 2024-09-14T04:05:22.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012066240000000001, 2024-09-14T04:05:22.567 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.567 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:05:22.567 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:22.567 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:05:22.567 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:05:22.567 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160018 2024-09-14T04:05:22.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.567 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.416844+0000", 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.416844+0000", 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.416842+0000", 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.416849+0000", 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2189999999999998e-06 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.416866+0000", 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6634e-05 2024-09-14T04:05:22.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.416872+0000", 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9649999999999996e-06 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423517+0000", 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0066453659999999998 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423619+0000", 2024-09-14T04:05:22.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010189 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423719+0000", 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9727000000000002e-05 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:160019 9.9 e165/152 9:921ab073:::benchmark_data_smithi114_153893_object30558:head v 165'9934, mlcod=165'9932)", 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.417976+0000", 2024-09-14T04:05:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.010523951, 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011005737999999999, 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.417976+0000", 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.417976+0000", 2024-09-14T04:05:22.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.417975+0000", 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.417981+0000", 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5659999999999997e-06 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.417989+0000", 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4519999999999997e-06 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.417994+0000", 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2479999999999998e-06 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418095+0000", 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101172 2024-09-14T04:05:22.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418124+0000", 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.968e-05 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:160017 9.5 e165/152 9:a7bf0dbb:::benchmark_data_smithi114_153893_object30556:head v 165'10036, mlcod=165'10032)", 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.418005+0000", 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.010494435999999999, 2024-09-14T04:05:22.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01103851, 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418005+0000", 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418005+0000", 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418005+0000", 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418008+0000", 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0019999999999998e-06 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418013+0000", 2024-09-14T04:05:22.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1590000000000001e-06 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.418016+0000", 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0520000000000002e-06 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426553+0000", 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0085368909999999996 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426615+0000", 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1808000000000002e-05 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.45572.0:160007 9.6 e165/152)", 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.423972+0000", 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0045278100000000002, 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.005123636, 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.579 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423972+0000", 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423972+0000", 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423971+0000", 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423978+0000", 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8680000000000001e-06 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423988+0000", 2024-09-14T04:05:22.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9320000000000008e-06 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423995+0000", 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5289999999999997e-06 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.45572.0:160012 9.7 e165/152)", 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.424013+0000", 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0044871010000000003, 2024-09-14T04:05:22.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0051232949999999999, 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.424013+0000", 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.424013+0000", 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.424012+0000", 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.424015+0000", 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.05e-06 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.424024+0000", 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6330000000000002e-06 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.424028+0000", 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3819999999999997e-06 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.585 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:160023 9.6 9.274fb976 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.426651+0000", 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0018488809999999999, 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002527634, 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160023 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426651+0000", 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426651+0000", 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426649+0000", 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:05:22.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426657+0000", 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5070000000000003e-06 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426669+0000", 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2122e-05 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426675+0000", 2024-09-14T04:05:22.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2519999999999996e-06 2024-09-14T04:05:22.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.589 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.589 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:160007 9.6 9:6849e69b:::benchmark_data_smithi114_153893_object30546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-14T04:05:22.589 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.401466+0000", 2024-09-14T04:05:22.589 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.02703347, 2024-09-14T04:05:22.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.027768452999999998, 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160007 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.401466+0000", 2024-09-14T04:05:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.401466+0000", 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.401465+0000", 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.401470+0000", 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2100000000000001e-06 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.401485+0000", 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.418e-05 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.401491+0000", 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8490000000000002e-06 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.417728+0000", 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016237075 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.417839+0000", 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111854 2024-09-14T04:05:22.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.417941+0000", 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010119300000000001 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423573+0000", 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0056323909999999996 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.423607+0000", 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3662999999999999e-05 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.45572.0:159997 9.7 e165/152)", 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.426689+0000", 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0018107450000000001, 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026162820000000002, 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.595 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426689+0000", 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426689+0000", 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426688+0000", 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426692+0000", 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1099999999999999e-06 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426698+0000", 2024-09-14T04:05:22.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2559999999999999e-06 2024-09-14T04:05:22.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426701+0000", 2024-09-14T04:05:22.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.455e-06 2024-09-14T04:05:22.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.598 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:160021 9.d e165/152 9:bededcf4:::benchmark_data_smithi114_153893_object30560:head v 165'10047, mlcod=165'10045)", 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.426706+0000", 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.001793701, 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002645746, 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426706+0000", 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426706+0000", 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426706+0000", 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967296 2024-09-14T04:05:22.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426711+0000", 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7430000000000002e-06 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426720+0000", 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2199999999999998e-06 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.426723+0000", 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2679999999999999e-06 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427892+0000", 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0011689770000000001 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427915+0000", 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.323e-05 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.602 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:160026 9.6 9.aa3f8106 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.427041+0000", 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0014590650000000001, 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023623509999999999, 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160026 2024-09-14T04:05:22.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427041+0000", 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427041+0000", 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427040+0000", 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427044+0000", 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.403e-06 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427059+0000", 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4524e-05 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427064+0000", 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2299999999999999e-06 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:160028 9.f 9.e2eb588f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.427608+0000", 2024-09-14T04:05:22.606 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00089160699999999999, 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001861442, 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160028 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427608+0000", 2024-09-14T04:05:22.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427608+0000", 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427607+0000", 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:22.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427612+0000", 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0579999999999997e-06 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427627+0000", 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4885000000000001e-05 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.427633+0000", 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6579999999999999e-06 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:160027 9.8 e165/152 9:191fee20:::benchmark_data_smithi114_153893_object30566:head v 165'9909, mlcod=165'9907)", 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:22.428030+0000", 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00046992699999999997, 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014882059999999999, 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:22.610 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.428030+0000", 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.428030+0000", 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:22.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.428029+0000", 2024-09-14T04:05:22.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:22.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:22.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.428038+0000", 2024-09-14T04:05:22.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5299999999999996e-06 2024-09-14T04:05:22.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.428051+0000", 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3264e-05 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.428056+0000", 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.366e-06 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.428938+0000", 2024-09-14T04:05:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00088184099999999996 2024-09-14T04:05:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:22.428960+0000", 2024-09-14T04:05:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1403000000000002e-05 2024-09-14T04:05:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:22.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:22.615 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:05:22.615 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 17 2024-09-14T04:05:22.615 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:05:22.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:22.674 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:22.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:22.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:22.812 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:22.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:22.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:22.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:22.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:22.958 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:23.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:23.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:23.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:23.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:23.206 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:23.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:23.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:23.348 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:23.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:23.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:23.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:23.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:23.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:23.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:23.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:23.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:23.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:23.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:23.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:23.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:23.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:23.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:23.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:23 smithi114 ceph-mon[106969]: pgmap v587: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 891 op/s 2024-09-14T04:05:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:23 smithi114 ceph-mon[103029]: pgmap v587: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 891 op/s 2024-09-14T04:05:23.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:23.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:23 smithi160 ceph-mon[88407]: pgmap v587: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 891 op/s 2024-09-14T04:05:23.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:24.028 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:24.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:24.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:24.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:24.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:24.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:24.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:24.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:24.295 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:24.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:24.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:24.433 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:24.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:24.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:05:24.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:24.562 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:24.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:24.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:24.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:24.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:24.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:24.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:24.814 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:24.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:24.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:24.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:24.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:25.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:25.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:25.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:25.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:25.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:25.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:25.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:25.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:25.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:25.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:25.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:25.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:25.596 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:25.632 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:25.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:25.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:25.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:25.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:25.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:25.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:25.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:25.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:25.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:26.037 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:26.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:26.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:25 smithi114 ceph-mon[106969]: pgmap v588: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-14T04:05:26.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:25 smithi114 ceph-mon[103029]: pgmap v588: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-14T04:05:26.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:25 smithi160 ceph-mon[88407]: pgmap v588: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-14T04:05:26.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:26.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:26.185 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:05:26.185 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:05:26.185 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:05:26.185 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:05:26.185 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:05:26.185 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:05:26.185 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:05:26.185 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:05:26.186 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:05:26.186 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:05:26.186 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:05:26.186 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:05:26.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:26.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:26.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:26.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:26.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:26.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:26.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:26.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:26.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:26.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:26.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:26.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:26.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:26.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:26.863 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:26.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:26.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:26.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-14T04:05:26.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:26.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:27.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:27.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:27.131 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:27.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:27.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:27.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:27.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:27.378 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:05:27.412 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:05:27.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-14T04:05:27.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:27.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:27.448 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:27.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:27.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:27.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:27.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:27.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:27.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:27.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:27.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:27.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:27.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:28.010 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:28.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:28.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:28.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:28.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:28.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:28.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:28.338 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:27 smithi114 ceph-mon[106969]: pgmap v589: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:05:28.338 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:27 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:28.338 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:27 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:28.339 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:27 smithi114 ceph-mon[103029]: pgmap v589: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:05:28.339 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:27 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:28.339 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:27 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:28.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:28.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:27 smithi160 ceph-mon[88407]: pgmap v589: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:05:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:27 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:27 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:28.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:28.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:28.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:28.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:28.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:28.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:28.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:28.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:28.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:28.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:28.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:28.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:28.905 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:28.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:29.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:29.013 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:05:29.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:29.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:29.120 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:28 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1604456317' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:29.121 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:28 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:29.136 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:28 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1604456317' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:29.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:28 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:29.137 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:28 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1604456317' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:29.137 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:28 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:29.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:29.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:29.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:29.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:29.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:29.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:29.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:29.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:29.453 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:05:29.453 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:05:29.453 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:05:29.454 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:05:29.454 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:05:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:29.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:29.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:29.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:29.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:29.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:29.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:29.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:29.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:29.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:29.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:29.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:29.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:30.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:30.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:30.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:30.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:30.108 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:29 smithi114 ceph-mon[106969]: pgmap v590: 89 pgs: 89 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 803 op/s 2024-09-14T04:05:30.108 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:29 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:30.108 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:29 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:05:30.108 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:29 smithi114 ceph-mon[106969]: osdmap e166: 8 total, 8 up, 8 in 2024-09-14T04:05:30.108 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:29 smithi114 ceph-mon[103029]: pgmap v590: 89 pgs: 89 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 803 op/s 2024-09-14T04:05:30.109 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:29 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:30.109 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:29 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:05:30.109 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:29 smithi114 ceph-mon[103029]: osdmap e166: 8 total, 8 up, 8 in 2024-09-14T04:05:30.130 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:29 smithi160 ceph-mon[88407]: pgmap v590: 89 pgs: 89 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 803 op/s 2024-09-14T04:05:30.130 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:29 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:30.130 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:29 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:05:30.130 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:29 smithi160 ceph-mon[88407]: osdmap e166: 8 total, 8 up, 8 in 2024-09-14T04:05:30.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:30.166 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-14T04:05:30.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:30.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:30.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:30.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:30.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:30.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:30.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:30.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:30.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:05:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:05:30.219Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:05:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:30.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:30.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:30.592 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:30.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:30.601 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:30.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:30.700 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:30.760 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:30.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:30.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:30.907 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:30.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:31.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:31.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:31.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:31.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:31.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:31.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:31.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:31.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:31.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:31.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:31.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:31.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:31.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:31.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:31.625 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:31.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:31.696 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:05:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:05:31] "GET /metrics HTTP/1.1" 200 36443 "" "Prometheus/2.51.0" 2024-09-14T04:05:31.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:31.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:31.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:31.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:31.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:31.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:31.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:32.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:32.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:32.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:32.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:05:32.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:32.188 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:32.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:32.235 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:31 smithi114 ceph-mon[103029]: pgmap v592: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-09-14T04:05:32.235 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:31 smithi114 ceph-mon[106969]: pgmap v592: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-09-14T04:05:32.282 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:31 smithi160 ceph-mon[88407]: pgmap v592: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 868 op/s 2024-09-14T04:05:32.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:32.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:32.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:32.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:32.453 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:32.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:32.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:32.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:32.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:32.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:32.711 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:05:32.711 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:05:32.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:32.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:32.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:32.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:32.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:32.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:33.007 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:33.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:33.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:33.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:33.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:33.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:33.294 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:33.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:33.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:33 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/975693060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:33.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:33 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/975693060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:33.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:33.411 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:33 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/975693060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:33.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:33.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:33.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:33.574 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:33.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:33.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-14T04:05:33.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:33.703 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:33.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:33.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:33.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:33.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:33.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:34.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:34.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:34.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:34.088 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:34.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:34.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:34.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:34.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:34.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:34.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:34.327 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:34 smithi114 ceph-mon[106969]: pgmap v593: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 864 op/s 2024-09-14T04:05:34.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:34 smithi114 ceph-mon[103029]: pgmap v593: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 864 op/s 2024-09-14T04:05:34.388 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:34 smithi160 ceph-mon[88407]: pgmap v593: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 864 op/s 2024-09-14T04:05:34.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:34.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:05:34.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:05:34.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:05:34.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:05:34.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:05:34.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:05:34.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:05:34.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:05:34.429 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:05:34.429 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:05:34.429 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:05:34.429 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:05:34.429 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:05:34.429 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:05:34.429 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:05:34.429 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:05:34.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:34.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:34.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:34.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:34.578 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:34.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:34.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:34.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:34.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:34.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:34.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:34.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:34.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:34.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:34.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:35.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:35.093 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:35.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:35.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:35.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:35.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:35.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:35.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:35.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:35.351 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:35.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:35.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:35.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:35.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:35.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:35.635 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:35.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:35.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:35.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:35.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:35.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:35.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:35.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:35.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:36.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:36.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:36.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:36.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:36.141 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:05:36.141 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:05:36.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:36.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:36.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:36.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:36.305 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:36 smithi114 ceph-mon[106969]: pgmap v594: 89 pgs: 89 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 836 op/s 2024-09-14T04:05:36.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:36 smithi114 ceph-mon[103029]: pgmap v594: 89 pgs: 89 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 836 op/s 2024-09-14T04:05:36.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:36.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:36.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:36 smithi160 ceph-mon[88407]: pgmap v594: 89 pgs: 89 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 836 op/s 2024-09-14T04:05:36.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:36.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:36.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:36.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:36.601 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:36.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:36.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:36.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:36.740 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:36.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:36.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:36.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-14T04:05:36.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:36.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:36.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:05:37.002 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:05:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:37.044 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:37.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:37.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:37.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:37.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:37.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:37.301 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:37.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:37.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:37.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:37.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:37.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:37.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:37.597 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:37.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:37.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:37.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:37.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:05:37.740 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:05:37.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-14T04:05:37.777 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:37.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:37.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:37.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:37.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:37.967 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[106969]: pgmap v595: 89 pgs: 89 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 836 op/s 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[103029]: pgmap v595: 89 pgs: 89 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 836 op/s 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:05:37.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:05:37.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:37.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:37.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:38.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:38.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:38.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:37 smithi160 ceph-mon[88407]: pgmap v595: 89 pgs: 89 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 836 op/s 2024-09-14T04:05:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:05:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:05:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:38.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:38.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:38.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:38.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:38.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:38.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:05:38.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:38.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:38.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:38.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:38.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:38.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:38.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:38.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:38.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:38.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:38.886 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:38.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:38.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:38.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:39.020 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:39.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:39.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:39.162 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:39.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:39.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:39.233 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:39.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:39.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:39.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:39.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:39.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:39.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:39.456 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:05:39.456 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:05:39.546 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:39.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:39.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:39.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:39.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:39.719 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:39.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:39.821 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:39.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:39.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:39.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:40.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:40.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:40.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:40.102 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:39 smithi114 ceph-mon[103029]: pgmap v596: 89 pgs: 89 active+clean; 5.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 831 op/s 2024-09-14T04:05:40.103 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:39 smithi114 ceph-mon[106969]: pgmap v596: 89 pgs: 89 active+clean; 5.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 831 op/s 2024-09-14T04:05:40.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:40.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:40.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:39 smithi160 ceph-mon[88407]: pgmap v596: 89 pgs: 89 active+clean; 5.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 831 op/s 2024-09-14T04:05:40.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-14T04:05:40.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:40.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:40.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:40.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:40.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:40.424 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:40.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:40.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:40.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:40.591 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:40.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:40.659 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:05:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:05:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:05:40.669 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:40.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:40.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:40.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:40.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:40.858 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:40.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:40.926 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:05:40.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:40.990 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:40.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:40.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:41.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:41.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:41.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3227635565' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:05:41.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3227635565' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:05:41.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3227635565' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:05:41.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:41.252 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:41.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:41.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:41.375 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:41.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:41.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:41.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:41.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:41.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:41.618 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:41.661 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:41.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:41.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:41.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:41.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:41.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:05:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:05:41] "GET /metrics HTTP/1.1" 200 36441 "" "Prometheus/2.51.0" 2024-09-14T04:05:41.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:41.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:41.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:42.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:42.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:42.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:42.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:42.214 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:42.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:42.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:41 smithi114 ceph-mon[103029]: pgmap v597: 89 pgs: 89 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-09-14T04:05:42.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:41 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:42.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:41 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3227635565' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:05:42.303 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:41 smithi114 ceph-mon[103029]: osdmap e167: 8 total, 8 up, 8 in 2024-09-14T04:05:42.303 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:41 smithi114 ceph-mon[106969]: pgmap v597: 89 pgs: 89 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-09-14T04:05:42.303 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:41 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:42.303 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:41 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3227635565' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:05:42.303 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:41 smithi114 ceph-mon[106969]: osdmap e167: 8 total, 8 up, 8 in 2024-09-14T04:05:42.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:42.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:42.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:42.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:41 smithi160 ceph-mon[88407]: pgmap v597: 89 pgs: 89 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-09-14T04:05:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:41 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:41 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3227635565' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:05:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:41 smithi160 ceph-mon[88407]: osdmap e167: 8 total, 8 up, 8 in 2024-09-14T04:05:42.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:42.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:42.503 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:42.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:42.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:42.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:42.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:42.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:42.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:42.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:42.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:42.906 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:42.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:43.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:43.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:43.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:43.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:43.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:43.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:43.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:43.303 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:43.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:43.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:43.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:43.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:43.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:43.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:43.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:43.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:43.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:43.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:43.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:43.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:43.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:43.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:43.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:43.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:44.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:43 smithi114 ceph-mon[103029]: pgmap v599: 89 pgs: 89 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 833 op/s 2024-09-14T04:05:44.045 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:43 smithi114 ceph-mon[106969]: pgmap v599: 89 pgs: 89 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 833 op/s 2024-09-14T04:05:44.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:44.128 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:44.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:44.215 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:05:44.215 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:05:44.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:44.251 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:05:44.251 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:05:44.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:44.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:44.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:44.276 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:05:44.276 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:05:44.306 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:05:44.306 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:05:44.337 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:05:44.338 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:05:44.354 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:43 smithi160 ceph-mon[88407]: pgmap v599: 89 pgs: 89 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 833 op/s 2024-09-14T04:05:44.360 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:05:44.361 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:05:44.361 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:05:44.361 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:05:44.361 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:05:44.361 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:05:44.361 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:05:44.362 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:05:44.362 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:05:44.362 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:05:44.362 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:05:44.362 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:05:44.368 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:05:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:05:44.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:44.398 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:05:44.398 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:05:44.408 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:44.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:44.427 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:05:44.427 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:05:44.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:44.546 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:44.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:44.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:44.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:44.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:44.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:44.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:05:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:44.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:44.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:44.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:44.999 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:44 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3985494015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:45.022 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:05:45.022 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:05:45.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:05:45.022 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:05:45.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.022 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.44156.0:184490 7.d 7:b1559fc1:::benchmark_data_smithi114_118301_object63625:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-14T04:05:45.022 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:56:12.793766+0000", 2024-09-14T04:05:45.022 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.22386947300004, 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0241563680000001, 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.44156", 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 184490 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.793766+0000", 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.793766+0000", 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.793765+0000", 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.793772+0000", 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6279999999999999e-06 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.793793+0000", 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.124e-05 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.793799+0000", 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7089999999999996e-06 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.793844+0000", 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4601999999999997e-05 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.793922+0000", 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7682000000000006e-05 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.793998+0000", 2024-09-14T04:05:45.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6266999999999993e-05 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.797448+0000", 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003450285 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.802715+0000", 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0052661940000000001 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.806939+0000", 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0042245850000000003 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.806948+0000", 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0680000000000003e-06 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.817872+0000", 2024-09-14T04:05:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0109234840000001 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.817923+0000", 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0980000000000003e-05 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.44156.0:184497 7.d 7:b77e189d:::benchmark_data_smithi114_118301_object63632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:56:12.810410+0000", 2024-09-14T04:05:45.029 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.20722575499997, 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.033187316, 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.44156", 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 184497 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.810410+0000", 2024-09-14T04:05:45.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.810410+0000", 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.810409+0000", 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.810413+0000", 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9990000000000002e-06 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.810424+0000", 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0574000000000001e-05 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.810429+0000", 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6759999999999998e-06 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.810469+0000", 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0179999999999998e-05 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.810478+0000", 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.85e-06 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.818169+0000", 2024-09-14T04:05:45.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.00769117 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.818237+0000", 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8097999999999998e-05 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.818306+0000", 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9282999999999994e-05 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.829510+0000", 2024-09-14T04:05:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011203924000000001 2024-09-14T04:05:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.835693+0000", 2024-09-14T04:05:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061828710000000004 2024-09-14T04:05:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843505+0000", 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0078115909999999997 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843560+0000", 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5770999999999998e-05 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843597+0000", 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7036999999999998e-05 2024-09-14T04:05:45.036 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.44156.0:184498 7.d 7:b3bdca11:::benchmark_data_smithi114_118301_object63633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:56:12.811255+0000", 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.20638062499995, 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0324790049999999, 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.44156", 2024-09-14T04:05:45.037 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 184498 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.811255+0000", 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.811255+0000", 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.811254+0000", 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.811258+0000", 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8510000000000001e-06 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.811268+0000", 2024-09-14T04:05:45.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4350000000000003e-06 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.811272+0000", 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.811301+0000", 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8929e-05 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.811306+0000", 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.685e-06 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.821540+0000", 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.010234439 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.821615+0000", 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5057999999999999e-05 2024-09-14T04:05:45.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.821680+0000", 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4815999999999997e-05 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.835648+0000", 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013968034000000001 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.835721+0000", 2024-09-14T04:05:45.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3348000000000003e-05 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843646+0000", 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0079245410000000002 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843696+0000", 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9666000000000002e-05 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843734+0000", 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8438e-05 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.44156.0:184518 7.d 7:b87e2b85:::benchmark_data_smithi114_118301_object63653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:56:12.846482+0000", 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.171153398, 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99739467599999998, 2024-09-14T04:05:45.044 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.44156", 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 184518 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.846482+0000", 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.846482+0000", 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.846482+0000", 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.846485+0000", 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6350000000000002e-06 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.846489+0000", 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6210000000000003e-06 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.846492+0000", 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.74e-06 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.846538+0000", 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6088999999999999e-05 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.846549+0000", 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1304000000000001e-05 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.824565+0000", 2024-09-14T04:05:45.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97801585199999996 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.824632+0000", 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6449e-05 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.824699+0000", 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7343999999999998e-05 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.835656+0000", 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010956634999999999 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.835755+0000", 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9609000000000001e-05 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843789+0000", 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080341639999999999 2024-09-14T04:05:45.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843842+0000", 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2612999999999999e-05 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843877+0000", 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5005000000000001e-05 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.051 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.053 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.053 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.44156.0:184554 7.d 7:bfccb052:::benchmark_data_smithi114_118301_object63689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:05:45.053 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:56:12.915433+0000", 2024-09-14T04:05:45.053 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.10220301599998, 2024-09-14T04:05:45.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.92857496500000003, 2024-09-14T04:05:45.053 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.053 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.053 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.44156", 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 184554 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.915433+0000", 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.915433+0000", 2024-09-14T04:05:45.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.915432+0000", 2024-09-14T04:05:45.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:45.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.915437+0000", 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8300000000000003e-06 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.915447+0000", 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0672e-05 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.915452+0000", 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2729999999999996e-06 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.915513+0000", 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1711000000000006e-05 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.915523+0000", 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6539999999999994e-06 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.827199+0000", 2024-09-14T04:05:45.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91167636900000004 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.827263+0000", 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3588000000000002e-05 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.827324+0000", 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0680999999999997e-05 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.835728+0000", 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0084045549999999993 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.842783+0000", 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0070550639999999998 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843930+0000", 2024-09-14T04:05:45.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0011465480000000001 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843973+0000", 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2781000000000003e-05 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.844008+0000", 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5077999999999999e-05 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.060 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.44156.0:184561 7.d 7:b36b0619:::benchmark_data_smithi114_118301_object63696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:56:12.931999+0000", 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.08563656299998, 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91212551200000003, 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.44156", 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:05:45.061 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 184561 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.931999+0000", 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.931999+0000", 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.931998+0000", 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.932003+0000", 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0860000000000001e-06 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.932014+0000", 2024-09-14T04:05:45.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0759e-05 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.932020+0000", 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6269999999999997e-06 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.932055+0000", 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5858999999999997e-05 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.932061+0000", 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6930000000000004e-06 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.829539+0000", 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.8974782 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.829599+0000", 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9383999999999999e-05 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.829661+0000", 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1864000000000003e-05 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.835735+0000", 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.006074042 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.842810+0000", 2024-09-14T04:05:45.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0070754219999999996 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.844053+0000", 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001243374 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.844091+0000", 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7898000000000002e-05 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.844125+0000", 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3330000000000001e-05 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.44156.0:184569 7.d 7:bf163c37:::benchmark_data_smithi114_118301_object63704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:56:12.955982+0000", 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.06165337799996, 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.900763545, 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.068 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.44156", 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/321433757", 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 184569 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.955982+0000", 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.955982+0000", 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.955981+0000", 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.955988+0000", 2024-09-14T04:05:45.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1890000000000001e-06 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.956001+0000", 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3142999999999999e-05 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.956008+0000", 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3369999999999996e-06 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.956063+0000", 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5291999999999998e-05 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:12.956070+0000", 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9469999999999997e-06 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.831962+0000", 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87589240700000004 2024-09-14T04:05:45.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.832063+0000", 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010055 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.832175+0000", 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000112236 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.835824+0000", 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0036485150000000002 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.843242+0000", 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0074186190000000004 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.856660+0000", 2024-09-14T04:05:45.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013417439999999999 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.856709+0000", 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9014000000000002e-05 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:56:13.856746+0000", 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7251e-05 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.075 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:145378 8.5 8:a9453ee3:::benchmark_data_smithi114_135745_object19406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:59:39.950294+0000", 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 365.06734172099999, 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.197874669, 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:45.076 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 145378 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.950294+0000", 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.950294+0000", 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.950293+0000", 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.950299+0000", 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4250000000000003e-06 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.950311+0000", 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1751e-05 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.950317+0000", 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9120000000000003e-06 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.950365+0000", 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8016999999999998e-05 2024-09-14T04:05:45.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.950524+0000", 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000159269 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 4,7", 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.950622+0000", 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8103000000000006e-05 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.955209+0000", 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045868289999999997 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.955353+0000", 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000143821 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.148079+0000", 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19272672699999999 2024-09-14T04:05:45.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.148130+0000", 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0685e-05 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.148169+0000", 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8528000000000001e-05 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45086.0:145367 8.6 e140/129 8:63b1182a:::benchmark_data_smithi114_135745_object19395:head v 140'9251, mlcod=140'9246)", 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:59:39.955534+0000", 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 365.06210216099998, 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20970409600000001, 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.955534+0000", 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.955534+0000", 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.955533+0000", 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.955542+0000", 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7080000000000003e-06 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.955551+0000", 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.935e-06 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.955555+0000", 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2529999999999998e-06 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.955597+0000", 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1665999999999997e-05 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.955614+0000", 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7516e-05 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.165197+0000", 2024-09-14T04:05:45.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20958238000000001 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.165238+0000", 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1183999999999997e-05 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45086.0:145372 8.6 e140/129 8:6d077100:::benchmark_data_smithi114_135745_object19400:head v 140'9252, mlcod=140'9249)", 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:59:39.961583+0000", 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 365.05605306500001, 2024-09-14T04:05:45.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20367992500000001, 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.961583+0000", 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.961583+0000", 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.961582+0000", 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:39.961590+0000", 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8590000000000005e-06 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.148034+0000", 2024-09-14T04:05:45.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18644419700000001 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.148043+0000", 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2879999999999998e-06 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.152898+0000", 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048543609999999997 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.152917+0000", 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9437999999999999e-05 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.165240+0000", 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012323324 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:59:40.165263+0000", 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2345999999999999e-05 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.091 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:163396 8.9 8:919dbb24:::benchmark_data_smithi114_135745_object37424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:00:01.275553+0000", 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 343.74208295599999, 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27013436899999999, 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:45.092 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 163396 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.275553+0000", 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.275553+0000", 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.275551+0000", 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.275558+0000", 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6100000000000002e-06 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.275571+0000", 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2587e-05 2024-09-14T04:05:45.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.275577+0000", 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1020000000000002e-06 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.275628+0000", 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1637e-05 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.275740+0000", 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111681 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.275858+0000", 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117513 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.288178+0000", 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01232042 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.293108+0000", 2024-09-14T04:05:45.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049303460000000004 2024-09-14T04:05:45.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.545508+0000", 2024-09-14T04:05:45.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.25239925699999999 2024-09-14T04:05:45.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:45.098 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:45.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:45.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.545642+0000", 2024-09-14T04:05:45.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013476800000000001 2024-09-14T04:05:45.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.545687+0000", 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4867000000000003e-05 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:163400 8.9 8:9b48072d:::benchmark_data_smithi114_135745_object37428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:00:01.282780+0000", 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 343.73485625000001, 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27722714100000001, 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.100 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 163400 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282780+0000", 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282780+0000", 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282778+0000", 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282784+0000", 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3149999999999999e-06 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282795+0000", 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1335000000000001e-05 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282801+0000", 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5559999999999998e-06 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282973+0000", 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017238 2024-09-14T04:05:45.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.283184+0000", 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021069700000000001 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.283324+0000", 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014027000000000001 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.293235+0000", 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0099108249999999998 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.293277+0000", 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1680000000000001e-05 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.559808+0000", 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.26653155000000001 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.559962+0000", 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015346899999999999 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.560007+0000", 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5080999999999997e-05 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:163401 8.9 8:908216ab:::benchmark_data_smithi114_135745_object37429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:00:01.282823+0000", 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 343.73481266200002, 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27736959500000002, 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 163401 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282823+0000", 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282823+0000", 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282823+0000", 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:45.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282826+0000", 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1690000000000001e-06 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282832+0000", 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3500000000000002e-06 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.109 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.282836+0000", 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9040000000000002e-06 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.288363+0000", 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0055270939999999998 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.288485+0000", 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012211899999999999 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.288603+0000", 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117348 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.293306+0000", 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0047029230000000003 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.294554+0000", 2024-09-14T04:05:45.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001248517 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.560042+0000", 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.26548748599999999 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.560159+0000", 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117481 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.560193+0000", 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3633000000000001e-05 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:163427 8.9 8:98f5aea9:::benchmark_data_smithi114_135745_object37455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:00:01.323969+0000", 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 343.69366633499999, 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.23639932899999999, 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.113 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 163427 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.323969+0000", 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.114 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.323969+0000", 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.323968+0000", 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.323974+0000", 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6180000000000003e-06 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.323986+0000", 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1783000000000001e-05 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.323992+0000", 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0780000000000001e-06 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.324033+0000", 2024-09-14T04:05:45.116 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1020999999999999e-05 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.324119+0000", 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5907000000000003e-05 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,5", 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.324195+0000", 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5947999999999999e-05 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:05:45.117 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.327732+0000", 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035373940000000001 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.329150+0000", 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014179660000000001 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.560246+0000", 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.23109606699999999 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.560336+0000", 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0116000000000003e-05 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:00:01.560369+0000", 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2537999999999999e-05 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.119 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:116736 9.1 e161/152 9:838b94d3:::benchmark_data_smithi114_153893_object116735:head v 161'7179, mlcod=161'7177)", 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:04:36.375372+0000", 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 68.642263823999997, 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025676483, 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:36.375372+0000", 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:36.375372+0000", 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:36.375371+0000", 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:36.375409+0000", 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8108000000000002e-05 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:36.375426+0000", 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6843000000000001e-05 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:36.375432+0000", 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0229999999999998e-06 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:36.375569+0000", 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000137374 2024-09-14T04:05:45.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:36.375621+0000", 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1363000000000003e-05 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:36.400807+0000", 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025186427000000001 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:36.401048+0000", 2024-09-14T04:05:45.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000241399 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:136543 9.b e163/152 9:d127e0c0:::benchmark_data_smithi114_153893_object7082:head v 163'8482, mlcod=163'8478)", 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:04:56.733611+0000", 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 48.284024983000002, 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.022884469000000001, 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:56.733611+0000", 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:56.733611+0000", 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:56.733610+0000", 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:45.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:56.733615+0000", 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8350000000000003e-06 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:56.733625+0000", 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0231000000000001e-05 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:56.733629+0000", 2024-09-14T04:05:45.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7110000000000001e-06 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:56.739766+0000", 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061372019999999996 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:56.739789+0000", 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2881e-05 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:56.756417+0000", 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016627877999999999 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:56.756495+0000", 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8230000000000001e-05 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.128 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:145561 9.b e164/152 9:de44e6ca:::benchmark_data_smithi114_153893_object16100:head v 164'9041, mlcod=164'9039)", 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:06.221024+0000", 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 38.796611724000002, 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021830288999999999, 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:06.221024+0000", 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:06.221024+0000", 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:06.221023+0000", 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:06.221031+0000", 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1640000000000001e-06 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:06.221044+0000", 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2879e-05 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:06.221051+0000", 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5740000000000001e-06 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:06.221205+0000", 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015443600000000001 2024-09-14T04:05:45.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:06.221258+0000", 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2719000000000001e-05 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:06.242822+0000", 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021563621000000002 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:06.242854+0000", 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2753000000000003e-05 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:148058 9.b e164/152 9:deb64890:::benchmark_data_smithi114_153893_object18597:head v 164'9209, mlcod=164'9206)", 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:08.956269+0000", 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 36.061366724000003, 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.023914872, 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.956269+0000", 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.956269+0000", 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.956268+0000", 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.956277+0000", 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1039999999999996e-06 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.956292+0000", 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4301000000000001e-05 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.956297+0000", 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3870000000000003e-06 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.958390+0000", 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020929870000000001 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.958439+0000", 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9354999999999998e-05 2024-09-14T04:05:45.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.980143+0000", 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021703574999999999 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.980184+0000", 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1077e-05 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:148062 9.b e164/152 9:db3849ff:::benchmark_data_smithi114_153893_object18601:head v 164'9210, mlcod=164'9206)", 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:08.959586+0000", 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 36.058050010999999, 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025268215, 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.959586+0000", 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.138 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.959586+0000", 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.959585+0000", 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.959594+0000", 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5340000000000005e-06 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.959609+0000", 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4430000000000001e-05 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.959615+0000", 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2559999999999999e-06 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.963514+0000", 2024-09-14T04:05:45.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0038990380000000001 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.963558+0000", 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3619999999999999e-05 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.984749+0000", 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021191379 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:08.984854+0000", 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104762 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:159194 9.b e165/152 9:d090eae8:::benchmark_data_smithi114_153893_object29733:head v 165'9898, mlcod=165'9894)", 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:21.494332+0000", 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 23.523303778999999, 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021904057000000001, 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:21.494332+0000", 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:21.494332+0000", 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:45.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:21.494330+0000", 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:21.494339+0000", 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7250000000000003e-06 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:21.494356+0000", 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7056999999999999e-05 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:21.494363+0000", 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1879999999999996e-06 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:21.495408+0000", 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001044583 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:21.495510+0000", 2024-09-14T04:05:45.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101517 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:21.516060+0000", 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020550451000000001 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:21.516236+0000", 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176057 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.146 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.147 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:45.147 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:45.147 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:05:45.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:45.210 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:44 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3985494015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:45.210 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:44 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3985494015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:45.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:45.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:45.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:05:45.250 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:05:45.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:45.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:45.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:45.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:45.505 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:45.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:45.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:45.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:45.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:45.779 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:45.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:45.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:45.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:45.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:45.931 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:45.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:45.945 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-14T04:05:46.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:46.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:46.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:45 smithi160 ceph-mon[88407]: pgmap v600: 89 pgs: 89 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 845 op/s 2024-09-14T04:05:46.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:46.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:46.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:46.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:45 smithi114 ceph-mon[103029]: pgmap v600: 89 pgs: 89 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 845 op/s 2024-09-14T04:05:46.244 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:45 smithi114 ceph-mon[106969]: pgmap v600: 89 pgs: 89 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 845 op/s 2024-09-14T04:05:46.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:46.336 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:46.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:46.437 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:46.438 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:46.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:46.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:46.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:46.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:46.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:46.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:46.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:46.768 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:46.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:46.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:46.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:46.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:46.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:47.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:47.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:47.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:47.141 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:47.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:47.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:47.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:47.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:47.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:47.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:47.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:47.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:47.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:47.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:47.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:47.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:47.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:05:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:47.737 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:47.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:47.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:47.884 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:47.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:47.986 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:48.021 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:48.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:48.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:47 smithi114 ceph-mon[103029]: pgmap v601: 89 pgs: 89 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 845 op/s 2024-09-14T04:05:48.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:47 smithi114 ceph-mon[106969]: pgmap v601: 89 pgs: 89 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 845 op/s 2024-09-14T04:05:48.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:47 smithi160 ceph-mon[88407]: pgmap v601: 89 pgs: 89 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 845 op/s 2024-09-14T04:05:48.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:48.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:48.175 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:48.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:48.303 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:48.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:48.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:48.437 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:48.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:48.527 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:05:48.527 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:05:48.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.527 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:182203 9.5 e167/152 9:aa859d59:::benchmark_data_smithi114_153893_object52742:head v 167'11407, mlcod=167'11403)", 2024-09-14T04:05:48.527 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:48.505975+0000", 2024-09-14T04:05:48.527 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.017859422999999999, 2024-09-14T04:05:48.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017896787000000001, 2024-09-14T04:05:48.527 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.505975+0000", 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.505975+0000", 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.505974+0000", 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.505980+0000", 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5620000000000003e-06 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.505987+0000", 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8010000000000004e-06 2024-09-14T04:05:48.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.505992+0000", 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.685e-06 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.506108+0000", 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011629999999999999 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:48.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.506125+0000", 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6668999999999999e-05 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.523794+0000", 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017668825999999999 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:182210 9.a e167/152 9:587b4021:::benchmark_data_smithi114_153893_object52749:head v 167'11441, mlcod=167'11439)", 2024-09-14T04:05:48.531 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:48.511068+0000", 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.012765682, 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012907470000000001, 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.511068+0000", 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:48.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.511068+0000", 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.511068+0000", 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.511072+0000", 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5009999999999998e-06 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.511079+0000", 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0190000000000001e-06 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.511084+0000", 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1030000000000004e-06 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:48.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.512531+0000", 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014470380000000001 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.512549+0000", 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7932e-05 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:182212 9.c e167/152 9:301b7e87:::benchmark_data_smithi114_153893_object52751:head v 167'11594, mlcod=167'11592)", 2024-09-14T04:05:48.535 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:48.514728+0000", 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0091059750000000005, 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0093084870000000007, 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.514728+0000", 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:48.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.514728+0000", 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.514727+0000", 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.514734+0000", 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2829999999999998e-06 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.514750+0000", 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6231000000000001e-05 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.514755+0000", 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4990000000000002e-06 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:48.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.515930+0000", 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0011745620000000001 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.515948+0000", 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7598999999999999e-05 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:182218 9.5 e167/152 9:aaf0562f:::benchmark_data_smithi114_153893_object52757:head v 167'11408, mlcod=167'11405)", 2024-09-14T04:05:48.539 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:48.518166+0000", 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0056684639999999998, 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059367930000000001, 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.518166+0000", 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:48.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.518166+0000", 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.518165+0000", 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.518171+0000", 2024-09-14T04:05:48.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5709999999999997e-06 2024-09-14T04:05:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.518182+0000", 2024-09-14T04:05:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1574e-05 2024-09-14T04:05:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:48.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.518188+0000", 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8490000000000002e-06 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.518244+0000", 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6224999999999998e-05 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.518264+0000", 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9601000000000001e-05 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.543 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:182214 9.a e167/152 9:54d7524b:::benchmark_data_smithi114_153893_object52753:head v 167'11442, mlcod=167'11439)", 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:48.519873+0000", 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0039606160000000001, 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0042859439999999999, 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:48.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519873+0000", 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519873+0000", 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519873+0000", 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519880+0000", 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7530000000000004e-06 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519893+0000", 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3675e-05 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:48.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519899+0000", 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7230000000000001e-06 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.524023+0000", 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.004124178 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.524043+0000", 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9837999999999999e-05 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:48.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.45572.0:182217 9.1 e167/152 9:83d9b73d:::benchmark_data_smithi114_153893_object52756:head v 167'11229, mlcod=167'11226)", 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:05:48.519920+0000", 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0039141549999999999, 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0042913919999999998, 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:05:48.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519920+0000", 2024-09-14T04:05:48.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519920+0000", 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519920+0000", 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:05:48.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519924+0000", 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7509999999999999e-06 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519932+0000", 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7039999999999994e-06 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.519936+0000", 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2989999999999998e-06 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.520034+0000", 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8004000000000006e-05 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:05:48.520053+0000", 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9196000000000001e-05 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:05:48.552 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:05:48.553 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 6 2024-09-14T04:05:48.553 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:05:48.553 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:48.594 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:48.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:48.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:48.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:48.731 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:48.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:48.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:48.864 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:48.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:48.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:48.952 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:05:48.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:49.006 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:49.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:49.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:49.143 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:49.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:48 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2819219046' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:49.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:49.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:49.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:49.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-14T04:05:49.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:48 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2819219046' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:49.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:48 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2819219046' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:05:49.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:49.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:49.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:49.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:49.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:49.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:49.720 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:49.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:49.761 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:49.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:49.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:49.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:50.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:50.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:50.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:50.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:50.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:49 smithi160 ceph-mon[88407]: pgmap v602: 89 pgs: 89 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 855 op/s 2024-09-14T04:05:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:49 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:05:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:49 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:05:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:49 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2819219046' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:05:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:49 smithi160 ceph-mon[88407]: osdmap e168: 8 total, 8 up, 8 in 2024-09-14T04:05:50.250 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-14T04:05:50.250 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:05:50.251 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-14T04:05:50.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-14T04:05:50.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:50.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:49 smithi114 ceph-mon[103029]: pgmap v602: 89 pgs: 89 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 855 op/s 2024-09-14T04:05:50.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:49 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:05:50.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:49 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:05:50.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:49 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2819219046' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:05:50.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:49 smithi114 ceph-mon[103029]: osdmap e168: 8 total, 8 up, 8 in 2024-09-14T04:05:50.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:49 smithi114 ceph-mon[106969]: pgmap v602: 89 pgs: 89 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 855 op/s 2024-09-14T04:05:50.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:49 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:05:50.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:49 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:05:50.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:49 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2819219046' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:05:50.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:49 smithi114 ceph-mon[106969]: osdmap e168: 8 total, 8 up, 8 in 2024-09-14T04:05:50.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:50.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:50.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:50.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:50.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:50.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:50.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:50.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:05:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:05:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:05:50.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:50.702 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:50.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:50.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:50.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:50.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:50.918 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:05:50.919 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:05:50.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:50.955 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:05:50.955 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:05:51.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:50 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/299866933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:51.002 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:05:51.003 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:05:51.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:50 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/299866933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:51.018 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:51.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:51.022 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:51.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:51.035 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:05:51.036 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:05:51.067 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:05:51.067 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:05:51.102 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:05:51.102 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:05:51.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:51.129 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:05:51.129 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:05:51.129 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:50 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/299866933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:05:51.164 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:05:51.165 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:05:51.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:51.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:51.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:51.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:51.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:51.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:51.449 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:51.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:51.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:51.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:51.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:51.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:51.618 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:05:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:05:51] "GET /metrics HTTP/1.1" 200 36441 "" "Prometheus/2.51.0" 2024-09-14T04:05:51.697 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:51.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:51.739 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:51.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:51.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:51.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:51.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:51.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:52.010 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:52.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:52.072 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:05:52.072 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:05:52.073 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:51 smithi160 ceph-mon[88407]: pgmap v604: 89 pgs: 89 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-14T04:05:52.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:52.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:52.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:52.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:05:52.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:51 smithi114 ceph-mon[106969]: pgmap v604: 89 pgs: 89 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-14T04:05:52.207 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:51 smithi114 ceph-mon[103029]: pgmap v604: 89 pgs: 89 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-14T04:05:52.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:52.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:52.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:52.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:52.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:52.427 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:52.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:52.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:52.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:52.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:52.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:52.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:52.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:52.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:52.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:52.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:52.945 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:52.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:52.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:53.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:53.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:53.108 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:53.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:53.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-14T04:05:53.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:53.264 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:53.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:53.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:53.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:53.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:53.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:53.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:53.545 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:05:53.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:53.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:53.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:53.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:53.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:53.700 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:53.700 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:53.700 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:53 smithi114 ceph-mon[106969]: pgmap v605: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-14T04:05:53.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:53.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:53.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:53 smithi114 ceph-mon[103029]: pgmap v605: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-14T04:05:53.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:53.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:53.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:05:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:05:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:53 smithi160 ceph-mon[88407]: pgmap v605: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-14T04:05:53.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:53.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:05:53.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:53.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:53.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:54.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:54.117 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:54.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:54.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:54.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:54.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:54.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:54.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:05:54.225+0000 7f3162f23640 -1 mon.a@0(leader).osd e168 definitely_dead 0 2024-09-14T04:05:54.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:54.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:54.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:54.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:54.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:54.643 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:54 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/461890257' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-14T04:05:54.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:54.660 INFO:teuthology.orchestra.run.smithi114.stderr:marked out osd.4. 2024-09-14T04:05:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:54.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:54.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:54.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:54.814 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:54.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:54.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:54 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/461890257' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-14T04:05:54.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:54 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-14T04:05:54.867 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:54 smithi160 ceph-mon[88407]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-14T04:05:54.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:54.939 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:54.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:55.012 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:54 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-14T04:05:55.012 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:54 smithi114 ceph-mon[103029]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-14T04:05:55.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:54 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/461890257' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-14T04:05:55.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:54 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-14T04:05:55.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:54 smithi114 ceph-mon[106969]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-14T04:05:55.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:55.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:55.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:55.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:55.188 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:55.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:55.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:55.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:55.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:55.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:55.405 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:05:55.405 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:05:55.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:55.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:55.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:55.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:55.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:55.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:55.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:55.696 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:55.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:55.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:55.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:55.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:55.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:55.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:55 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-14T04:05:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:55 smithi114 ceph-mon[103029]: osdmap e169: 8 total, 8 up, 7 in 2024-09-14T04:05:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:55 smithi114 ceph-mon[103029]: pgmap v607: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-14T04:05:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:55 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-14T04:05:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:55 smithi114 ceph-mon[106969]: osdmap e169: 8 total, 8 up, 7 in 2024-09-14T04:05:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:55 smithi114 ceph-mon[106969]: pgmap v607: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-14T04:05:55.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:55.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:55.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:55.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:55 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-14T04:05:55.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:55 smithi160 ceph-mon[88407]: osdmap e169: 8 total, 8 up, 7 in 2024-09-14T04:05:55.993 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:55 smithi160 ceph-mon[88407]: pgmap v607: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-14T04:05:56.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-14T04:05:56.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:56.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:56.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:56.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:56.205 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:56.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:56.260 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:56.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:56.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:56.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:56.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:56.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:56.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:56.553 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:56.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:56.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:56.661 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:05:56.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:56.700 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:56.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:56.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:56.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:56.825 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:56.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:56.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:56.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:56 smithi114 ceph-mon[106969]: osdmap e170: 8 total, 8 up, 7 in 2024-09-14T04:05:56.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:56 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2227349617' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:05:56.930 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:56 smithi114 ceph-mon[103029]: osdmap e170: 8 total, 8 up, 7 in 2024-09-14T04:05:56.930 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:56 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2227349617' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:05:56.950 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:56.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:56.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:57.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:05:57.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:57.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:05:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:56 smithi160 ceph-mon[88407]: osdmap e170: 8 total, 8 up, 7 in 2024-09-14T04:05:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:56 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2227349617' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:05:57.176 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:57.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:57.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:57.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:57.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:57.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:57.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:57.463 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:57.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:57.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:57.594 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:57.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:57.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:57.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:05:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:57.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:57.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:05:57.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:57.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:57.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:58.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:58.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:58.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:58.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2227349617' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[103029]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 168'11794 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[103029]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 168'11947 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[103029]: 9.e starting backfill to osd.0 from (0'0,0'0] MAX to 168'11859 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[103029]: osdmap e171: 8 total, 8 up, 7 in 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[103029]: pgmap v610: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 737 op/s 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2227349617' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[106969]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 168'11794 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[106969]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 168'11947 2024-09-14T04:05:58.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[106969]: 9.e starting backfill to osd.0 from (0'0,0'0] MAX to 168'11859 2024-09-14T04:05:58.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[106969]: osdmap e171: 8 total, 8 up, 7 in 2024-09-14T04:05:58.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:57 smithi114 ceph-mon[106969]: pgmap v610: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 737 op/s 2024-09-14T04:05:58.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:58.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:57 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:05:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:57 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2227349617' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:05:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:57 smithi160 ceph-mon[88407]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 168'11794 2024-09-14T04:05:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:57 smithi160 ceph-mon[88407]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 168'11947 2024-09-14T04:05:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:57 smithi160 ceph-mon[88407]: 9.e starting backfill to osd.0 from (0'0,0'0] MAX to 168'11859 2024-09-14T04:05:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:57 smithi160 ceph-mon[88407]: osdmap e171: 8 total, 8 up, 7 in 2024-09-14T04:05:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:57 smithi160 ceph-mon[88407]: pgmap v610: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 737 op/s 2024-09-14T04:05:58.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:58.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:58.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:58.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:05:58.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:58.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:05:58.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:58.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:58.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:58.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:05:58.590 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:05:58.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:58.692 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:58.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:58.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:58.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:58.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:58.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:58.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:58.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:59.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:59.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:59.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:59.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:05:59.236 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:59.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:05:59.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:59.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:59.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:59.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:05:59.512 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:59.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:05:59.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:05:59.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:59.642 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:59.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:05:59.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:05:59.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:59.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:05:59.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:05:59.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:05:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:05:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:05:59.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:00.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:00.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:00.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:00.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:05:59 smithi114 ceph-mon[103029]: pgmap v611: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 0 B/s, 0 objects/s recovering 2024-09-14T04:06:00.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:05:59 smithi114 ceph-mon[106969]: pgmap v611: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 0 B/s, 0 objects/s recovering 2024-09-14T04:06:00.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:00.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:05:59 smithi160 ceph-mon[88407]: pgmap v611: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 0 B/s, 0 objects/s recovering 2024-09-14T04:06:00.194 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:00.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:00.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:00.302 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:06:00.302 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:06:00.325 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:00.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:00.354 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-14T04:06:00.354 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:06:00.354 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.5297225658462861 2024-09-14T04:06:00.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 3 0.5297225658462861 2024-09-14T04:06:00.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:00.484 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:00.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:00.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:00.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:06:00.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:00.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:00.658 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:00.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:00.659 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:06:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:06:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:06:00.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:00.792 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:00.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:00.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:00.926 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:00.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:01.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:01.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:01.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:01.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:01.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:01.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:01.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:01.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:01.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:01.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:01.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:01.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:01.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:01.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:01.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:01.676 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:06:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:06:01] "GET /metrics HTTP/1.1" 200 36443 "" "Prometheus/2.51.0" 2024-09-14T04:06:01.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:01.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:01.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:01.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:01.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:01.881 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:01.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:01.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:01.957 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:01 smithi114 ceph-mon[103029]: pgmap v612: 89 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 86 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 350 op/s; 12882/209877 objects misplaced (6.138%); 2.6 MiB/s, 1 keys/s, 43 objects/s recovering 2024-09-14T04:06:01.957 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:01 smithi114 ceph-mon[106969]: pgmap v612: 89 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 86 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 350 op/s; 12882/209877 objects misplaced (6.138%); 2.6 MiB/s, 1 keys/s, 43 objects/s recovering 2024-09-14T04:06:01.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:02.022 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:02.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:02.069 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:06:02.069 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:06:02.069 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:06:02.069 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:06:02.069 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:06:02.069 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:06:02.070 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:06:02.070 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:06:02.070 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:06:02.070 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:06:02.070 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:06:02.070 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:06:02.090 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:06:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:06:02.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:02.134 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:06:02.134 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:06:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:01 smithi160 ceph-mon[88407]: pgmap v612: 89 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 86 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 350 op/s; 12882/209877 objects misplaced (6.138%); 2.6 MiB/s, 1 keys/s, 43 objects/s recovering 2024-09-14T04:06:02.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:02.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:02.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:02.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-14T04:06:02.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:02.326 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:02.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:02.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:02.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:02.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:02.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:02.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:02.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:02.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:02.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:02.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:02.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:02.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:02 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2340603725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:02 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2340603725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:02.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:02.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:02.994 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:03.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:03.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:03.096 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:02 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2340603725' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:03.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:03.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:03.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:03.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:03.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:03.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:03.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:03.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:03.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:03.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:03.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:03.585 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:03.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:03.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:03.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:03.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:03.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:03.870 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:03.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:03 smithi114 ceph-mon[106969]: pgmap v613: 89 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 86 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 364 op/s; 12882/207435 objects misplaced (6.210%); 1.9 MiB/s, 1 keys/s, 32 objects/s recovering 2024-09-14T04:06:03.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:03.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:03 smithi114 ceph-mon[103029]: pgmap v613: 89 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 86 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 364 op/s; 12882/207435 objects misplaced (6.210%); 1.9 MiB/s, 1 keys/s, 32 objects/s recovering 2024-09-14T04:06:03.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:04.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:04.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:04.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:04.141 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:04.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:03 smithi160 ceph-mon[88407]: pgmap v613: 89 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 86 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 364 op/s; 12882/207435 objects misplaced (6.210%); 1.9 MiB/s, 1 keys/s, 32 objects/s recovering 2024-09-14T04:06:04.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:04.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:04.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:04.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:04.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:04.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:04.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:04.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:04.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:04.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:04.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:04.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:04.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:04.823 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:04.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:04.924 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:04.962 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:04.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:05.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:05.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:05.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:05.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:05.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:05.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:05.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:05.321 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:05.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:05.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:05.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:05.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:05.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:05.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:05.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:05.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:05.658 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:05.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:05.760 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:05.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:05.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:05.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:05.946 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:05.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:06.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:06.084 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:06.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:06.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:06.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:05 smithi114 ceph-mon[106969]: pgmap v614: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 11620/196743 objects misplaced (5.906%); 5.2 MiB/s, 1 keys/s, 84 objects/s recovering 2024-09-14T04:06:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:05 smithi114 ceph-mon[103029]: pgmap v614: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 11620/196743 objects misplaced (5.906%); 5.2 MiB/s, 1 keys/s, 84 objects/s recovering 2024-09-14T04:06:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:05 smithi160 ceph-mon[88407]: pgmap v614: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 11620/196743 objects misplaced (5.906%); 5.2 MiB/s, 1 keys/s, 84 objects/s recovering 2024-09-14T04:06:06.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:06.226 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:06.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:06.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:06.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:06.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:06.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:06.491 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:06.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:06.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:06.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:06.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:06.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:06.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:06.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:06.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:06.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:06.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:06.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:07.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:07.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:07.093 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:07.095 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:07.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:07.129 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:07.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:07.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:07.242 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:07.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:07.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:07.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:07.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:07.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:07.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:07.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:07.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:07.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:07.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:07.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:07.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:07.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:07.813 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:07.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:07.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:07.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:07.951 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:07.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:08.044 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:07 smithi114 ceph-mon[106969]: pgmap v615: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 11620/196743 objects misplaced (5.906%); 4.7 MiB/s, 0 keys/s, 76 objects/s recovering 2024-09-14T04:06:08.044 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:08.045 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:08.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:07 smithi114 ceph-mon[103029]: pgmap v615: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 11620/196743 objects misplaced (5.906%); 4.7 MiB/s, 0 keys/s, 76 objects/s recovering 2024-09-14T04:06:08.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:08.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:08.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:08.082 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:08.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:07 smithi160 ceph-mon[88407]: pgmap v615: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 11620/196743 objects misplaced (5.906%); 4.7 MiB/s, 0 keys/s, 76 objects/s recovering 2024-09-14T04:06:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:08.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:08.217 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:08.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:08.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:08.353 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:08.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:08.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:08.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:08.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:08.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:08.624 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:08.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:08.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:08.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:08.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:08.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:08.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:08.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:08.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:09.028 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:09.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:09.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:09.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:06:09.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:09.164 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:09.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:09.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:09.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:09.313 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:09.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:09.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:09.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:09.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:09.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:09.598 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:09.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:09.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:09.737 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:09.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:09.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:09.839 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:09.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:09 smithi114 ceph-mon[103029]: pgmap v616: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 603 op/s; 11620/194466 objects misplaced (5.975%); 3.9 MiB/s, 0 keys/s, 63 objects/s recovering 2024-09-14T04:06:09.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:09 smithi114 ceph-mon[106969]: pgmap v616: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 603 op/s; 11620/194466 objects misplaced (5.975%); 3.9 MiB/s, 0 keys/s, 63 objects/s recovering 2024-09-14T04:06:09.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:09.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:09.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:10.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:10.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:10.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:10.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:10.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:09 smithi160 ceph-mon[88407]: pgmap v616: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 603 op/s; 11620/194466 objects misplaced (5.975%); 3.9 MiB/s, 0 keys/s, 63 objects/s recovering 2024-09-14T04:06:10.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:10.275 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:10.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:10.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:10.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:10.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:10.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:10.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:10.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:10.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:06:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:06:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:06:10.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:10.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:10.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:10.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:10.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:10.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:10.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:10.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:11.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:11.131 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:11.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:11.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:11.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:11.273 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:11.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:11.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:11.413 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:11.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:11.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:11.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:11.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:06:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:06:11] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-09-14T04:06:11.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:11.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:11.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:11.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:11.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:11.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:11.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:11.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:11.954 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:11.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:11.971 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:11 smithi114 ceph-mon[103029]: pgmap v617: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 10409/184377 objects misplaced (5.645%); 6.6 MiB/s, 0 keys/s, 106 objects/s recovering 2024-09-14T04:06:11.972 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:11 smithi114 ceph-mon[106969]: pgmap v617: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 10409/184377 objects misplaced (5.645%); 6.6 MiB/s, 0 keys/s, 106 objects/s recovering 2024-09-14T04:06:12.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:12.085 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:12.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:12.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:11 smithi160 ceph-mon[88407]: pgmap v617: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 10409/184377 objects misplaced (5.645%); 6.6 MiB/s, 0 keys/s, 106 objects/s recovering 2024-09-14T04:06:12.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:12.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:12.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:12.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:12.359 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:12.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:12.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:12.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:12.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:12.554 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:06:12.554 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:06:12.554 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:06:12.554 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:06:12.554 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:06:12.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:12.623 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:12.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:12.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:12.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:12.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:12.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:12.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:12.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:12.895 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:12.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:12.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:13.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:13.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:13.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:13.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:13.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:13.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:13.305 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:13.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:13.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:13.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-14T04:06:13.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:13.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:13.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:13.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:13.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:13.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:13.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:13.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:13.757 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:06:13.768 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:13.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:13.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:13.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:13.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:13.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:14.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:14.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:14.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:14.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:13 smithi114 ceph-mon[106969]: pgmap v618: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 10409/181902 objects misplaced (5.722%); 5.3 MiB/s, 84 objects/s recovering 2024-09-14T04:06:14.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:13 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3056458734' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:14.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:13 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:14.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:13 smithi114 ceph-mon[103029]: pgmap v618: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 10409/181902 objects misplaced (5.722%); 5.3 MiB/s, 84 objects/s recovering 2024-09-14T04:06:14.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:13 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3056458734' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:14.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:13 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:13 smithi160 ceph-mon[88407]: pgmap v618: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 10409/181902 objects misplaced (5.722%); 5.3 MiB/s, 84 objects/s recovering 2024-09-14T04:06:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:13 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3056458734' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:13 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:14.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:14.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:14.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:14.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:14.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:14.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:14.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:14.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:14.457 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:14.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:14.558 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:14.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:14.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:14.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:14.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:14.756 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:14.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:14.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:14.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:14 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:06:14.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:14 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:06:14.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:14 smithi114 ceph-mon[106969]: osdmap e172: 8 total, 8 up, 7 in 2024-09-14T04:06:14.874 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:14 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:06:14.874 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:14 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:06:14.874 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:14 smithi114 ceph-mon[103029]: osdmap e172: 8 total, 8 up, 7 in 2024-09-14T04:06:14.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:14.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:15.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:15.033 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:15.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:15.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:15.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:14 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:06:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:14 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:06:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:14 smithi160 ceph-mon[88407]: osdmap e172: 8 total, 8 up, 7 in 2024-09-14T04:06:15.184 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:15.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:15.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:15.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:15.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:15.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:15.451 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:15.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:15.553 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:15.590 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:15.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:15.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:15.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:15.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:15.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:15.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:15.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:15.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:15.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:15.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:16.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:16.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:16.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:16.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:15 smithi114 ceph-mon[103029]: pgmap v620: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 860 op/s; 9098/170910 objects misplaced (5.323%); 6.8 MiB/s, 109 objects/s recovering 2024-09-14T04:06:16.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:15 smithi114 ceph-mon[106969]: pgmap v620: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 860 op/s; 9098/170910 objects misplaced (5.323%); 6.8 MiB/s, 109 objects/s recovering 2024-09-14T04:06:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:15 smithi160 ceph-mon[88407]: pgmap v620: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 860 op/s; 9098/170910 objects misplaced (5.323%); 6.8 MiB/s, 109 objects/s recovering 2024-09-14T04:06:16.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:16.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:16.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:16.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:16.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:16.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:16.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:16.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:16.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:16.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:16.633 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:16.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:16.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:16.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:16.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:16.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:16.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:16.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:17.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:17.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:06:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:17.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:17.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:17.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:17.174 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:17.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:17.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:17.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:17.322 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:17.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:17.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:17.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:17.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:17.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:17.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:17.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:17.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:17.737 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:17.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:17.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:17.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:17.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:17.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:18.019 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:18.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:18.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:18.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:17 smithi114 ceph-mon[106969]: pgmap v621: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 860 op/s; 9098/170910 objects misplaced (5.323%); 6.8 MiB/s, 109 objects/s recovering 2024-09-14T04:06:18.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:17 smithi114 ceph-mon[103029]: pgmap v621: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 860 op/s; 9098/170910 objects misplaced (5.323%); 6.8 MiB/s, 109 objects/s recovering 2024-09-14T04:06:18.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:18.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:18.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:17 smithi160 ceph-mon[88407]: pgmap v621: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 860 op/s; 9098/170910 objects misplaced (5.323%); 6.8 MiB/s, 109 objects/s recovering 2024-09-14T04:06:18.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:18.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:18.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:18.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:18.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:18.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:18.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:18.541 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:18.557 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:18.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:18.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:18.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:18.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:18.708 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:18.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:18.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:18.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:18.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:18.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:18.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:18.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:19.082 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:19.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:19.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:19.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:19.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:19.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:19.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:19.254 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:06:19.254 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:06:19.319 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:06:19.320 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:06:19.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:19.367 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:06:19.367 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:06:19.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:19.389 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:19.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:19.413 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:06:19.413 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:06:19.450 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:06:19.450 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:06:19.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:19.492 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:06:19.492 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:06:19.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:19.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:19.529 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:06:19.529 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:06:19.559 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:06:19.559 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:06:19.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:19.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:19.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:19.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:19.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:19.786 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 0.529723 (879b) 2024-09-14T04:06:19.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:19.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:19.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:19.926 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:19.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:20.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:20.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:20.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:20.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:20.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:19 smithi114 ceph-mon[103029]: pgmap v622: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 9098/168540 objects misplaced (5.398%); 6.8 MiB/s, 109 objects/s recovering 2024-09-14T04:06:20.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:19 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2656258182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:19 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4267794523' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5297225658462861} : dispatch 2024-09-14T04:06:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:19 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5297225658462861} : dispatch 2024-09-14T04:06:20.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:19 smithi114 ceph-mon[106969]: pgmap v622: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 9098/168540 objects misplaced (5.398%); 6.8 MiB/s, 109 objects/s recovering 2024-09-14T04:06:20.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2656258182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:20.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4267794523' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5297225658462861} : dispatch 2024-09-14T04:06:20.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:19 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5297225658462861} : dispatch 2024-09-14T04:06:20.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:19 smithi160 ceph-mon[88407]: pgmap v622: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 9098/168540 objects misplaced (5.398%); 6.8 MiB/s, 109 objects/s recovering 2024-09-14T04:06:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:19 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2656258182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:19 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4267794523' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5297225658462861} : dispatch 2024-09-14T04:06:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:19 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5297225658462861} : dispatch 2024-09-14T04:06:20.174 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:20.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:20.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:20.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:20.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:20.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:20.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:20.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:20.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:20.550 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:20.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:20.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:06:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:06:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:06:20.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:20.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:20.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:20.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:20.806 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:20.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:20.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:20.931 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:20.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:21.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:21.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:21.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:21.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:21.086 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:20 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.5297225658462861}]': finished 2024-09-14T04:06:21.086 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:20 smithi114 ceph-mon[103029]: osdmap e173: 8 total, 8 up, 7 in 2024-09-14T04:06:21.086 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:20 smithi114 ceph-mon[103029]: osdmap e174: 8 total, 8 up, 7 in 2024-09-14T04:06:21.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:20 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.5297225658462861}]': finished 2024-09-14T04:06:21.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:20 smithi114 ceph-mon[106969]: osdmap e173: 8 total, 8 up, 7 in 2024-09-14T04:06:21.086 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:20 smithi114 ceph-mon[106969]: osdmap e174: 8 total, 8 up, 7 in 2024-09-14T04:06:21.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:20 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.5297225658462861}]': finished 2024-09-14T04:06:21.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:20 smithi160 ceph-mon[88407]: osdmap e173: 8 total, 8 up, 7 in 2024-09-14T04:06:21.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:20 smithi160 ceph-mon[88407]: osdmap e174: 8 total, 8 up, 7 in 2024-09-14T04:06:21.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:21.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:21.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:21.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:21.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:21.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:21.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:21.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:21.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:21.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:21.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:21.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:21.563 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:21.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:21.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:21.690 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:21.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:21.696 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:06:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:06:21] "GET /metrics HTTP/1.1" 200 36468 "" "Prometheus/2.51.0" 2024-09-14T04:06:21.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:21.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:21.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:21.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:21.951 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:21.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:21.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:22.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:22.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:22.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:22.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:22.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:21 smithi114 ceph-mon[106969]: pgmap v624: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 995 op/s; 7935/157989 objects misplaced (5.023%); 8.4 MiB/s, 134 objects/s recovering 2024-09-14T04:06:22.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:21 smithi114 ceph-mon[106969]: osdmap e175: 8 total, 8 up, 7 in 2024-09-14T04:06:22.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:21 smithi114 ceph-mon[103029]: pgmap v624: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 995 op/s; 7935/157989 objects misplaced (5.023%); 8.4 MiB/s, 134 objects/s recovering 2024-09-14T04:06:22.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:21 smithi114 ceph-mon[103029]: osdmap e175: 8 total, 8 up, 7 in 2024-09-14T04:06:22.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:21 smithi160 ceph-mon[88407]: pgmap v624: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 995 op/s; 7935/157989 objects misplaced (5.023%); 8.4 MiB/s, 134 objects/s recovering 2024-09-14T04:06:22.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:21 smithi160 ceph-mon[88407]: osdmap e175: 8 total, 8 up, 7 in 2024-09-14T04:06:22.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:22.210 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:22.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:22.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:22.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:22.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:22.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:22.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:22.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:22.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:22.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:22.634 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:22.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:22.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:22.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:22.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:22.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:22.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:22.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:23.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:23.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:23.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:23.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:22 smithi114 ceph-mon[106969]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 172'12827 2024-09-14T04:06:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:22 smithi114 ceph-mon[106969]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 172'12846 2024-09-14T04:06:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:22 smithi114 ceph-mon[106969]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 172'12616 2024-09-14T04:06:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:22 smithi114 ceph-mon[106969]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 172'13144 2024-09-14T04:06:23.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:23.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:22 smithi114 ceph-mon[103029]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 172'12827 2024-09-14T04:06:23.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:22 smithi114 ceph-mon[103029]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 172'12846 2024-09-14T04:06:23.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:22 smithi114 ceph-mon[103029]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 172'12616 2024-09-14T04:06:23.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:22 smithi114 ceph-mon[103029]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 172'13144 2024-09-14T04:06:23.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:23.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:22 smithi160 ceph-mon[88407]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 172'12827 2024-09-14T04:06:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:22 smithi160 ceph-mon[88407]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 172'12846 2024-09-14T04:06:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:22 smithi160 ceph-mon[88407]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 172'12616 2024-09-14T04:06:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:22 smithi160 ceph-mon[88407]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 172'13144 2024-09-14T04:06:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:23.175 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:23.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:23.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:23.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:23.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:23.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:23.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:23.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:23.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:23.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:23.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:23.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:23.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:23.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:23.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:23.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:23.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:23.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:23.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:23.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:24.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:24.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:24.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:24.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:24.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:23 smithi114 ceph-mon[106969]: pgmap v627: 89 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 85 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 10910/156327 objects misplaced (6.979%); 6.4 MiB/s, 102 objects/s recovering 2024-09-14T04:06:24.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:23 smithi114 ceph-mon[106969]: osdmap e176: 8 total, 8 up, 7 in 2024-09-14T04:06:24.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:23 smithi114 ceph-mon[103029]: pgmap v627: 89 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 85 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 10910/156327 objects misplaced (6.979%); 6.4 MiB/s, 102 objects/s recovering 2024-09-14T04:06:24.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:23 smithi114 ceph-mon[103029]: osdmap e176: 8 total, 8 up, 7 in 2024-09-14T04:06:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:23 smithi160 ceph-mon[88407]: pgmap v627: 89 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 85 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 10910/156327 objects misplaced (6.979%); 6.4 MiB/s, 102 objects/s recovering 2024-09-14T04:06:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:23 smithi160 ceph-mon[88407]: osdmap e176: 8 total, 8 up, 7 in 2024-09-14T04:06:24.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:24.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:24.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:24.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:24.373 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:24.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:24.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:24.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:24.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:24.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:24.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:06:24.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:24.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:24.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:24.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:24.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:24.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:24.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:24.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:24.920 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:06:24.920 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:06:24.946 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:24.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:25.020 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:24 smithi114 ceph-mon[106969]: osdmap e177: 8 total, 8 up, 7 in 2024-09-14T04:06:25.020 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:24 smithi114 ceph-mon[103029]: osdmap e177: 8 total, 8 up, 7 in 2024-09-14T04:06:25.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:25.085 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:25.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:24 smithi160 ceph-mon[88407]: osdmap e177: 8 total, 8 up, 7 in 2024-09-14T04:06:25.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:25.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:25.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:25.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:25.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:25.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:25.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:25.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:25.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:25.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:25.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:25.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:25.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:25.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:25.746 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:25.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:25.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:25.860 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:06:25.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:25.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:25.984 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:26.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:26.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:26.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:26.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:25 smithi114 ceph-mon[106969]: pgmap v630: 89 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 83 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 482 op/s; 17577/152199 objects misplaced (11.549%); 41 MiB/s, 657 objects/s recovering 2024-09-14T04:06:26.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3198381375' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:06:26.094 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:25 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:06:26.094 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:25 smithi114 ceph-mon[103029]: pgmap v630: 89 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 83 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 482 op/s; 17577/152199 objects misplaced (11.549%); 41 MiB/s, 657 objects/s recovering 2024-09-14T04:06:26.094 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3198381375' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:06:26.094 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:25 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:06:26.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:26.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-14T04:06:26.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:26.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:25 smithi160 ceph-mon[88407]: pgmap v630: 89 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 83 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 482 op/s; 17577/152199 objects misplaced (11.549%); 41 MiB/s, 657 objects/s recovering 2024-09-14T04:06:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3198381375' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:06:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:25 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:06:26.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:26.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:26.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:26.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:26.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:26.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:26.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:26.468 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-14T04:06:26.468 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:06:26.468 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-14T04:06:26.469 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-14T04:06:26.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.4 2024-09-14T04:06:26.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:26.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:26.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:26.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:26.706 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:26.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:26.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:26.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:26.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:26.870 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:26 smithi160 systemd[1]: Stopping Ceph osd.4 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:06:26.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:26.956 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:26.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:26.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:27.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:26 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:06:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:26 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:06:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:26 smithi114 ceph-mon[106969]: osdmap e178: 8 total, 8 up, 7 in 2024-09-14T04:06:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:26 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:06:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:26 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:06:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:26 smithi114 ceph-mon[103029]: osdmap e178: 8 total, 8 up, 7 in 2024-09-14T04:06:27.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:27.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:26 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:06:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:26 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:06:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:26 smithi160 ceph-mon[88407]: osdmap e178: 8 total, 8 up, 7 in 2024-09-14T04:06:27.159 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:27 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[37995]: 2024-09-14T04:06:27.049+0000 7fc17434a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:06:27.159 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:27 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[37995]: 2024-09-14T04:06:27.049+0000 7fc17434a640 -1 osd.4 178 *** Got signal Terminated *** 2024-09-14T04:06:27.159 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:27 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[37995]: 2024-09-14T04:06:27.049+0000 7fc17434a640 -1 osd.4 178 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T04:06:27.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:27.254 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:27.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:27.293 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:27.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:27.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:27.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:27.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:27.515 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:27.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:27.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:27.655 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:27.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:27.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:27.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:27.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:27.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:27.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:27.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:27.927 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:27.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:28.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:28.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:28.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:28.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:27 smithi114 ceph-mon[103029]: pgmap v632: 89 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 83 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 394 op/s; 17577/152199 objects misplaced (11.549%); 34 MiB/s, 537 objects/s recovering 2024-09-14T04:06:28.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:27 smithi114 ceph-mon[103029]: osd.4 marked itself down and dead 2024-09-14T04:06:28.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:27 smithi114 ceph-mon[106969]: pgmap v632: 89 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 83 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 394 op/s; 17577/152199 objects misplaced (11.549%); 34 MiB/s, 537 objects/s recovering 2024-09-14T04:06:28.065 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:27 smithi114 ceph-mon[106969]: osd.4 marked itself down and dead 2024-09-14T04:06:28.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:27 smithi160 ceph-mon[88407]: pgmap v632: 89 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 83 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 394 op/s; 17577/152199 objects misplaced (11.549%); 34 MiB/s, 537 objects/s recovering 2024-09-14T04:06:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:27 smithi160 ceph-mon[88407]: osd.4 marked itself down and dead 2024-09-14T04:06:28.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:28.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:28.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:28.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:28.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:28.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:28.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:28.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:28.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:28.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:28.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:28.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:28.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:28.681 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:28.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:28.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:28.807 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:28.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:28.815 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:28 smithi160 podman[114263]: 2024-09-14 04:06:28.511343588 +0000 UTC m=+1.609922713 container died e89693bb3b7632b0fae2a7c2c6401ef1ce89b6b43e291981e0c3f23797791203 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4, org.label-schema.vendor=CentOS, RELEASE=quincy-eb16b97, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-14T04:06:28.909 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:28.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:28.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:28.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:06:29.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:29.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:29.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:28 smithi160 ceph-mon[88407]: osdmap e179: 8 total, 7 up, 7 in 2024-09-14T04:06:29.158 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:28 smithi160 podman[114263]: 2024-09-14 04:06:28.913672205 +0000 UTC m=+2.012251328 container cleanup e89693bb3b7632b0fae2a7c2c6401ef1ce89b6b43e291981e0c3f23797791203 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=quincy-eb16b97, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T04:06:29.158 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:28 smithi160 bash[114263]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4 2024-09-14T04:06:29.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:29.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:29.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:29.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:28 smithi114 ceph-mon[103029]: osdmap e179: 8 total, 7 up, 7 in 2024-09-14T04:06:29.267 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:28 smithi114 ceph-mon[106969]: osdmap e179: 8 total, 7 up, 7 in 2024-09-14T04:06:29.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:29.367 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:29.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:29.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:29.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:29.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:29.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:29.650 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:29.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:29.658 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:29 smithi160 podman[114437]: 2024-09-14 04:06:29.207550742 +0000 UTC m=+0.692873453 container remove e89693bb3b7632b0fae2a7c2c6401ef1ce89b6b43e291981e0c3f23797791203 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=quincy-eb16b97, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T04:06:29.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:29.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:29.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:29.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:29.930 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:29.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:30.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:30.046 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:29 smithi160 ceph-mon[88407]: pgmap v634: 89 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 83 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 392 op/s; 16536/149418 objects misplaced (11.067%); 36 MiB/s, 575 objects/s recovering 2024-09-14T04:06:30.046 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:29 smithi160 ceph-mon[88407]: osdmap e180: 8 total, 7 up, 7 in 2024-09-14T04:06:30.046 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:29 smithi160 ceph-mon[88407]: 9.c continuing backfill to osd.0 from (160'6329,178'13465] MIN to 178'13465 2024-09-14T04:06:30.046 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:29 smithi160 podman[114587]: 2024-09-14 04:06:29.625262286 +0000 UTC m=+0.026727335 image pull ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6 2024-09-14T04:06:30.046 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:29 smithi160 podman[114587]: 2024-09-14 04:06:29.733663438 +0000 UTC m=+0.135128472 container create 86cfa7d55f8320c7cea7e00f1f4626eca25c0bb9552f71431f821cb357f8d749 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-14T04:06:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:30.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:30.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:30.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:30.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:29 smithi114 ceph-mon[103029]: pgmap v634: 89 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 83 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 392 op/s; 16536/149418 objects misplaced (11.067%); 36 MiB/s, 575 objects/s recovering 2024-09-14T04:06:30.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:29 smithi114 ceph-mon[103029]: osdmap e180: 8 total, 7 up, 7 in 2024-09-14T04:06:30.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:29 smithi114 ceph-mon[103029]: 9.c continuing backfill to osd.0 from (160'6329,178'13465] MIN to 178'13465 2024-09-14T04:06:30.221 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:29 smithi114 ceph-mon[106969]: pgmap v634: 89 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_wait, 83 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 392 op/s; 16536/149418 objects misplaced (11.067%); 36 MiB/s, 575 objects/s recovering 2024-09-14T04:06:30.222 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:29 smithi114 ceph-mon[106969]: osdmap e180: 8 total, 7 up, 7 in 2024-09-14T04:06:30.222 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:29 smithi114 ceph-mon[106969]: 9.c continuing backfill to osd.0 from (160'6329,178'13465] MIN to 178'13465 2024-09-14T04:06:30.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:30.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:30.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:30.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:30.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:30.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:30.383 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:06:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:06:30.216Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:06:30.383 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:30 smithi160 podman[114587]: 2024-09-14 04:06:30.045264733 +0000 UTC m=+0.446729779 container init 86cfa7d55f8320c7cea7e00f1f4626eca25c0bb9552f71431f821cb357f8d749 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731) 2024-09-14T04:06:30.383 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:30 smithi160 podman[114587]: 2024-09-14 04:06:30.055845978 +0000 UTC m=+0.457311018 container start 86cfa7d55f8320c7cea7e00f1f4626eca25c0bb9552f71431f821cb357f8d749 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-14T04:06:30.383 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:30 smithi160 podman[114587]: 2024-09-14 04:06:30.179098514 +0000 UTC m=+0.580563553 container attach 86cfa7d55f8320c7cea7e00f1f4626eca25c0bb9552f71431f821cb357f8d749 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-eb16b97, org.label-schema.build-date=20240731) 2024-09-14T04:06:30.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:30.504 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:30.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:30.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:30.638 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:30.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:30.657 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:30 smithi160 conmon[114634]: conmon 86cfa7d55f8320c7cea7 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-86cfa7d55f8320c7cea7e00f1f4626eca25c0bb9552f71431f821cb357f8d749.scope/memory.events 2024-09-14T04:06:30.658 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:30 smithi160 podman[114587]: 2024-09-14 04:06:30.384418382 +0000 UTC m=+0.785883422 container died 86cfa7d55f8320c7cea7e00f1f4626eca25c0bb9552f71431f821cb357f8d749 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=quincy-eb16b97, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-14T04:06:30.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:30.777 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:30.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:30.854 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:06:30.854 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:06:30.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:30.919 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:30.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:31.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:31.060 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:30 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1535023862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:31.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:30 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1535023862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:31.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:31.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:31.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:30 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1535023862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:31.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:31.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:31.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:31.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:31.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.4.service' 2024-09-14T04:06:31.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:31.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:31.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:31.362 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:06:31.363 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-14T04:06:31.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:31.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:31.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:31.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:31.627 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:31.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:31.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:31.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:31.851 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:06:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:06:31] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-09-14T04:06:31.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:31.916 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:31.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:32.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:32.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:32.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:32.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:31 smithi114 ceph-mon[106969]: pgmap v636: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 587 op/s; 4907/141615 objects degraded (3.465%); 9225/141615 objects misplaced (6.514%); 24 MiB/s, 386 objects/s recovering 2024-09-14T04:06:32.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:31 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 4907/141615 objects degraded (3.465%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:06:32.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:31 smithi114 ceph-mon[103029]: pgmap v636: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 587 op/s; 4907/141615 objects degraded (3.465%); 9225/141615 objects misplaced (6.514%); 24 MiB/s, 386 objects/s recovering 2024-09-14T04:06:32.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:31 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 4907/141615 objects degraded (3.465%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:06:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:31 smithi160 ceph-mon[88407]: pgmap v636: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 587 op/s; 4907/141615 objects degraded (3.465%); 9225/141615 objects misplaced (6.514%); 24 MiB/s, 386 objects/s recovering 2024-09-14T04:06:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:31 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 4907/141615 objects degraded (3.465%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:06:32.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:32.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:32.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:32.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:32.291 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:06:32.291 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:06:32.291 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:06:32.291 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:06:32.292 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:06:32.292 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:06:32.292 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:06:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:06:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:06:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:06:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:06:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-14T04:06:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:06:32.292 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:06:32.293 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:06:32.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:32.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:32.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:32.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:32.471 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:32.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:32.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:32.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:32.609 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:32.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:32.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:32.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:32.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:32.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:32.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:32.873 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:32.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:32.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:32.980 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-14T04:06:32.981 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-14T04:06:33.023 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:33.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:33.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:33.134 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:33.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:33.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:33.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:33.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:33.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:33.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:33.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:33.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-14T04:06:33.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:33.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:33.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:33.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:33.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:33.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:33.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:33.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:33.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:33.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:33.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:33.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:33.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:34.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:34.073 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:34.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:34.089 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:33 smithi114 ceph-mon[103029]: pgmap v637: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 4907/139095 objects degraded (3.528%); 8182/139095 objects misplaced (5.882%); 30 MiB/s, 478 objects/s recovering 2024-09-14T04:06:34.089 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:33 smithi114 ceph-mon[106969]: pgmap v637: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 4907/139095 objects degraded (3.528%); 8182/139095 objects misplaced (5.882%); 30 MiB/s, 478 objects/s recovering 2024-09-14T04:06:34.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:34.210 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:34.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:34.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:34.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:34.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:34.394 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:33 smithi160 ceph-mon[88407]: pgmap v637: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 4907/139095 objects degraded (3.528%); 8182/139095 objects misplaced (5.882%); 30 MiB/s, 478 objects/s recovering 2024-09-14T04:06:34.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:34.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:34.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:34.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:34.655 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:34.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:34.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:34.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:34.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:34.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:34.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:34.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:34.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:06:35.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:35.088 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:35.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:35.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:35.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:35.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:35.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:35.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:35.348 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:35.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:35.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:35.479 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:35.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:35.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:35.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:35.612 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:35.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:35.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:35.736 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:35.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:35.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:35.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:35.873 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:35.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:35.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:36.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:36.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:36.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:36.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:36.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:36.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:36.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:36.363 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7] 2024-09-14T04:06:36.363 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-09-14T04:06:36.363 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-09-14T04:06:36.364 DEBUG:teuthology.orchestra.run.smithi160:osd.4> sudo journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.4.service 2024-09-14T04:06:36.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl start ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.4 2024-09-14T04:06:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:36.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:36.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:36.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:35 smithi114 ceph-mon[106969]: pgmap v638: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 3938/129546 objects degraded (3.040%); 7010/129546 objects misplaced (5.411%); 34 MiB/s, 548 objects/s recovering 2024-09-14T04:06:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:35 smithi114 ceph-mon[103029]: pgmap v638: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 3938/129546 objects degraded (3.040%); 7010/129546 objects misplaced (5.411%); 34 MiB/s, 548 objects/s recovering 2024-09-14T04:06:36.391 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:35 smithi160 ceph-mon[88407]: pgmap v638: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 3938/129546 objects degraded (3.040%); 7010/129546 objects misplaced (5.411%); 34 MiB/s, 548 objects/s recovering 2024-09-14T04:06:36.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:36.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:36.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:36.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:36.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:36.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:36.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:36.658 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:36 smithi160 systemd[1]: Starting Ceph osd.4 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:06:36.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:36.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:36.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:36.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:36.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:36.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:36.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:37.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:37.051 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:36 smithi160 podman[115112]: 2024-09-14 04:06:36.662318301 +0000 UTC m=+0.029938806 image pull ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6 2024-09-14T04:06:37.051 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:36 smithi160 podman[115112]: 2024-09-14 04:06:36.786525551 +0000 UTC m=+0.154146050 container create 22a7edcfdcaf56f41910082848626a97d29257a77f4f90a4def18c82d0bc39fc (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-14T04:06:37.067 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:37.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:37.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:37.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:37.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:37.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:37.307 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 podman[115112]: 2024-09-14 04:06:37.050359376 +0000 UTC m=+0.417979873 container init 22a7edcfdcaf56f41910082848626a97d29257a77f4f90a4def18c82d0bc39fc (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate, RELEASE=quincy-eb16b97, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-14T04:06:37.307 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 podman[115112]: 2024-09-14 04:06:37.060686464 +0000 UTC m=+0.428306956 container start 22a7edcfdcaf56f41910082848626a97d29257a77f4f90a4def18c82d0bc39fc (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-14T04:06:37.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:37.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:37.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:37.475 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:37.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:37.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:37.578 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 podman[115112]: 2024-09-14 04:06:37.30608893 +0000 UTC m=+0.673709427 container attach 22a7edcfdcaf56f41910082848626a97d29257a77f4f90a4def18c82d0bc39fc (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-eb16b97) 2024-09-14T04:06:37.578 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate[115125]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-14T04:06:37.578 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 bash[115112]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-14T04:06:37.578 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate[115125]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-14T04:06:37.578 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 bash[115112]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-14T04:06:37.578 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate[115125]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-14T04:06:37.578 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 bash[115112]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-14T04:06:37.579 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate[115125]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-14T04:06:37.579 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 bash[115112]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-14T04:06:37.579 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate[115125]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-14T04:06:37.579 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 bash[115112]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-14T04:06:37.579 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate[115125]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-14T04:06:37.579 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 bash[115112]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-14T04:06:37.579 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate[115125]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-14T04:06:37.579 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 bash[115112]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-14T04:06:37.603 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:37.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:37.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:37.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:37.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:37.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:37.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:37.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:37.908 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:37 smithi160 podman[115112]: 2024-09-14 04:06:37.576212671 +0000 UTC m=+0.943833171 container died 22a7edcfdcaf56f41910082848626a97d29257a77f4f90a4def18c82d0bc39fc (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-14T04:06:37.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:37.998 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:37.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:38.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:38.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:37 smithi114 ceph-mon[103029]: pgmap v639: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 3938/129546 objects degraded (3.040%); 7010/129546 objects misplaced (5.411%); 25 MiB/s, 401 objects/s recovering 2024-09-14T04:06:38.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:06:38.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:38.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:38.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:37 smithi114 ceph-mon[106969]: pgmap v639: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 3938/129546 objects degraded (3.040%); 7010/129546 objects misplaced (5.411%); 25 MiB/s, 401 objects/s recovering 2024-09-14T04:06:38.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:06:38.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:38.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:38.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:38.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:38.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:38.257 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:38.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:38.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:38.390 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:38.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-mon[88407]: pgmap v639: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 3938/129546 objects degraded (3.040%); 7010/129546 objects misplaced (5.411%); 25 MiB/s, 401 objects/s recovering 2024-09-14T04:06:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:06:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:38.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:38.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:38.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:38.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:38.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:38.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:38.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:38.817 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:38.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:38.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:38.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:38.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:39.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:39.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:39.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:39.192 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:39.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:39.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:39.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:39.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:39.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:38 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3938/126495 objects degraded (3.113%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:06:39.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:38 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3938/126495 objects degraded (3.113%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:06:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:39.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:39.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:38 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3938/126495 objects degraded (3.113%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:06:39.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:39.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:39.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:39.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:39.646 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:39.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:39.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:39.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:39.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:39.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:39.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:39.901 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:39.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:40.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:40.024 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:40.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:40.126 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:40.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:40.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:39 smithi160 ceph-mon[88407]: pgmap v640: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 3938/126495 objects degraded (3.113%); 6384/126495 objects misplaced (5.047%); 26 MiB/s, 410 objects/s recovering 2024-09-14T04:06:40.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:40.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:40.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:40.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:39 smithi114 ceph-mon[103029]: pgmap v640: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 3938/126495 objects degraded (3.113%); 6384/126495 objects misplaced (5.047%); 26 MiB/s, 410 objects/s recovering 2024-09-14T04:06:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:39 smithi114 ceph-mon[106969]: pgmap v640: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 3938/126495 objects degraded (3.113%); 6384/126495 objects misplaced (5.047%); 26 MiB/s, 410 objects/s recovering 2024-09-14T04:06:40.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:40.408 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:40.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:40.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:40.535 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:40.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:40.564 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:40 smithi160 podman[115284]: 2024-09-14 04:06:40.329266832 +0000 UTC m=+2.743815002 container remove 22a7edcfdcaf56f41910082848626a97d29257a77f4f90a4def18c82d0bc39fc (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-eb16b97, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-14T04:06:40.565 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:06:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:06:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:06:40.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:40.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:40.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:40.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:40.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:40.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:40.834 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:40 smithi160 podman[115586]: 2024-09-14 04:06:40.468392965 +0000 UTC m=+0.023492354 image pull ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6 2024-09-14T04:06:40.834 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:40 smithi160 podman[115586]: 2024-09-14 04:06:40.596970369 +0000 UTC m=+0.152069747 container create f0bdd007efc97506375670ae82baa52fc5b536b4842ccfd06220394d7030839f (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97) 2024-09-14T04:06:40.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:40.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:40.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:40.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:06:41.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:41.024 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:06:41.039 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:41.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:41.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:41.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:41 smithi160 ceph-mon[88407]: osdmap e181: 8 total, 7 up, 7 in 2024-09-14T04:06:41.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:41 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1541927043' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:41.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:41 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:41.159 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:40 smithi160 podman[115586]: 2024-09-14 04:06:40.833572204 +0000 UTC m=+0.388671594 container init f0bdd007efc97506375670ae82baa52fc5b536b4842ccfd06220394d7030839f (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-eb16b97, org.label-schema.license=GPLv2) 2024-09-14T04:06:41.159 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:40 smithi160 podman[115586]: 2024-09-14 04:06:40.83944375 +0000 UTC m=+0.394543126 container start f0bdd007efc97506375670ae82baa52fc5b536b4842ccfd06220394d7030839f (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-14T04:06:41.159 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:40 smithi160 ceph-osd[115678]: -- 172.21.15.160:0/3540709436 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f97c9f8d00 con 0x55f97c9cb400 2024-09-14T04:06:41.159 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:40 smithi160 bash[115586]: f0bdd007efc97506375670ae82baa52fc5b536b4842ccfd06220394d7030839f 2024-09-14T04:06:41.159 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:40 smithi160 systemd[1]: Started Ceph osd.4 for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T04:06:41.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:41.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:41.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:41.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:41.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:41.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:41.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:41 smithi114 ceph-mon[103029]: osdmap e181: 8 total, 7 up, 7 in 2024-09-14T04:06:41.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:41 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1541927043' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:41.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:41 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:41.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:41 smithi114 ceph-mon[106969]: osdmap e181: 8 total, 7 up, 7 in 2024-09-14T04:06:41.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:41 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1541927043' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:41.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:41 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:06:41.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:41.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:41.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:41.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:41.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:41.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:41.658 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:06:41.382+0000 7f19558bb440 -1 Falling back to public interface 2024-09-14T04:06:41.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:41.701 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:06:41.702 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:06:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:06:41] "GET /metrics HTTP/1.1" 200 36333 "" "Prometheus/2.51.0" 2024-09-14T04:06:41.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:41.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:41.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:41.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:41.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:41.904 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:06:41.905 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:06:41.905 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:06:41.905 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:06:41.905 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:06:41.947 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:41.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:41.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:42.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:42.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:42.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:42.135 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:42 smithi160 ceph-mon[88407]: pgmap v642: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 813 op/s; 3007/117198 objects degraded (2.566%); 5232/117198 objects misplaced (4.464%); 24 MiB/s, 382 objects/s recovering 2024-09-14T04:06:42.136 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:42 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:06:42.136 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:42 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:06:42.136 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:42 smithi160 ceph-mon[88407]: osdmap e182: 8 total, 7 up, 7 in 2024-09-14T04:06:42.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:42.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:42.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:42.271 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:42 smithi114 ceph-mon[106969]: pgmap v642: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 813 op/s; 3007/117198 objects degraded (2.566%); 5232/117198 objects misplaced (4.464%); 24 MiB/s, 382 objects/s recovering 2024-09-14T04:06:42.272 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:42 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:06:42.272 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:42 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:06:42.272 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:42 smithi114 ceph-mon[106969]: osdmap e182: 8 total, 7 up, 7 in 2024-09-14T04:06:42.272 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:42 smithi114 ceph-mon[103029]: pgmap v642: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 83 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 813 op/s; 3007/117198 objects degraded (2.566%); 5232/117198 objects misplaced (4.464%); 24 MiB/s, 382 objects/s recovering 2024-09-14T04:06:42.272 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:42 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:06:42.272 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:42 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:06:42.272 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:42 smithi114 ceph-mon[103029]: osdmap e182: 8 total, 7 up, 7 in 2024-09-14T04:06:42.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:42.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:42.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:42.503 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:42.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:42.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:42.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:42.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:42.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:42.756 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:42.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:42.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:42.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:42.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:42.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:43.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:43.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:43.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:43.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:43.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:43.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:43.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:43.303 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:43.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:43.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:43 smithi114 ceph-mon[103029]: osdmap e183: 8 total, 7 up, 7 in 2024-09-14T04:06:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:43 smithi114 ceph-mon[106969]: osdmap e183: 8 total, 7 up, 7 in 2024-09-14T04:06:43.382 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:43 smithi160 ceph-mon[88407]: osdmap e183: 8 total, 7 up, 7 in 2024-09-14T04:06:43.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:43.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:43.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:43.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:43.594 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:43.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:43.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:43.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:43.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:43.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:43.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:43.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:43.943 INFO:teuthology.orchestra.run.smithi160.stderr:no valid command found; 10 closest matches: 2024-09-14T04:06:43.943 INFO:teuthology.orchestra.run.smithi160.stderr:0 2024-09-14T04:06:43.943 INFO:teuthology.orchestra.run.smithi160.stderr:1 2024-09-14T04:06:43.943 INFO:teuthology.orchestra.run.smithi160.stderr:2 2024-09-14T04:06:43.943 INFO:teuthology.orchestra.run.smithi160.stderr:abort 2024-09-14T04:06:43.943 INFO:teuthology.orchestra.run.smithi160.stderr:assert 2024-09-14T04:06:43.944 INFO:teuthology.orchestra.run.smithi160.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:06:43.944 INFO:teuthology.orchestra.run.smithi160.stderr:bluefs files list 2024-09-14T04:06:43.944 INFO:teuthology.orchestra.run.smithi160.stderr:bluefs stats 2024-09-14T04:06:43.944 INFO:teuthology.orchestra.run.smithi160.stderr:bluestore bluefs device info [] 2024-09-14T04:06:43.944 INFO:teuthology.orchestra.run.smithi160.stderr:config diff 2024-09-14T04:06:43.944 INFO:teuthology.orchestra.run.smithi160.stderr:admin_socket: invalid command 2024-09-14T04:06:43.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:44.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:44.024 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:44.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:44.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:44.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:44.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:44.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:44.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:44 smithi160 ceph-mon[88407]: pgmap v645: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 84 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 792 op/s; 3007/115269 objects degraded (2.609%); 4967/115269 objects misplaced (4.309%); 21 MiB/s, 334 objects/s recovering 2024-09-14T04:06:44.279 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:44 smithi160 ceph-mon[88407]: osdmap e184: 8 total, 7 up, 7 in 2024-09-14T04:06:44.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:44.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:44.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:44.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:44 smithi114 ceph-mon[106969]: pgmap v645: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 84 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 792 op/s; 3007/115269 objects degraded (2.609%); 4967/115269 objects misplaced (4.309%); 21 MiB/s, 334 objects/s recovering 2024-09-14T04:06:44.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:44 smithi114 ceph-mon[106969]: osdmap e184: 8 total, 7 up, 7 in 2024-09-14T04:06:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:44 smithi114 ceph-mon[103029]: pgmap v645: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 84 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 792 op/s; 3007/115269 objects degraded (2.609%); 4967/115269 objects misplaced (4.309%); 21 MiB/s, 334 objects/s recovering 2024-09-14T04:06:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:44 smithi114 ceph-mon[103029]: osdmap e184: 8 total, 7 up, 7 in 2024-09-14T04:06:44.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:44.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:44.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:44.561 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:44.604 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:44.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:44.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:44.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:44.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:44.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:44.883 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:44.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:44.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:06:45.002 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-14T04:06:45.025 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:45.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:45.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:45.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-14T04:06:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:45.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:45.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:45.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:45.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:45.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:45.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:45 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3007/115269 objects degraded (2.609%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:06:45.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:45 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3007/115269 objects degraded (2.609%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:06:45.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:45 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3007/115269 objects degraded (2.609%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:06:45.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:45.488 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:45.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:45.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:45.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:45.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:45.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:45.781 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:45.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:45.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:45.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:45.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:45.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:46.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:46.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:46.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:46.125 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:46.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:46 smithi160 ceph-mon[88407]: pgmap v647: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 503 op/s; 1401/110112 objects degraded (1.272%); 1863/110112 objects misplaced (1.692%); 62 MiB/s, 995 objects/s recovering 2024-09-14T04:06:46.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:46.228 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:46.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:46.303 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:46 smithi114 ceph-mon[106969]: pgmap v647: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 503 op/s; 1401/110112 objects degraded (1.272%); 1863/110112 objects misplaced (1.692%); 62 MiB/s, 995 objects/s recovering 2024-09-14T04:06:46.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:46 smithi114 ceph-mon[103029]: pgmap v647: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 503 op/s; 1401/110112 objects degraded (1.272%); 1863/110112 objects misplaced (1.692%); 62 MiB/s, 995 objects/s recovering 2024-09-14T04:06:46.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:46.388 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:46.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:46.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:46.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:46.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:46.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:06:46.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:46.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:46.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:46.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:46.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:46.856 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:46.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:46.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:47.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:47.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:47.143 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:47.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:47.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:47.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:47.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:47.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:47.421 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:47.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:47.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:47.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:47.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:47.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:47.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:47.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:47.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:47.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:47.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:47.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:47.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:47.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:47.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:48.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:48.135 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:48.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:48.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:48.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:48.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:48.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:48 smithi114 ceph-mon[106969]: pgmap v648: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 393 op/s; 1401/110112 objects degraded (1.272%); 1863/110112 objects misplaced (1.692%); 49 MiB/s, 778 objects/s recovering 2024-09-14T04:06:48.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:48 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/198508269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:48.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:48 smithi114 ceph-mon[103029]: pgmap v648: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 393 op/s; 1401/110112 objects degraded (1.272%); 1863/110112 objects misplaced (1.692%); 49 MiB/s, 778 objects/s recovering 2024-09-14T04:06:48.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:48 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/198508269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:48.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:48 smithi160 ceph-mon[88407]: pgmap v648: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 393 op/s; 1401/110112 objects degraded (1.272%); 1863/110112 objects misplaced (1.692%); 49 MiB/s, 778 objects/s recovering 2024-09-14T04:06:48.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:48 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/198508269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:06:48.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:48.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:48.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:48.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:48.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:48.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:48.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:48.719 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:48.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:48.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:48.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:48.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:48.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:49.003 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:49.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:49.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:49.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:49.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:49.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:49.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:49.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:49.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:49.351 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:06:49.351 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:06:49.351 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:06:49.351 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:06:49.351 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:06:49.351 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:06:49.351 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:06:49.351 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:06:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:06:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:06:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:06:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-14T04:06:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:06:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:06:49.352 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:06:49.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:49.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:49.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:49.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:49.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:49.568 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:49.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:49.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:49.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:49.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:49.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:49.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:49.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:49.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:49.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:50.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:06:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:50.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:50.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:50.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:50.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:50.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:50.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:50.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-14T04:06:50.303 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-14T04:06:50.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:50 smithi114 ceph-mon[106969]: pgmap v649: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 439 op/s; 1401/107067 objects degraded (1.309%); 1863/107067 objects misplaced (1.740%); 38 MiB/s, 607 objects/s recovering 2024-09-14T04:06:50.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:50 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1401/107067 objects degraded (1.309%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:06:50.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:50 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:50.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:50 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:50.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:50 smithi114 ceph-mon[103029]: pgmap v649: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 439 op/s; 1401/107067 objects degraded (1.309%); 1863/107067 objects misplaced (1.740%); 38 MiB/s, 607 objects/s recovering 2024-09-14T04:06:50.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:50 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1401/107067 objects degraded (1.309%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:06:50.304 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:50 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:50.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:50 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:50.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:06:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:06:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:06:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:50 smithi160 ceph-mon[88407]: pgmap v649: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 439 op/s; 1401/107067 objects degraded (1.309%); 1863/107067 objects misplaced (1.740%); 38 MiB/s, 607 objects/s recovering 2024-09-14T04:06:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:50 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1401/107067 objects degraded (1.309%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:06:50.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:50 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:50.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:50 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:50.431 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:50.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:50.450 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:06:50.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:50.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:50.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:50.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:50.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:50.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:50.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:50.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:50.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:50.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:50.938 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:06:50.938 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:06:50.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:51.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:51.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:51.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:51.168 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:51.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:51.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:51.213 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:51.270 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:51.307 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:51.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:51.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:51.445 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:51.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:51.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:51.547 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:51.581 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:51.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:51.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:06:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:06:51] "GET /metrics HTTP/1.1" 200 36333 "" "Prometheus/2.51.0" 2024-09-14T04:06:51.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:51.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:51.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:51.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:51.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-14T04:06:51.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:51.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:51.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:51.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:51.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:52.051 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:06:52.017+0000 7f19558bb440 -1 osd.4 178 log_to_monitors true 2024-09-14T04:06:52.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:52.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:52 smithi114 ceph-mon[106969]: pgmap v650: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 701/96942 objects degraded (0.723%); 844/96942 objects misplaced (0.871%); 44 MiB/s, 699 objects/s recovering 2024-09-14T04:06:52.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:52 smithi114 ceph-mon[106969]: from='osd.4 [v2:172.21.15.160:6800/1799506225,v1:172.21.15.160:6801/1799506225]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-14T04:06:52.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:52 smithi114 ceph-mon[106969]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-14T04:06:52.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:52 smithi114 ceph-mon[103029]: pgmap v650: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 701/96942 objects degraded (0.723%); 844/96942 objects misplaced (0.871%); 44 MiB/s, 699 objects/s recovering 2024-09-14T04:06:52.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:52 smithi114 ceph-mon[103029]: from='osd.4 [v2:172.21.15.160:6800/1799506225,v1:172.21.15.160:6801/1799506225]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-14T04:06:52.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:52 smithi114 ceph-mon[103029]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-14T04:06:52.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:52.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:52.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:52.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:52.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:52.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:06:52.369 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:52 smithi160 ceph-mon[88407]: pgmap v650: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 701/96942 objects degraded (0.723%); 844/96942 objects misplaced (0.871%); 44 MiB/s, 699 objects/s recovering 2024-09-14T04:06:52.370 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:52 smithi160 ceph-mon[88407]: from='osd.4 [v2:172.21.15.160:6800/1799506225,v1:172.21.15.160:6801/1799506225]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-14T04:06:52.370 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:52 smithi160 ceph-mon[88407]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-14T04:06:52.370 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:06:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:06:52.073+0000 7f194be64640 -1 osd.4 178 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T04:06:52.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:52.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:52.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:52.525 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:06:52.525 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:06:52.525 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-14T04:06:52.525 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:06:52.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:52.581 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:52.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:52.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:52.711 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:52.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:52.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:52.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:52.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:52.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:52.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:52.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:53.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:53.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:53.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:53 smithi160 ceph-mon[88407]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-14T04:06:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:53 smithi160 ceph-mon[88407]: osdmap e185: 8 total, 7 up, 7 in 2024-09-14T04:06:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:53 smithi160 ceph-mon[88407]: from='osd.4 [v2:172.21.15.160:6800/1799506225,v1:172.21.15.160:6801/1799506225]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-14T04:06:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:53 smithi160 ceph-mon[88407]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-14T04:06:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:53.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:53.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:53.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:53.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:53.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:53.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:53.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:53.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[103029]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-14T04:06:53.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[103029]: osdmap e185: 8 total, 7 up, 7 in 2024-09-14T04:06:53.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[103029]: from='osd.4 [v2:172.21.15.160:6800/1799506225,v1:172.21.15.160:6801/1799506225]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-14T04:06:53.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[103029]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-14T04:06:53.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:53.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:53.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[106969]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-14T04:06:53.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[106969]: osdmap e185: 8 total, 7 up, 7 in 2024-09-14T04:06:53.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[106969]: from='osd.4 [v2:172.21.15.160:6800/1799506225,v1:172.21.15.160:6801/1799506225]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-14T04:06:53.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[106969]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]} : dispatch 2024-09-14T04:06:53.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:06:53.374 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:06:53.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-14T04:06:53.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:53.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:53.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:53.523 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:53.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:53.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:53.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:53.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:53.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:53.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:53.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:53.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:53.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:53.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:53.973 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:53.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:54.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:54.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:54.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:54.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:54.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:54.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:54.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:54 smithi114 ceph-mon[103029]: pgmap v652: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 701/96942 objects degraded (0.723%); 844/96942 objects misplaced (0.871%); 38 MiB/s, 601 objects/s recovering 2024-09-14T04:06:54.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:54 smithi114 ceph-mon[103029]: osd.4 [v2:172.21.15.160:6800/1799506225,v1:172.21.15.160:6801/1799506225] boot 2024-09-14T04:06:54.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:54 smithi114 ceph-mon[103029]: osdmap e186: 8 total, 8 up, 7 in 2024-09-14T04:06:54.281 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-14T04:06:54.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:54 smithi114 ceph-mon[106969]: pgmap v652: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 701/96942 objects degraded (0.723%); 844/96942 objects misplaced (0.871%); 38 MiB/s, 601 objects/s recovering 2024-09-14T04:06:54.281 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:54 smithi114 ceph-mon[106969]: osd.4 [v2:172.21.15.160:6800/1799506225,v1:172.21.15.160:6801/1799506225] boot 2024-09-14T04:06:54.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:54 smithi114 ceph-mon[106969]: osdmap e186: 8 total, 8 up, 7 in 2024-09-14T04:06:54.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-14T04:06:54.288 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:06:54.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:54.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:54.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:54 smithi160 ceph-mon[88407]: pgmap v652: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 86 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 701/96942 objects degraded (0.723%); 844/96942 objects misplaced (0.871%); 38 MiB/s, 601 objects/s recovering 2024-09-14T04:06:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:54 smithi160 ceph-mon[88407]: osd.4 [v2:172.21.15.160:6800/1799506225,v1:172.21.15.160:6801/1799506225] boot 2024-09-14T04:06:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:54 smithi160 ceph-mon[88407]: osdmap e186: 8 total, 8 up, 7 in 2024-09-14T04:06:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-14T04:06:54.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:54.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:54.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:54.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:54.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:54.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:54.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:54.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:54.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:54.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:54.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:54.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:54.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:54.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:54.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:54.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:55.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:55.107 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:55.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:55.164 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:55 smithi114 ceph-mon[103029]: osdmap e187: 8 total, 8 up, 7 in 2024-09-14T04:06:55.164 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:55 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 701/96942 objects degraded (0.723%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:06:55.164 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:55 smithi114 ceph-mon[103029]: osdmap e188: 8 total, 8 up, 7 in 2024-09-14T04:06:55.164 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:55 smithi114 ceph-mon[106969]: osdmap e187: 8 total, 8 up, 7 in 2024-09-14T04:06:55.165 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:55 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 701/96942 objects degraded (0.723%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:06:55.165 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:55 smithi114 ceph-mon[106969]: osdmap e188: 8 total, 8 up, 7 in 2024-09-14T04:06:55.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:06:55.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:55.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:06:55.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:55.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:55.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:55 smithi160 ceph-mon[88407]: osdmap e187: 8 total, 8 up, 7 in 2024-09-14T04:06:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:55 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 701/96942 objects degraded (0.723%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:06:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:55 smithi160 ceph-mon[88407]: osdmap e188: 8 total, 8 up, 7 in 2024-09-14T04:06:55.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:55.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:55.525 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:55.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:55.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:55.671 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:55.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:55.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:55.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:55.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:55.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:55.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:55.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:55.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:56.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:56.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:56.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:56.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:06:56.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:56.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:06:56.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:56.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:56.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:56.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:56 smithi114 ceph-mon[103029]: pgmap v655: 89 pgs: 2 peering, 87 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 30 MiB/s, 475 objects/s recovering 2024-09-14T04:06:56.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:56 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 701/96942 objects degraded (0.723%), 1 pg degraded) 2024-09-14T04:06:56.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:56 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:06:56.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:56 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4223267365' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:06:56.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:56 smithi114 ceph-mon[106969]: pgmap v655: 89 pgs: 2 peering, 87 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 30 MiB/s, 475 objects/s recovering 2024-09-14T04:06:56.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:56 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 701/96942 objects degraded (0.723%), 1 pg degraded) 2024-09-14T04:06:56.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:56 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:06:56.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:56 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4223267365' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:06:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:56 smithi160 ceph-mon[88407]: pgmap v655: 89 pgs: 2 peering, 87 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 30 MiB/s, 475 objects/s recovering 2024-09-14T04:06:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:56 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 701/96942 objects degraded (0.723%), 1 pg degraded) 2024-09-14T04:06:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:56 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:06:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:56 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4223267365' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:06:56.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:56.484 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:56.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:56.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:56.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:56.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:56.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:56.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:56.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:56.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:56.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:56.886 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:56.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:56.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:57.027 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:57.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:57.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:57.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:57.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:57.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:57.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:57.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:57.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:57 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:06:57.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:57 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4223267365' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:06:57.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:57 smithi114 ceph-mon[106969]: osdmap e189: 8 total, 8 up, 7 in 2024-09-14T04:06:57.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:57 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:06:57.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:57 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4223267365' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:06:57.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:57 smithi114 ceph-mon[103029]: osdmap e189: 8 total, 8 up, 7 in 2024-09-14T04:06:57.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:57.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:57 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:06:57.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:57 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4223267365' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:06:57.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:57 smithi160 ceph-mon[88407]: osdmap e189: 8 total, 8 up, 7 in 2024-09-14T04:06:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:57.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:57.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:57.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:57.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:57.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:57.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:57.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:57.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:57.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:57.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:57.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:57.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:57.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:57.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:57.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:58.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:58.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:58.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:58.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:58.239 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:58.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:58.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:58.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:06:58 smithi114 ceph-mon[106969]: pgmap v658: 89 pgs: 2 peering, 87 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 23 MiB/s, 360 objects/s recovering 2024-09-14T04:06:58.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:06:58 smithi114 ceph-mon[103029]: pgmap v658: 89 pgs: 2 peering, 87 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 23 MiB/s, 360 objects/s recovering 2024-09-14T04:06:58.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:58.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:58.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:06:58 smithi160 ceph-mon[88407]: pgmap v658: 89 pgs: 2 peering, 87 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 23 MiB/s, 360 objects/s recovering 2024-09-14T04:06:58.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:58.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:58.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:58.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:58.645 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:58.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:58.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:06:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:58.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:58.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:06:58.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:58.913 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:58.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:59.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:06:59.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:06:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:59.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:06:59.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:06:59.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:59.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:06:59.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:59.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:59.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:59.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:06:59.454 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:59.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:06:59.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:59.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:59.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:59.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:59.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:59.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:06:59.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:06:59.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:59.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:59.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:06:59.959 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:06:59.992 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:06:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:06:59.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:00.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:00.127 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:00.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:00.162 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:07:00.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:00.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:00.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:00.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:00.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:00.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:00 smithi114 ceph-mon[103029]: pgmap v659: 89 pgs: 2 peering, 87 active+clean; 1.7 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 16 MiB/s, 256 objects/s recovering 2024-09-14T04:07:00.382 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:00 smithi114 ceph-mon[106969]: pgmap v659: 89 pgs: 2 peering, 87 active+clean; 1.7 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 16 MiB/s, 256 objects/s recovering 2024-09-14T04:07:00.404 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:00.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:00 smithi160 ceph-mon[88407]: pgmap v659: 89 pgs: 2 peering, 87 active+clean; 1.7 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 16 MiB/s, 256 objects/s recovering 2024-09-14T04:07:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:07:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:07:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:07:00.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:00.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:00.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:00.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:00.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:00.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:00.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:00.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:00.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:00.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:00.915 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:00.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:00.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:01.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:01.097 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:01.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:01.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:01.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:01.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:01.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:01.374 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:01.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:01.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:01.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:01.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:01.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:01.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:01.651 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:01.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:01.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:01.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:01.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:01.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:01.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:07:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:07:01] "GET /metrics HTTP/1.1" 200 36330 "" "Prometheus/2.51.0" 2024-09-14T04:07:01.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:01.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:01.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:02.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:02.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:02.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:02.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:02.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:02.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:02.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:02.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:02.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:02.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:02.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:02.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:02 smithi114 ceph-mon[106969]: pgmap v660: 89 pgs: 89 active+clean; 1.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 14 MiB/s, 217 objects/s recovering 2024-09-14T04:07:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:02 smithi114 ceph-mon[103029]: pgmap v660: 89 pgs: 89 active+clean; 1.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 14 MiB/s, 217 objects/s recovering 2024-09-14T04:07:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:02 smithi160 ceph-mon[88407]: pgmap v660: 89 pgs: 89 active+clean; 1.5 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 14 MiB/s, 217 objects/s recovering 2024-09-14T04:07:02.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:02.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:02.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:02.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:02.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:02.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:02.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:02.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:02.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:02.770 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:02.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:02.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:02.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:02.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:02.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:02.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:07:03.005 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:03.035 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:03.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:03.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:03.137 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:03.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:03.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:03.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:03.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:03.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:03.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:03.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:03.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:03.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:03.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:03.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:03.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:03.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:03.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:03.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:03.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:03.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:03.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:04.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:04.083 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:04.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:04.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:04.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:04.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:04.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:04.352 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:04.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:04.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:04 smithi160 ceph-mon[88407]: pgmap v661: 89 pgs: 89 active+clean; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 500 op/s 2024-09-14T04:07:04.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:04.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:04.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:04.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:04 smithi114 ceph-mon[106969]: pgmap v661: 89 pgs: 89 active+clean; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 500 op/s 2024-09-14T04:07:04.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:04 smithi114 ceph-mon[103029]: pgmap v661: 89 pgs: 89 active+clean; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 500 op/s 2024-09-14T04:07:04.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:04.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:04.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:04.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:04.771 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:04.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:04.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:04.890 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:07:04.890 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:07:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:04.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:05.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:05.070 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:05.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:05.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:05.210 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:05.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:05.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:05.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:05.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:05.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:05.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:05 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3118127347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:05.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:05 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3118127347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:05 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3118127347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:05.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:05.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:05.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:05.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:05.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:05.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:05.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:05.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:05.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:05.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:05.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:05.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:05.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:05.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:06.020 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:06.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:06.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:06.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:06.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:06.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:06.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:06.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:06.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:06 smithi160 ceph-mon[88407]: pgmap v662: 89 pgs: 89 active+clean; 1.2 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 923 op/s 2024-09-14T04:07:06.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:06.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:06.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:06.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:06.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:06 smithi114 ceph-mon[103029]: pgmap v662: 89 pgs: 89 active+clean; 1.2 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 923 op/s 2024-09-14T04:07:06.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:06 smithi114 ceph-mon[106969]: pgmap v662: 89 pgs: 89 active+clean; 1.2 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 923 op/s 2024-09-14T04:07:06.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:06.693 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:06.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:06.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-14T04:07:06.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:06.855 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:07:06.883 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:07:06.883 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:07:06.883 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:07:06.883 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:07:06.883 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:07:06.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:07:06.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:07:06.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:07:06.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:07:06.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:07:06.884 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:07:06.884 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:07:06.885 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:07:06.885 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:07:06.885 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:07:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:06.886 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:06.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:06.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:07.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:07.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:07.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:07.163 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:07.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:07.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:07.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:07.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:07.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:07.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:07.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:07.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:07.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:07.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:07.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:07.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:07.726 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:07.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:07.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:07.861 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:07.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:07.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:07.993 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:07.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:07.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:08.017 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:08.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:08.096 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:08.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:08.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:08.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:08.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:08.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:08.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:08.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[106969]: pgmap v663: 89 pgs: 89 active+clean; 1.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 839 op/s 2024-09-14T04:07:08.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:08.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.412 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[103029]: pgmap v663: 89 pgs: 89 active+clean; 1.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 839 op/s 2024-09-14T04:07:08.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:08.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:08.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:08.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:08.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:08.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:08 smithi160 ceph-mon[88407]: pgmap v663: 89 pgs: 89 active+clean; 1.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 839 op/s 2024-09-14T04:07:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:08.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:08.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:08.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:08.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:08.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:08.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:08.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:08.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:08.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:09.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:09.102 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:09.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:09.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:09.204 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:09.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:09.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:09.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:09.364 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:09.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:09.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:07:09.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:07:09.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:09.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:07:09.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:07:09.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:09.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:09.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:09.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:09.589 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:09.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:09.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:09.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:07:09.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:07:09.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:09.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:07:09.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:09.777 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:09.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:09.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:09.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:09.933 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:09.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:10.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:10.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:10.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:10.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:10.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:10.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:10.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:10.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:10.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:10.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:10.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:10.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:10.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:10.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:10.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:10 smithi114 ceph-mon[103029]: pgmap v664: 89 pgs: 89 active+clean; 1.1 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-09-14T04:07:10.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:10 smithi114 ceph-mon[106969]: pgmap v664: 89 pgs: 89 active+clean; 1.1 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-09-14T04:07:10.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:10.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:07:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:07:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:07:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:10 smithi160 ceph-mon[88407]: pgmap v664: 89 pgs: 89 active+clean; 1.1 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-09-14T04:07:10.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:10.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:10.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:10.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:10.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:10.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:11.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:11.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:11.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:11.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:11.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:11.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:11.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:11.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:11.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:11.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:11.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:11.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:11.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:11.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:11.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:11.520 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:07:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:07:11] "GET /metrics HTTP/1.1" 200 36454 "" "Prometheus/2.51.0" 2024-09-14T04:07:11.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:11.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:11.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:11.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:11.759 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:11.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:11.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:11.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:11.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:11.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:11.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:12.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:12.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:12.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:12.164 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:12.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:12.220 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:12 smithi114 ceph-mon[106969]: pgmap v665: 89 pgs: 89 active+clean; 920 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T04:07:12.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:12 smithi114 ceph-mon[103029]: pgmap v665: 89 pgs: 89 active+clean; 920 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T04:07:12.235 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:07:12.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:12.293 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:12.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:12.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:12.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:12.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:12.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:12.511 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:12 smithi160 ceph-mon[88407]: pgmap v665: 89 pgs: 89 active+clean; 920 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-14T04:07:12.512 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:12 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1031868861' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:07:12.512 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:12 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:07:12.512 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:12 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1836938487' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:07:12.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:12.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:12.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:12.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:12.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:12 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1031868861' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:07:12.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:12 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:07:12.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:12 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1836938487' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:07:12.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:12 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1031868861' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:07:12.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:12 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:07:12.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:12 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1836938487' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:07:12.622 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-14T04:07:12.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:12.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:12.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:12.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:12.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:12.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:12.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:12.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:12.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:13.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:13.089 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:13.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:13.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:13.229 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:13.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:13.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:13.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:13.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:13.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:13.505 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:13.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:13.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:13.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:13 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:07:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:13 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:07:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:13 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1836938487' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:07:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:13 smithi114 ceph-mon[106969]: osdmap e190: 8 total, 8 up, 7 in 2024-09-14T04:07:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:13 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:07:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:13 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:07:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:13 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1836938487' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:07:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:13 smithi114 ceph-mon[103029]: osdmap e190: 8 total, 8 up, 7 in 2024-09-14T04:07:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:13.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:13.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:13 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:07:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:13 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:07:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:13 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1836938487' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:07:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:13 smithi160 ceph-mon[88407]: osdmap e190: 8 total, 8 up, 7 in 2024-09-14T04:07:13.755 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:13.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:13.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:13.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:13.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:13.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:14.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:14.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:14.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:14.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:14.192 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:14.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:14.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:14.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:14.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:14.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:14.460 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:14.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:14.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:14.595 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:14.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:14.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:14 smithi114 ceph-mon[103029]: pgmap v667: 89 pgs: 89 active+clean; 920 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-09-14T04:07:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:14 smithi114 ceph-mon[106969]: pgmap v667: 89 pgs: 89 active+clean; 920 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-09-14T04:07:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:14 smithi160 ceph-mon[88407]: pgmap v667: 89 pgs: 89 active+clean; 920 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 976 op/s 2024-09-14T04:07:14.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:14.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:14.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:14.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-14T04:07:14.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:14.874 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:14.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:14.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:15.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:15.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:15.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:15.174 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:15.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:15.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:15.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:15.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:15.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:15.451 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:15.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:15.553 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:15.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:15.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:15.588 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:15.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:15.737 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:15.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:15.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:15.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:15.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:15.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:16.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:16.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:16.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:16.151 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:16.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:16.169 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:16.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:07:16.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:16.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:16.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:16.403 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:16 smithi114 ceph-mon[106969]: pgmap v668: 89 pgs: 89 active+clean; 625 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 959 op/s 2024-09-14T04:07:16.403 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:16 smithi114 ceph-mon[103029]: pgmap v668: 89 pgs: 89 active+clean; 625 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 959 op/s 2024-09-14T04:07:16.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:16.476 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:16.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:16.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:16.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:16 smithi160 ceph-mon[88407]: pgmap v668: 89 pgs: 89 active+clean; 625 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 959 op/s 2024-09-14T04:07:16.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:16.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:16.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:16.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:16.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:16.913 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:16.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:17.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:17.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:17.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:17.137 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:07:17.137 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:07:17.137 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:07:17.137 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:07:17.137 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:07:17.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:17.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:17.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:17.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:17.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:17.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:17.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:17.457 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:17.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:17.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:17.597 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:17.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:17.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:17.731 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:17.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:17.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:17.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:17.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:17.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:18.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:18.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:18.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:18.137 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:18.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:18.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:18.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:18 smithi114 ceph-mon[103029]: pgmap v669: 89 pgs: 89 active+clean; 625 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 959 op/s 2024-09-14T04:07:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:18.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:18.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:18.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:18.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:18.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:18.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:18.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:18.575 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:18 smithi114 ceph-mon[106969]: pgmap v669: 89 pgs: 89 active+clean; 625 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 959 op/s 2024-09-14T04:07:18.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:18 smithi160 ceph-mon[88407]: pgmap v669: 89 pgs: 89 active+clean; 625 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 959 op/s 2024-09-14T04:07:18.685 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:18.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:18.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:18.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:18.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:18.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:18.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:18.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:18.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:18.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:18.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:19.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:19.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:19.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:19.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:19.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:19.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:19.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:19.362 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:19.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:19.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:19.504 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:19.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:19.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:19.633 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:19.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:19.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:19.771 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:19.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:19.872 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:19.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:19.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:20.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:20.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:20.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:20.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:20.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:20.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:20.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:20.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:20.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:20.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:20.446 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:20.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:20.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:20.564 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:20 smithi114 ceph-mon[103029]: pgmap v670: 89 pgs: 89 active+clean; 544 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-14T04:07:20.564 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:20 smithi114 ceph-mon[106969]: pgmap v670: 89 pgs: 89 active+clean; 544 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-14T04:07:20.582 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:20.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:20.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:07:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:07:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:07:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:20 smithi160 ceph-mon[88407]: pgmap v670: 89 pgs: 89 active+clean; 544 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 972 op/s 2024-09-14T04:07:20.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:20.723 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:20.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:20.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:20.846 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:20.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:20.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:20.923 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:07:20.924 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:07:20.959 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:07:20.959 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:07:20.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:20.991 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:07:20.991 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:07:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:21.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:21.024 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:07:21.024 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:07:21.052 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:07:21.052 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:07:21.084 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:07:21.085 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:07:21.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:21.121 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:07:21.121 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:07:21.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:21.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:21.150 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:07:21.151 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:07:21.248 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:21.282 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:21.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:21.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:21.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/229453948' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:21.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/229453948' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:21.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:21.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-14T04:07:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:21.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:21.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:21.586 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:21.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/229453948' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:21.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:21.722 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:07:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:07:21] "GET /metrics HTTP/1.1" 200 36454 "" "Prometheus/2.51.0" 2024-09-14T04:07:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:21.733 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:21.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:21.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:21.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:21.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:21.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:22.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:22.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:22.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:22.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:22.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:22.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:22.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:22.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:22.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:22 smithi114 ceph-mon[106969]: pgmap v671: 89 pgs: 89 active+clean; 303 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 986 op/s 2024-09-14T04:07:22.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:22 smithi114 ceph-mon[103029]: pgmap v671: 89 pgs: 89 active+clean; 303 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 986 op/s 2024-09-14T04:07:22.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:22.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:22.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:22.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:22.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:22.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:22.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:22 smithi160 ceph-mon[88407]: pgmap v671: 89 pgs: 89 active+clean; 303 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 986 op/s 2024-09-14T04:07:22.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:22.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:22.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:22.814 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:22.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:22.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:22.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:22.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:22.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:23.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:23.082 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:23.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:23.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:23.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:23.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:23.259 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-14T04:07:23.309 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:23.336 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:23.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:23.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:23.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:23.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:23.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:23.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:23.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:23.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:23.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:23.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:23.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4123372448' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:07:23.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:23.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:23 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4123372448' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:07:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:23 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4123372448' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:07:23.710 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 129458 objects 2024-09-14T04:07:23.710 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :154.141 2024-09-14T04:07:23.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:23.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:23.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:23.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:23.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:23.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:23.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:24.005 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:24.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:24.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:24.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:24.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:24.147 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:24.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:24.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:24.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:24.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:24.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:24.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:24.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:24.404 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:24.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:24.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:24.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:24.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:24.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:24.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:24 smithi114 ceph-mon[103029]: pgmap v672: 89 pgs: 89 active+clean; 303 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 941 op/s 2024-09-14T04:07:24.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:24 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4123372448' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:07:24.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:24 smithi114 ceph-mon[103029]: osdmap e191: 8 total, 8 up, 7 in 2024-09-14T04:07:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:24 smithi114 ceph-mon[106969]: pgmap v672: 89 pgs: 89 active+clean; 303 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 941 op/s 2024-09-14T04:07:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:24 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4123372448' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:07:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:24 smithi114 ceph-mon[106969]: osdmap e191: 8 total, 8 up, 7 in 2024-09-14T04:07:24.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:24.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:24.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:24 smithi160 ceph-mon[88407]: pgmap v672: 89 pgs: 89 active+clean; 303 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 941 op/s 2024-09-14T04:07:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:24 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4123372448' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:07:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:24 smithi160 ceph-mon[88407]: osdmap e191: 8 total, 8 up, 7 in 2024-09-14T04:07:24.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:24.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:24.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:24.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:24.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:24.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:24.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:24.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:25.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:25.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:25.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:25.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:25.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:25.155 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:25.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:25.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:25.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:25.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:25.283 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:07:25.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:25.406 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:25.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:25.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:25.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:25.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:25.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:25.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3752131908' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:07:25.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:25 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:07:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3752131908' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:07:25.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:25 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:07:25.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:25.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:25.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3752131908' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:07:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:25 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:07:25.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:25.776 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:25.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:25.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:25.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:25.904 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:25.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:26.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:26.029 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:26.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:26.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:26.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:26.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:26.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:26.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:26.279 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:26.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:26.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:26.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:26.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:26.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:26.533 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:26.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:26.581 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:26 smithi114 ceph-mon[103029]: pgmap v674: 89 pgs: 89 active+clean; 15 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-09-14T04:07:26.581 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:26 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:07:26.581 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:26 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:07:26.581 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:26 smithi114 ceph-mon[103029]: osdmap e192: 8 total, 8 up, 7 in 2024-09-14T04:07:26.582 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:26 smithi114 ceph-mon[106969]: pgmap v674: 89 pgs: 89 active+clean; 15 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-09-14T04:07:26.582 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:26 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:07:26.582 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:26 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:07:26.582 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:26 smithi114 ceph-mon[106969]: osdmap e192: 8 total, 8 up, 7 in 2024-09-14T04:07:26.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-14T04:07:26.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:26 smithi160 ceph-mon[88407]: pgmap v674: 89 pgs: 89 active+clean; 15 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 975 op/s 2024-09-14T04:07:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:26 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:07:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:26 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:07:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:26 smithi160 ceph-mon[88407]: osdmap e192: 8 total, 8 up, 7 in 2024-09-14T04:07:26.661 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:26.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:26.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:07:26.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:26.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:26.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:26.886 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:26.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:26.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:26.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:27.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:27.035 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:27.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:27.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:27.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:27.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:27.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:27.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:27.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:27.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:27.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:27.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:27.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:27.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:27.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:27.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:27.666 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:27.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:27.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:27.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:27.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:27.895 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:27.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:27.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:28.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:28.046 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:28.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:28.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:28.172 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:28.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:28.219 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:28.219 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:28.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:28.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:28.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:28.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:28.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:28.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:28.419 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:28.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:28.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:28.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:28.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:28.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:28.600 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:28 smithi114 ceph-mon[103029]: pgmap v676: 89 pgs: 89 active+clean; 15 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-14T04:07:28.600 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:28 smithi114 ceph-mon[106969]: pgmap v676: 89 pgs: 89 active+clean; 15 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-14T04:07:28.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:28 smithi160 ceph-mon[88407]: pgmap v676: 89 pgs: 89 active+clean; 15 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-14T04:07:28.658 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:07:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=cleanup t=2024-09-14T04:07:28.194288686Z level=info msg="Completed cleanup jobs" duration=13.388637ms 2024-09-14T04:07:28.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:28.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:28.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:28.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:28.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:28.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:28.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:28.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:29.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:29.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:29.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:29.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:29.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:29.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:29.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:29.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:29.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:29.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:29.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:29.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:29.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:29.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:29.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:07:29.514 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-14T04:07:29.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-14T04:07:29.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:29.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:29.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:29.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:29.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:29.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:29.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:29.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:29.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:29.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:29.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:29.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:29.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:30.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:30.085 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:30.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:30.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:30.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:30.212 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:30.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:30.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:30.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:30.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:30.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:30 smithi114 ceph-mon[103029]: pgmap v677: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 604 op/s 2024-09-14T04:07:30.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:30 smithi114 ceph-mon[106969]: pgmap v677: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 604 op/s 2024-09-14T04:07:30.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:30.459 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:30.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:30.470 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:30 smithi160 ceph-mon[88407]: pgmap v677: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 604 op/s 2024-09-14T04:07:30.471 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:07:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:07:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:07:30.561 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:30.587 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:30.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:30.593 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:30.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:30.710 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:30.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:30.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:30.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:30.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:30.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:30.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:30.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:31.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:31.079 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:31.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:31.181 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:31.202 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:31.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:31.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:31.326 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:31.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:31.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:31.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:31.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:31.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:31.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:31.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:31.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:31.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:07:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:07:31] "GET /metrics HTTP/1.1" 200 36451 "" "Prometheus/2.51.0" 2024-09-14T04:07:31.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:31.694 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:31.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:31.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:31.817 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:31.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:31.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:31.935 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:31.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:31.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:32.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:32.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:32.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:32.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:32.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:32.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:32.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:32.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:32.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:32.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:32.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:32.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:32.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:32.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:32.559 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:32.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:32.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:32 smithi114 ceph-mon[103029]: pgmap v678: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 604 op/s 2024-09-14T04:07:32.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:32 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2454451445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:07:32.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:32 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:07:32.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:32 smithi114 ceph-mon[106969]: pgmap v678: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 604 op/s 2024-09-14T04:07:32.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:32 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2454451445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:07:32.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:32 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:07:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:32 smithi160 ceph-mon[88407]: pgmap v678: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 604 op/s 2024-09-14T04:07:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:32 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2454451445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:07:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:32 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:07:32.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:32.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:32.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:32.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:32.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:32.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:32.831 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:07:32.831 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:07:32.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:32.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:32.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:32.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:33.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:33.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:33.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:33.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:33.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:33.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:33.255 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-14T04:07:33.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:33.300 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:33.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:33.402 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:33.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:33.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:33.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:33 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:07:33.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:33 smithi114 ceph-mon[106969]: osdmap e193: 8 total, 8 up, 7 in 2024-09-14T04:07:33.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:33 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:07:33.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:33 smithi114 ceph-mon[103029]: osdmap e193: 8 total, 8 up, 7 in 2024-09-14T04:07:33.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:33.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:33.548 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:33.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:33.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:33.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:33 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:07:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:33 smithi160 ceph-mon[88407]: osdmap e193: 8 total, 8 up, 7 in 2024-09-14T04:07:33.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:33.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:33.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:33.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:33.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:33.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:33.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:33.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:33.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:33.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:33.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-14T04:07:34.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:34.043 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:34.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:34.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:34.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:34.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:34.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:34.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:34.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:34.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:34.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:34.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:34.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:34.537 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:34.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:34.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:34 smithi114 ceph-mon[103029]: pgmap v680: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 29 op/s 2024-09-14T04:07:34.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:34 smithi114 ceph-mon[106969]: pgmap v680: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 29 op/s 2024-09-14T04:07:34.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:34 smithi160 ceph-mon[88407]: pgmap v680: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 29 op/s 2024-09-14T04:07:34.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:34.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:34.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:34.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:34.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:34.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:34.908 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:34.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:35.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:35.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:35.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:35.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:35.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:35.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:35.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:35.281 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:35.281 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:07:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:07:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:35.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:35.298 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:07:35.298 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:07:35.315 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:07:35.315 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:07:35.331 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:07:35.331 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:07:35.347 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:07:35.347 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:07:35.363 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:07:35.363 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:07:35.379 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:07:35.379 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:07:35.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:35.396 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:07:35.396 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:07:35.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:35.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:35.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:35.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:35.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:35.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:35 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/464701483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:35.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:35 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/464701483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:35.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:35 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/464701483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:35.659 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:35.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:35.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:35.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:35.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:35.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:35.883 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:35.910 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:35.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:36.012 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:36.037 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:36.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:36.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:36.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:36.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:36.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:36.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:36.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:36.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:36.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:36.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:36.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:36.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:36.426 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:36 smithi114 ceph-mon[106969]: pgmap v681: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 25 op/s 2024-09-14T04:07:36.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:36 smithi114 ceph-mon[103029]: pgmap v681: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 25 op/s 2024-09-14T04:07:36.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:36.538 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:36.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:36.639 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:36 smithi160 ceph-mon[88407]: pgmap v681: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 25 op/s 2024-09-14T04:07:36.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:36.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:36.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:36.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:36.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:36.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:36.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:36.917 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:36.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:37.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:37.046 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:37.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:37.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:37.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:37.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:37.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:37.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:37.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:37.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:37.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:37.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:37.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:37.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:37.553 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:37.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:37.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:37.677 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:37.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:37.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:37.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:37.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:37.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:37.930 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:37.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:37.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-14T04:07:38.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:38.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:38.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:38.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:38.184 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:38.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:38.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:38.308 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:38.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:38.378 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:38.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:38.431 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:38.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:38.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:38 smithi114 ceph-mon[106969]: pgmap v682: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 23 op/s 2024-09-14T04:07:38.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:38.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:38 smithi114 ceph-mon[103029]: pgmap v682: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 23 op/s 2024-09-14T04:07:38.524 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:38.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:38.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:38.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:38 smithi160 ceph-mon[88407]: pgmap v682: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 23 op/s 2024-09-14T04:07:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:38.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:38.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:38.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:38.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:38.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:38.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:38.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:38.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:38.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:39.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:39.060 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:39.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:39.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:07:39.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:39.174 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:39.187 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:39.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:39.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:39.314 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:07:39.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:39.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:39.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:39.323 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_4' removed 2024-09-14T04:07:39.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:39.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:39.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:39.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:39.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:39.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:39.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3060362807' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:07:39.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:39 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:07:39.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2347790162' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:07:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3060362807' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:07:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:39 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:07:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2347790162' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:07:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3060362807' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:07:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:39 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:07:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2347790162' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:07:39.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:39.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:39.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:39.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:39.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:39.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:39.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:39.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:39.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:40.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:40.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:40.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:40.166 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:40.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:40.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:40.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:40.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:40.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:40.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:40.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:40.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:40.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:40.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:40.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[103029]: pgmap v683: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:07:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:07:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:07:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2347790162' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:07:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[103029]: osdmap e194: 8 total, 8 up, 7 in 2024-09-14T04:07:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[106969]: pgmap v683: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:40.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:07:40.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:07:40.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:07:40.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2347790162' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:07:40.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:40 smithi114 ceph-mon[106969]: osdmap e194: 8 total, 8 up, 7 in 2024-09-14T04:07:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:07:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:07:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:07:40.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:40 smithi160 ceph-mon[88407]: pgmap v683: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:40.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:40 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:07:40.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:40 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:07:40.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:40 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:07:40.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2347790162' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:07:40.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:40 smithi160 ceph-mon[88407]: osdmap e194: 8 total, 8 up, 7 in 2024-09-14T04:07:40.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:40.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:40.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:40.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:40.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:40.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:40.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:40.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:40.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:41.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:41.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:41.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:41.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:41.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:41.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:41.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:41.300 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:41.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:41.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:41.358 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-14T04:07:41.359 INFO:tasks.radosbench:Beginning radosbench... 2024-09-14T04:07:41.359 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-14T04:07:41.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-09-14T04:07:41.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:41.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:41.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:41.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:41.549 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:41.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:41.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:07:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:07:41] "GET /metrics HTTP/1.1" 200 36443 "" "Prometheus/2.51.0" 2024-09-14T04:07:41.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:41.673 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:41.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:41.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:41.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:41.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:41.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:41.919 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:41.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:42.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:42.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:42.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:42.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:42.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:42.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:42.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:42.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:42.288 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:42.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:42.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:42.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:42.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:42.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:42.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:42.540 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:42.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:42.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:42 smithi114 ceph-mon[103029]: pgmap v685: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:42.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:42 smithi114 ceph-mon[106969]: pgmap v685: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:42.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:42 smithi160 ceph-mon[88407]: pgmap v685: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:42.668 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:42.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:42.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:42.832 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:42.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:42.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:07:42.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:42.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:42.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:42.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:43.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:43.085 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:43.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:43.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:43.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:43.212 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:43.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:43.257 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:07:43.257 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:07:43.257 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-14T04:07:43.257 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:07:43.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:43.324 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-14T04:07:43.348 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:43.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:43.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:43.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:43.476 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:43.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:43.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:43.587 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:43 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3302615601' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:07:43.587 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:43 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:07:43.587 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:43 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3302615601' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:07:43.587 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:43 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:07:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:43.600 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:43.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:43 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3302615601' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:07:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:43 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:07:43.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:43.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:43.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:43.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:43.850 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:43.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:43.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-14T04:07:43.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:43.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:43.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:44.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:44.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:44.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:44.174 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:44.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:44.232 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:44.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:44.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:44.359 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:44.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:44.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:44.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:44.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:44.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:44.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:44.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:44.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:44 smithi114 ceph-mon[103029]: pgmap v686: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:44 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:07:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:44 smithi114 ceph-mon[103029]: osdmap e195: 8 total, 8 up, 7 in 2024-09-14T04:07:44.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:44 smithi114 ceph-mon[106969]: pgmap v686: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:44 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:07:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:44 smithi114 ceph-mon[106969]: osdmap e195: 8 total, 8 up, 7 in 2024-09-14T04:07:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:44 smithi160 ceph-mon[88407]: pgmap v686: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:44 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:07:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:44 smithi160 ceph-mon[88407]: osdmap e195: 8 total, 8 up, 7 in 2024-09-14T04:07:44.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-14T04:07:44.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:44.740 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:44.764 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:44.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:44.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:44.886 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:44.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:44.980 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:44.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:45.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:45.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:45.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:45.138 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:45.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:45.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:45.263 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:45.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:45.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:45.386 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:45.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:45.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:45.509 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:45.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:45.591 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:45 smithi114 ceph-mon[103029]: osdmap e196: 8 total, 8 up, 7 in 2024-09-14T04:07:45.591 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:45 smithi114 ceph-mon[106969]: osdmap e196: 8 total, 8 up, 7 in 2024-09-14T04:07:45.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:45.637 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:45.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:45.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:45 smithi160 ceph-mon[88407]: osdmap e196: 8 total, 8 up, 7 in 2024-09-14T04:07:45.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:45.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:45.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:45.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:45.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:45.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:45.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:45.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:45.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:46.013 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:46.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:46.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:46.140 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:46.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:46.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:46.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:46.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:46.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:46.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:46.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:46.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:46.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:46.519 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:46.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:46.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:46 smithi114 ceph-mon[106969]: pgmap v689: 73 pgs: 11 peering, 62 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:46.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:46 smithi114 ceph-mon[106969]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-14T04:07:46.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:46 smithi114 ceph-mon[103029]: pgmap v689: 73 pgs: 11 peering, 62 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:46.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:46 smithi114 ceph-mon[103029]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-14T04:07:46.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:46.651 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:46.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:46.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:46 smithi160 ceph-mon[88407]: pgmap v689: 73 pgs: 11 peering, 62 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:46 smithi160 ceph-mon[88407]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-14T04:07:46.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:46.775 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:46.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:46.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:46.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:46.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:47.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:47.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:47.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:47.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:47.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:47.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:47.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:47.277 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:47.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:47.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:47.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:47.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:47.407 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:47.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:47.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:47.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:47.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:47.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:47.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:47.758 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:47.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:47.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:47.834 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:47.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:47.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:47.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:48.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:48.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:48.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:48.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:48.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:48.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:48.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:48.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:48.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:48.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:48.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:48.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:48.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:48.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:48.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:48.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:48.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:48.591 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:48 smithi114 ceph-mon[103029]: pgmap v690: 73 pgs: 11 peering, 62 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:48.591 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:48 smithi114 ceph-mon[106969]: pgmap v690: 73 pgs: 11 peering, 62 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:48.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:48.651 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:48.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:48 smithi160 ceph-mon[88407]: pgmap v690: 73 pgs: 11 peering, 62 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:48.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:48.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:48.775 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:48.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:48.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:48.907 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:48.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:49.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:49.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:49.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:49.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:49.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:49.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:49.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:49.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:49.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:49.407 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:49.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:49.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:49.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:49.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:49.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:49.659 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:49.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:49.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:49.786 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:49.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:49.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:49.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:49.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:49.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:50.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:50.037 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:50.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:50.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:50.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:50.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:50.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:50.284 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:50.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:50.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:50 smithi114 ceph-mon[103029]: pgmap v691: 73 pgs: 11 peering, 62 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:50.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:50 smithi114 ceph-mon[106969]: pgmap v691: 73 pgs: 11 peering, 62 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:50.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:50.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:50.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:50.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:50.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:50.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:50.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:50.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:50 smithi160 ceph-mon[88407]: pgmap v691: 73 pgs: 11 peering, 62 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:50.659 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:07:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:07:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:07:50.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:50.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:50.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:50.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:50.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:50.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:50.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:50.905 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:50.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:51.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:51.033 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:51.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:51.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:51.160 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:51.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:51.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:51.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:51.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:51.353 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:07:51.353 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:07:51.353 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:07:51.353 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:07:51.353 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:07:51.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:51.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:51.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:51.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:51.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:51.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:51.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:51.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/76368890' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:07:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:51 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:07:51.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:07:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:07:51] "GET /metrics HTTP/1.1" 200 36443 "" "Prometheus/2.51.0" 2024-09-14T04:07:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:51 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/76368890' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:07:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:51 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:07:51.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:51.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:51 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/76368890' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:07:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:51 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:07:51.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:51.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:51.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:51.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:51.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:51.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:51.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:51.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:51.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:52.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:52.037 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:52.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:52.137 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:52.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:52.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:52.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:52.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:52.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:52.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:52.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:52.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:52.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:52.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:52.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:52.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:52.553 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:07:52.553 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:07:52.568 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:07:52.573 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:07:52.584 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:07:52.584 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:07:52.600 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:07:52.601 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:07:52.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[106969]: pgmap v692: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:52.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[106969]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-14T04:07:52.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:07:52.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:07:52.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:07:52.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[106969]: osdmap e197: 8 total, 8 up, 7 in 2024-09-14T04:07:52.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[103029]: pgmap v692: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:52.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[103029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-14T04:07:52.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:07:52.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:07:52.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:07:52.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:52 smithi114 ceph-mon[103029]: osdmap e197: 8 total, 8 up, 7 in 2024-09-14T04:07:52.616 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:07:52.617 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:07:52.633 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:07:52.633 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:07:52.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:52.648 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:07:52.649 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:07:52.654 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:07:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:52 smithi160 ceph-mon[88407]: pgmap v692: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:52 smithi160 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-14T04:07:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:52 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:07:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:52 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:07:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:52 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:07:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:52 smithi160 ceph-mon[88407]: osdmap e197: 8 total, 8 up, 7 in 2024-09-14T04:07:52.665 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:07:52.665 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:07:52.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:52.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:52.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:52.771 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:52.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:52.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:52.895 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:52.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:52.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:53.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:53.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:53.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:53.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:53.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:53.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:53.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:53.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:53.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:53.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:53.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:53.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:53.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:53.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]} : dispatch 2024-09-14T04:07:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-14T04:07:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]} : dispatch 2024-09-14T04:07:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4042826217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:53.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]} : dispatch 2024-09-14T04:07:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-14T04:07:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]} : dispatch 2024-09-14T04:07:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4042826217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:53.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]} : dispatch 2024-09-14T04:07:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]} : dispatch 2024-09-14T04:07:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]} : dispatch 2024-09-14T04:07:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4042826217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:07:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:07:53.674 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:53.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:53.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:53.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:53.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:53.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:53.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:53.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:53.931 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:53.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:54.033 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:54.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:54.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:54.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:54.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:54.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:54.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:54.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:54.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:54.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:54.356 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_5' created 2024-09-14T04:07:54.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-14T04:07:54.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:54.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:54.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:54.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:54.564 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:54.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:54.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[106969]: pgmap v694: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]}]': finished 2024-09-14T04:07:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-14T04:07:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]}]': finished 2024-09-14T04:07:54.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[106969]: osdmap e198: 8 total, 8 up, 7 in 2024-09-14T04:07:54.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2174502428' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-14T04:07:54.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-14T04:07:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[103029]: pgmap v694: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]}]': finished 2024-09-14T04:07:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-14T04:07:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]}]': finished 2024-09-14T04:07:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[103029]: osdmap e198: 8 total, 8 up, 7 in 2024-09-14T04:07:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2174502428' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-14T04:07:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:54 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-14T04:07:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:54 smithi160 ceph-mon[88407]: pgmap v694: 73 pgs: 73 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]}]': finished 2024-09-14T04:07:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-14T04:07:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 2]}]': finished 2024-09-14T04:07:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:54 smithi160 ceph-mon[88407]: osdmap e198: 8 total, 8 up, 7 in 2024-09-14T04:07:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:54 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2174502428' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-14T04:07:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:54 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-14T04:07:54.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:54.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:54.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:54.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:54.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:54.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:54.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:54.935 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:54.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:54.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:55.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:55.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:55.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:55.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:55.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:55.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:55.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:55.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:55.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:55.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:55.434 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:55.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:55.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:55.557 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:55.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:55.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:55.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:55 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-14T04:07:55.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:55 smithi114 ceph-mon[103029]: osdmap e199: 8 total, 8 up, 7 in 2024-09-14T04:07:55.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:55 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-14T04:07:55.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:55 smithi114 ceph-mon[106969]: osdmap e199: 8 total, 8 up, 7 in 2024-09-14T04:07:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:55 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-14T04:07:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:55 smithi160 ceph-mon[88407]: osdmap e199: 8 total, 8 up, 7 in 2024-09-14T04:07:55.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:55.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:55.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:55.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-14T04:07:55.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:55.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:55.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:55.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:55.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:55.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:55.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:56.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:56.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:56.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:56.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:56.193 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:56.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:56.203 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:56.240 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:56.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:56.322 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:56.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:56.365 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-14T04:07:56.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:56.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:56.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:56.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:56 smithi114 ceph-mon[106969]: pgmap v697: 89 pgs: 1 creating+peering, 15 unknown, 1 peering, 72 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:56.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:56 smithi114 ceph-mon[106969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:07:56.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:56 smithi114 ceph-mon[106969]: osdmap e200: 8 total, 8 up, 7 in 2024-09-14T04:07:56.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:56 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4066956572' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:07:56.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:56 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:07:56.456 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:56 smithi114 ceph-mon[103029]: pgmap v697: 89 pgs: 1 creating+peering, 15 unknown, 1 peering, 72 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:56.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:56 smithi114 ceph-mon[103029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:07:56.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:56 smithi114 ceph-mon[103029]: osdmap e200: 8 total, 8 up, 7 in 2024-09-14T04:07:56.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:56 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4066956572' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:07:56.457 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:56 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:07:56.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:56.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:56.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:56 smithi160 ceph-mon[88407]: pgmap v697: 89 pgs: 1 creating+peering, 15 unknown, 1 peering, 72 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:56 smithi160 ceph-mon[88407]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:07:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:56 smithi160 ceph-mon[88407]: osdmap e200: 8 total, 8 up, 7 in 2024-09-14T04:07:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:56 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4066956572' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:07:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:56 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:07:56.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:56.702 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:56.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:56.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:56.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:56.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:56.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:56.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:56.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:56.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:07:57.054 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:57.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:57.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:57.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-14T04:07:57.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:57.210 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:57.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:57.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:57.337 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:57.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:57.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:57.460 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:57.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:57.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:57.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:57.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:57.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:57.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:57 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:07:57.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:57 smithi114 ceph-mon[103029]: osdmap e201: 8 total, 8 up, 7 in 2024-09-14T04:07:57.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:57 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:07:57.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:57 smithi114 ceph-mon[106969]: osdmap e201: 8 total, 8 up, 7 in 2024-09-14T04:07:57.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:57 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:07:57.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:57 smithi160 ceph-mon[88407]: osdmap e201: 8 total, 8 up, 7 in 2024-09-14T04:07:57.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:57.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:57.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:57.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:57.830 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:57.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:57.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:07:57.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:57.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:07:58.054 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:58.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:58.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:07:58.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:58.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:07:58.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:58.205 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:58.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:58.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:58.329 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:58.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:58.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:07:58.458 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:58.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:07:58.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:58.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:58.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:58.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:58 smithi114 ceph-mon[103029]: pgmap v700: 89 pgs: 1 creating+peering, 13 unknown, 1 peering, 74 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:58.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:07:58 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:58.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:58 smithi114 ceph-mon[106969]: pgmap v700: 89 pgs: 1 creating+peering, 13 unknown, 1 peering, 74 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:58.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:07:58 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:58 smithi160 ceph-mon[88407]: pgmap v700: 89 pgs: 1 creating+peering, 13 unknown, 1 peering, 74 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:07:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:07:58 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:07:58.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:58.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:58.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:58.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:58.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:58.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:58.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:58.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:58.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:58.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:59.031 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:07:59.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:59.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:59.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:59.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:59.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:59.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:59.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:07:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:59.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:59.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:07:59.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:07:59.459 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:59.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:07:59.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:07:59.589 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:59.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:07:59.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:07:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:59.716 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:07:59.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:07:59.817 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:07:59.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:59.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:07:59.945 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:07:59.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:07:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:07:59.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:00.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:00.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:00.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:00.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:00.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:00.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:00.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:00.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:00.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:00.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:00.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:00.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:00.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:00.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:00.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:00.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:00.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:00.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:00.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:00.600 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:00 smithi160 ceph-mon[88407]: pgmap v701: 89 pgs: 1 creating+peering, 11 unknown, 1 peering, 76 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:08:00.659 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:08:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:08:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:08:00.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:00.701 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:00 smithi114 ceph-mon[103029]: pgmap v701: 89 pgs: 1 creating+peering, 11 unknown, 1 peering, 76 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:08:00.701 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:00 smithi114 ceph-mon[106969]: pgmap v701: 89 pgs: 1 creating+peering, 11 unknown, 1 peering, 76 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:08:00.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:00.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:00.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:00.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:00.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:00.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:00.973 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:00.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:01.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:01.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:01.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:01.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:01.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:01.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:01.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:01.352 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:01.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:01.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:01.377 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-14T04:08:01.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:01.479 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:01.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:01.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:01.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:01.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:01.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:01.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:01 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3584691114' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:08:01.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:01 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3584691114' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:08:01.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:08:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:08:01] "GET /metrics HTTP/1.1" 200 34080 "" "Prometheus/2.51.0" 2024-09-14T04:08:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:01 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3584691114' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:08:01.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:01.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:01.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:01.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:01.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:01.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:01.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:01.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:01.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:01.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:02.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:02.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:02.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:02.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:02.234 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:02.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:02.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:02.360 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:02.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:02.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:02 smithi114 ceph-mon[103029]: pgmap v702: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:08:02.414 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:02 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3584691114' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:08:02.414 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:02 smithi114 ceph-mon[103029]: osdmap e202: 8 total, 8 up, 7 in 2024-09-14T04:08:02.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:02 smithi114 ceph-mon[106969]: pgmap v702: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:08:02.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:02 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3584691114' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:08:02.414 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:02 smithi114 ceph-mon[106969]: osdmap e202: 8 total, 8 up, 7 in 2024-09-14T04:08:02.414 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:08:02.415 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:08:02.421 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:02.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:02.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:02.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:02.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:02.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:02.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:02.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:02 smithi160 ceph-mon[88407]: pgmap v702: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:08:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:02 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3584691114' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:08:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:02 smithi160 ceph-mon[88407]: osdmap e202: 8 total, 8 up, 7 in 2024-09-14T04:08:02.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:02.736 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:02.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:02.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:02.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:02.860 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:02.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:02.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:02.984 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:02.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:03.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:03.107 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:03.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:03.177 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-14T04:08:03.179 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-14T04:08:03.179 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-14T04:08:03.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:03.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-14T04:08:03.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-14T04:08:03.230 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_188657 2024-09-14T04:08:03.230 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:08:03.230 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-14T04:08:03.234 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:03.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:03.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:03.369 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:03.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:03.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:03.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:03.503 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:03.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:03.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-14T04:08:03.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:03.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:03.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:03 smithi114 ceph-mon[103029]: pgmap v704: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:08:03.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:03 smithi114 ceph-mon[103029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:08:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:03 smithi114 ceph-mon[106969]: pgmap v704: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:08:03.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:03 smithi114 ceph-mon[106969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:08:03.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:03.790 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:03.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:03.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:03.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:03 smithi160 ceph-mon[88407]: pgmap v704: 89 pgs: 89 active+clean; 582 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-14T04:08:03.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:03 smithi160 ceph-mon[88407]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:08:03.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:03.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:03.942 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:04.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:04.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:04.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:04.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:04.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:04.204 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:04.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:04.226 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 3671 3655 228.425 228.438 0.00358981 0.00436465 2024-09-14T04:08:04.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:04.338 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:04.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:04.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:04.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:04.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:04.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:04.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:04.615 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:04.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:04.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:04.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:04.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:04.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:04.885 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:04.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:04.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:05.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:05.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:05.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:05.157 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:05.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:05.227 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 7246 7230 225.921 223.438 0.00210475 0.00441827 2024-09-14T04:08:05.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:05.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:05.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:05.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:05.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:05.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:05.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:05.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:05.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:05.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:05.689 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:05.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:05.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:05.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:05.812 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:05.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:05.866 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:05 smithi114 ceph-mon[106969]: pgmap v705: 89 pgs: 89 active+clean; 122 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 15 MiB/s wr, 233 op/s 2024-09-14T04:08:05.867 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:05 smithi114 ceph-mon[103029]: pgmap v705: 89 pgs: 89 active+clean; 122 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 15 MiB/s wr, 233 op/s 2024-09-14T04:08:05.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:05.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:05.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:06.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:06.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:06.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:05 smithi160 ceph-mon[88407]: pgmap v705: 89 pgs: 89 active+clean; 122 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 15 MiB/s wr, 233 op/s 2024-09-14T04:08:06.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:06.227 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 9913 9897 206.17 166.688 0.00442385 0.00484177 2024-09-14T04:08:06.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:06.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:06.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:06.372 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:06.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:06.474 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:06.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:06.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:06.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:06.642 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:06.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:06.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:06.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:06.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:06.789 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-14T04:08:06.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:06.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3407342987' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:08:06.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:06 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:08:06.923 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3407342987' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:08:06.923 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:06 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:08:06.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:06.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:06.977 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:08:06.977 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:08:07.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:07.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:07.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:07.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3407342987' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:08:07.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:06 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:08:07.169 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:07.202 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:07.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:07.227 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 12909 12893 201.436 187.25 0.00548838 0.00495818 2024-09-14T04:08:07.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:07.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:07.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:07.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:07.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:07.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:07.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:07.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-14T04:08:07.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:07.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:07.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:07.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-14T04:08:07.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:07.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:07.811 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:08:07.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:07.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:07.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:07.915 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:08:07.915 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:08:07.948 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:08:07.948 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:08:07.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:07.980 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:08:07.980 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:08:08.012 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:08.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:08.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:08.014 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:08:08.014 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:08:08.045 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:08:08.045 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:08:08.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[103029]: pgmap v706: 89 pgs: 89 active+clean; 204 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 20 MiB/s wr, 325 op/s 2024-09-14T04:08:08.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:08:08.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[103029]: osdmap e203: 8 total, 8 up, 7 in 2024-09-14T04:08:08.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/847692866' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:08.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:08.046 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:08.046 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[106969]: pgmap v706: 89 pgs: 89 active+clean; 204 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 20 MiB/s wr, 325 op/s 2024-09-14T04:08:08.046 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:08:08.046 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[106969]: osdmap e203: 8 total, 8 up, 7 in 2024-09-14T04:08:08.046 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/847692866' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:08.046 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:08.047 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:08.079 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:08:08.079 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:08:08.106 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:08.114 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:08:08.114 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:08:08.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:08.148 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:08:08.149 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:08:08.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:08.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:07 smithi160 ceph-mon[88407]: pgmap v706: 89 pgs: 89 active+clean; 204 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 20 MiB/s wr, 325 op/s 2024-09-14T04:08:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:07 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:08:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:07 smithi160 ceph-mon[88407]: osdmap e203: 8 total, 8 up, 7 in 2024-09-14T04:08:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:07 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/847692866' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:07 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:08.227 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 15589 15573 194.645 167.5 0.00620892 0.0051318 2024-09-14T04:08:08.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:08.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:08.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:08.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:08.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:08.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:08.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:08.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:08.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:08.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:08.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:08.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:08.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:08.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:08.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:08.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:08.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:08 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:08:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:08 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:08:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:08 smithi160 ceph-mon[88407]: osdmap e204: 8 total, 8 up, 7 in 2024-09-14T04:08:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3639559593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:08.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:08.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:09.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:09.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:09.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:09.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:08 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:08:09.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:08 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:08:09.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:08 smithi114 ceph-mon[106969]: osdmap e204: 8 total, 8 up, 7 in 2024-09-14T04:08:09.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:09.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3639559593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:09.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:08 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:08:09.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:08 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:08:09.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:08 smithi114 ceph-mon[103029]: osdmap e204: 8 total, 8 up, 7 in 2024-09-14T04:08:09.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:09.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3639559593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:09.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:09.227 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 18266 18250 190.087 167.312 0.00624813 0.00525541 2024-09-14T04:08:09.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:09.259 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:09.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:09.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:09.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:09.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:09.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:09.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:09.536 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:09.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:09.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:09.679 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:09.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:09.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:09.826 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:09.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:09.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:09.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:09.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:10.018 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:09 smithi160 ceph-mon[88407]: pgmap v709: 89 pgs: 89 active+clean; 315 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 43 MiB/s wr, 684 op/s 2024-09-14T04:08:10.018 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:08:10.019 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.019 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.019 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.019 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.047 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:10.070 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:08:10.071 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:08:10.071 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:08:10.071 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:08:10.071 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:08:10.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:10.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:10.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:10.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[103029]: pgmap v709: 89 pgs: 89 active+clean; 315 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 43 MiB/s wr, 684 op/s 2024-09-14T04:08:10.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:08:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[106969]: pgmap v709: 89 pgs: 89 active+clean; 315 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 43 MiB/s wr, 684 op/s 2024-09-14T04:08:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:08:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:10.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:10.228 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 20664 20648 184.321 149.875 0.00731782 0.00541878 2024-09-14T04:08:10.258 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:10.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:10.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:10.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:08:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:08:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:08:10.410 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:10.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:10.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:10.554 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:10.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:10.655 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:10.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:10.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:10.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:10.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:10.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:10.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:10.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:11.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:11.095 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:11.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:11.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:08:11.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:11.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:11.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:11.228 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 23148 23132 180.686 155.25 0.00599123 0.00552863 2024-09-14T04:08:11.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:11.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:11.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:11.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:11.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:11.492 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:11.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:11.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:11.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:11.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:08:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:08:11] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-14T04:08:11.623 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:11.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:11.725 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:11.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:11.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:11.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:11.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:11.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:11.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:12.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:12.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:12.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:08:12.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:11 smithi114 ceph-mon[103029]: pgmap v710: 89 pgs: 89 active+clean; 1017 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-14T04:08:12.081 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:11 smithi114 ceph-mon[106969]: pgmap v710: 89 pgs: 89 active+clean; 1017 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-14T04:08:12.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:12.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:12.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:12.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:11 smithi160 ceph-mon[88407]: pgmap v710: 89 pgs: 89 active+clean; 1017 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-09-14T04:08:12.228 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 25087 25071 174.074 121.188 0.00498617 0.00573814 2024-09-14T04:08:12.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:12.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:12.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:12.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-14T04:08:12.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:12.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:12.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:12.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:12.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:12.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:12.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:12.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:12.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:12.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:12.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:12.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:12.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:12.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:12.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:12.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:12.991 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:12.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:13.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:13.138 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:13.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:13.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 27336 27320 170.713 140.562 0.00206809 0.00585131 2024-09-14T04:08:13.240 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:13.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:13.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:13.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:13.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:13.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:13.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:13.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:13.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:13.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:13.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:13.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:13.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:13.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:13.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:13.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:13.962 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:13 smithi114 ceph-mon[106969]: pgmap v711: 89 pgs: 89 active+clean; 1.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-14T04:08:13.963 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:13 smithi114 ceph-mon[103029]: pgmap v711: 89 pgs: 89 active+clean; 1.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-14T04:08:13.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:13.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:14.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:14.134 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:14.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:14.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:13 smithi160 ceph-mon[88407]: pgmap v711: 89 pgs: 89 active+clean; 1.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-14T04:08:14.199 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:14.213 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:14.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 29128 29112 165.375 112 0.0159821 0.00604031 2024-09-14T04:08:14.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:14.305 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:14.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:14.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:14.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:14.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:14.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:14.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:14.591 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:14.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:14.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:14.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:14.742 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:14.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:14.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:14.870 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:14.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:14.873 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:14 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2471387281' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:08:14.889 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-14T04:08:14.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:14.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:14.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:15.007 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:14 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2471387281' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:08:15.008 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:14 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2471387281' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:08:15.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:15.139 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:15.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:15.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 30837 30821 160.493 106.812 0.00560748 0.0062246 2024-09-14T04:08:15.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:15.270 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:15.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:15.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:15.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:15.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:15.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:15.537 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:15.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:15.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:15.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:15.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:15.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:15.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:15.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:15.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:15.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:15.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:16.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:16.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:16.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:15 smithi160 ceph-mon[88407]: pgmap v712: 89 pgs: 89 active+clean; 1.7 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 190 MiB/s wr, 3.04k op/s 2024-09-14T04:08:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2471387281' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:08:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:15 smithi160 ceph-mon[88407]: osdmap e205: 8 total, 8 up, 7 in 2024-09-14T04:08:16.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:16.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:15 smithi114 ceph-mon[106969]: pgmap v712: 89 pgs: 89 active+clean; 1.7 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 190 MiB/s wr, 3.04k op/s 2024-09-14T04:08:16.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2471387281' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:08:16.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:15 smithi114 ceph-mon[106969]: osdmap e205: 8 total, 8 up, 7 in 2024-09-14T04:08:16.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:15 smithi114 ceph-mon[103029]: pgmap v712: 89 pgs: 89 active+clean; 1.7 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 190 MiB/s wr, 3.04k op/s 2024-09-14T04:08:16.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2471387281' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:08:16.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:15 smithi114 ceph-mon[103029]: osdmap e205: 8 total, 8 up, 7 in 2024-09-14T04:08:16.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:16.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:16.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 32742 32726 157.306 119.062 0.00536281 0.00635166 2024-09-14T04:08:16.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:16.348 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:16.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:16.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:16.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:16.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:16.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:16.609 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:16.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:16.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:16.739 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:16.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:16.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:16.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:16.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:16.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:17.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:17.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:17.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:17.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:17.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:17.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 34473 34457 153.797 108.188 0.00606429 0.00649608 2024-09-14T04:08:17.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:17.277 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:17.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:17.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:17.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:17.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:17.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:17.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:17.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:17.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:17.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:17.665 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:08:17.665 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:08:17.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:08:17.665 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:08:17.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.666 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:96273 8.8 8:1df25a9c:::benchmark_data_smithi114_135745_object96272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-14T04:08:17.666 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:58:44.762236+0000", 2024-09-14T04:08:17.666 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.898369301, 2024-09-14T04:08:17.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3075659540000002, 2024-09-14T04:08:17.666 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.666 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:08:17.666 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.666 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:08:17.666 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:08:17.667 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 96273 2024-09-14T04:08:17.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.667 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.762236+0000", 2024-09-14T04:08:17.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.762236+0000", 2024-09-14T04:08:17.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.762235+0000", 2024-09-14T04:08:17.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:17.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.762267+0000", 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2759000000000003e-05 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.762285+0000", 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7770000000000001e-05 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.762291+0000", 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7869999999999998e-06 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.762530+0000", 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023915899999999999 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.762627+0000", 2024-09-14T04:08:17.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7243000000000004e-05 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 7", 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.762742+0000", 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011458700000000001 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.767496+0000", 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0047539899999999996 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.789712+0000", 2024-09-14T04:08:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.022215763999999999 2024-09-14T04:08:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.789723+0000", 2024-09-14T04:08:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0829000000000001e-05 2024-09-14T04:08:17.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775020+0000", 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98529696 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775031+0000", 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.147e-05 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.069708+0000", 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.294676707 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.069802+0000", 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4487999999999994e-05 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.674 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:96291 8.8 8:1999e514:::benchmark_data_smithi114_135745_object96290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:58:44.783231+0000", 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.87737439600005, 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3101303780000002, 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 96291 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.675 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.783231+0000", 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.783231+0000", 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.783230+0000", 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.784790+0000", 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015596690000000001 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.784801+0000", 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1147e-05 2024-09-14T04:08:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.784807+0000", 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9460000000000003e-06 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.789774+0000", 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049671159999999997 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.789781+0000", 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775062+0000", 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.985280767 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775069+0000", 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0480000000000001e-06 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.070966+0000", 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.295897122 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.071109+0000", 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014324499999999999 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:08:17.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.071267+0000", 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000157804 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.083855+0000", 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012588389 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.085923+0000", 2024-09-14T04:08:17.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020681580000000001 2024-09-14T04:08:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.092032+0000", 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061084809999999998 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.092082+0000", 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9755000000000001e-05 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.093362+0000", 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001279896 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:96295 8.8 8:109ab187:::benchmark_data_smithi114_135745_object96294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:58:44.785604+0000", 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.87500119399999, 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3088485209999998, 2024-09-14T04:08:17.684 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 96295 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.785604+0000", 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.785604+0000", 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.785603+0000", 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.785659+0000", 2024-09-14T04:08:17.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.575e-05 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.785669+0000", 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0162e-05 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.785675+0000", 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7529999999999998e-06 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.789808+0000", 2024-09-14T04:08:17.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041336350000000001 2024-09-14T04:08:17.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.789824+0000", 2024-09-14T04:08:17.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5553000000000002e-05 2024-09-14T04:08:17.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775254+0000", 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98543005699999997 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775263+0000", 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5930000000000006e-06 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.074855+0000", 2024-09-14T04:08:17.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2995920830000001 2024-09-14T04:08:17.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.075006+0000", 2024-09-14T04:08:17.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015100399999999999 2024-09-14T04:08:17.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:08:17.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.075169+0000", 2024-09-14T04:08:17.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016295799999999999 2024-09-14T04:08:17.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.083904+0000", 2024-09-14T04:08:17.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0087355680000000008 2024-09-14T04:08:17.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.085930+0000", 2024-09-14T04:08:17.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002025804 2024-09-14T04:08:17.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.093414+0000", 2024-09-14T04:08:17.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0074842529999999997 2024-09-14T04:08:17.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.093471+0000", 2024-09-14T04:08:17.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6817999999999997e-05 2024-09-14T04:08:17.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.094453+0000", 2024-09-14T04:08:17.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0009817370000000001 2024-09-14T04:08:17.694 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.694 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.695 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:96313 8.8 8:14cb781a:::benchmark_data_smithi114_135745_object96312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-14T04:08:17.695 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:58:44.820745+0000", 2024-09-14T04:08:17.695 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 572.83986060799998, 2024-09-14T04:08:17.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2749012629999998, 2024-09-14T04:08:17.695 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.695 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:08:17.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:08:17.696 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 96313 2024-09-14T04:08:17.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.696 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.820745+0000", 2024-09-14T04:08:17.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.820745+0000", 2024-09-14T04:08:17.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.820743+0000", 2024-09-14T04:08:17.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:17.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.820804+0000", 2024-09-14T04:08:17.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0476000000000002e-05 2024-09-14T04:08:17.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.820820+0000", 2024-09-14T04:08:17.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6123000000000001e-05 2024-09-14T04:08:17.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.820827+0000", 2024-09-14T04:08:17.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9469999999999997e-06 2024-09-14T04:08:17.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.820869+0000", 2024-09-14T04:08:17.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2511e-05 2024-09-14T04:08:17.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:44.820876+0000", 2024-09-14T04:08:17.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1140000000000002e-06 2024-09-14T04:08:17.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775303+0000", 2024-09-14T04:08:17.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95442786599999996 2024-09-14T04:08:17.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775311+0000", 2024-09-14T04:08:17.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9379999999999992e-06 2024-09-14T04:08:17.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.079852+0000", 2024-09-14T04:08:17.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3045408250000001 2024-09-14T04:08:17.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.080019+0000", 2024-09-14T04:08:17.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016644399999999999 2024-09-14T04:08:17.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:08:17.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.080192+0000", 2024-09-14T04:08:17.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000173363 2024-09-14T04:08:17.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.087128+0000", 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0069359690000000002 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.087182+0000", 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3720000000000001e-05 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.094670+0000", 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0074881260000000003 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.094718+0000", 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7735000000000003e-05 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.095646+0000", 2024-09-14T04:08:17.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00092869299999999997 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:96281 8.0 8:0ff15a85:::benchmark_data_smithi114_135745_object96280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:58:45.775795+0000", 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 571.88481082299995, 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.316139416, 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:08:17.707 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 96281 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775795+0000", 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775795+0000", 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775794+0000", 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775882+0000", 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7942000000000006e-05 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775893+0000", 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0547000000000001e-05 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775898+0000", 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1109999999999999e-06 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775929+0000", 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1106000000000003e-05 2024-09-14T04:08:17.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775933+0000", 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0609999999999997e-06 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.059185+0000", 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2832525480000001 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for degraded object", 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.059984+0000", 2024-09-14T04:08:17.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00079864499999999995 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.091733+0000", 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.031748816999999999 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.091934+0000", 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020118100000000001 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45086.0:96307 8.0 8:0d75c986:::benchmark_data_smithi114_135745_object96306:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T03:58:45.775905+0000", 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 571.88470031400004, 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.316800266, 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45086", 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/879913483", 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 96307 2024-09-14T04:08:17.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775905+0000", 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775905+0000", 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775905+0000", 2024-09-14T04:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775961+0000", 2024-09-14T04:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6473e-05 2024-09-14T04:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775970+0000", 2024-09-14T04:08:17.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6230000000000003e-06 2024-09-14T04:08:17.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.775974+0000", 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5390000000000001e-06 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.776002+0000", 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8036999999999999e-05 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:45.776009+0000", 2024-09-14T04:08:17.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9779999999999999e-06 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.060030+0000", 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2840211690000001 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.060178+0000", 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014808000000000001 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-14T04:08:17.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.060365+0000", 2024-09-14T04:08:17.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018694499999999999 2024-09-14T04:08:17.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.070940+0000", 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010574414000000001 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.091888+0000", 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020948494000000002 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.091922+0000", 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3837999999999997e-05 2024-09-14T04:08:17.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.091985+0000", 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3318000000000006e-05 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T03:58:47.092705+0000", 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000720004 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220880 9.f 9:f96ede54:::benchmark_data_smithi114_153893_object91419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.196403+0000", 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.464202221999997, 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86833886999999998, 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220880 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196403+0000", 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196403+0000", 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196401+0000", 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:08:17.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196409+0000", 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.131e-06 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196425+0000", 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6512e-05 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196431+0000", 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2910000000000004e-06 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.197080+0000", 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00064894099999999999 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.197087+0000", 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9630000000000001e-06 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.044872+0000", 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84778513200000005 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.044981+0000", 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108917 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.045069+0000", 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7620000000000005e-05 2024-09-14T04:08:17.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.055218+0000", 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010148634 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058484+0000", 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032667180000000001 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064649+0000", 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061650619999999998 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064698+0000", 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8887000000000001e-05 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064742+0000", 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3894999999999999e-05 2024-09-14T04:08:17.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220882 9.f 9:f2f60c4f:::benchmark_data_smithi114_153893_object91421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.201420+0000", 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.459185484000002, 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86631311799999999, 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.729 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220882 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201420+0000", 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201420+0000", 2024-09-14T04:08:17.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201418+0000", 2024-09-14T04:08:17.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:08:17.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201427+0000", 2024-09-14T04:08:17.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.721e-06 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201440+0000", 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2972e-05 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201446+0000", 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5430000000000001e-06 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201484+0000", 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8325000000000003e-05 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201492+0000", 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6620000000000005e-06 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.048696+0000", 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84720400699999998 2024-09-14T04:08:17.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.048765+0000", 2024-09-14T04:08:17.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9080999999999998e-05 2024-09-14T04:08:17.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.048846+0000", 2024-09-14T04:08:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:17.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:17.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:17.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1446999999999995e-05 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058553+0000", 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0097070750000000008 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058613+0000", 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0253000000000001e-05 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067623+0000", 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090100319999999994 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067687+0000", 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3064999999999993e-05 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067733+0000", 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6666999999999999e-05 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.738 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.739 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.739 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220900 9.f 9:ff716adf:::benchmark_data_smithi114_153893_object91439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.739 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.216323+0000", 2024-09-14T04:08:17.739 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.444282762, 2024-09-14T04:08:17.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88042596299999998, 2024-09-14T04:08:17.739 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.739 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220900 2024-09-14T04:08:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216323+0000", 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216323+0000", 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216322+0000", 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216328+0000", 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9449999999999998e-06 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216337+0000", 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6059999999999992e-06 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216343+0000", 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5729999999999998e-06 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216588+0000", 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024550400000000001 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216594+0000", 2024-09-14T04:08:17.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1920000000000002e-06 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.051961+0000", 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83536656399999998 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.052033+0000", 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1588999999999995e-05 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.052105+0000", 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2805999999999994e-05 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.061647+0000", 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0095416719999999993 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067789+0000", 2024-09-14T04:08:17.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061421050000000001 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096653+0000", 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.028863903999999999 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096700+0000", 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6773000000000001e-05 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096749+0000", 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8934999999999998e-05 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220903 9.f 9:f4af0834:::benchmark_data_smithi114_153893_object91442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.217818+0000", 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.442787062999997, 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87909124900000002, 2024-09-14T04:08:17.747 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220903 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217818+0000", 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217818+0000", 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217818+0000", 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217823+0000", 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3249999999999998e-06 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217833+0000", 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0246e-05 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217839+0000", 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6069999999999998e-06 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217875+0000", 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5756999999999997e-05 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217881+0000", 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5629999999999997e-06 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.055244+0000", 2024-09-14T04:08:17.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83736241899999997 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.055357+0000", 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011300500000000001 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.055477+0000", 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119938 2024-09-14T04:08:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064752+0000", 2024-09-14T04:08:17.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0092754980000000001 2024-09-14T04:08:17.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.087734+0000", 2024-09-14T04:08:17.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.022982473 2024-09-14T04:08:17.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096831+0000", 2024-09-14T04:08:17.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090970070000000007 2024-09-14T04:08:17.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096881+0000", 2024-09-14T04:08:17.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9044e-05 2024-09-14T04:08:17.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096910+0000", 2024-09-14T04:08:17.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9207e-05 2024-09-14T04:08:17.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.757 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.757 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.757 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220908 9.f 9:f2004091:::benchmark_data_smithi114_153893_object91447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.757 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.222577+0000", 2024-09-14T04:08:17.758 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.438028837000005, 2024-09-14T04:08:17.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87448498699999999, 2024-09-14T04:08:17.758 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.758 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.758 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.758 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220908 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222577+0000", 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222577+0000", 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222575+0000", 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222581+0000", 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9739999999999999e-06 2024-09-14T04:08:17.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222595+0000", 2024-09-14T04:08:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3331e-05 2024-09-14T04:08:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222601+0000", 2024-09-14T04:08:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4980000000000003e-06 2024-09-14T04:08:17.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222652+0000", 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0711000000000003e-05 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222661+0000", 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8449999999999992e-06 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058645+0000", 2024-09-14T04:08:17.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83598437800000003 2024-09-14T04:08:17.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058734+0000", 2024-09-14T04:08:17.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9221000000000005e-05 2024-09-14T04:08:17.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058834+0000", 2024-09-14T04:08:17.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9858000000000005e-05 2024-09-14T04:08:17.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067744+0000", 2024-09-14T04:08:17.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0089099409999999993 2024-09-14T04:08:17.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.087771+0000", 2024-09-14T04:08:17.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020027117000000001 2024-09-14T04:08:17.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096969+0000", 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0091981110000000001 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097015+0000", 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5170999999999998e-05 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097062+0000", 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7061000000000002e-05 2024-09-14T04:08:17.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.768 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.768 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220919 9.f 9:f2584fce:::benchmark_data_smithi114_153893_object91458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.768 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.231868+0000", 2024-09-14T04:08:17.768 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.428737460999997, 2024-09-14T04:08:17.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.873098869, 2024-09-14T04:08:17.769 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.769 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.769 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.769 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.769 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.769 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220919 2024-09-14T04:08:17.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.770 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231868+0000", 2024-09-14T04:08:17.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231868+0000", 2024-09-14T04:08:17.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231867+0000", 2024-09-14T04:08:17.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:08:17.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231871+0000", 2024-09-14T04:08:17.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3699999999999997e-06 2024-09-14T04:08:17.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231879+0000", 2024-09-14T04:08:17.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9780000000000005e-06 2024-09-14T04:08:17.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231883+0000", 2024-09-14T04:08:17.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0609999999999997e-06 2024-09-14T04:08:17.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.234088+0000", 2024-09-14T04:08:17.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002204559 2024-09-14T04:08:17.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.234095+0000", 2024-09-14T04:08:17.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0679999999999999e-06 2024-09-14T04:08:17.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.061684+0000", 2024-09-14T04:08:17.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.827589456 2024-09-14T04:08:17.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.061769+0000", 2024-09-14T04:08:17.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4770000000000003e-05 2024-09-14T04:08:17.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.061840+0000", 2024-09-14T04:08:17.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.046e-05 2024-09-14T04:08:17.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067748+0000", 2024-09-14T04:08:17.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059083110000000003 2024-09-14T04:08:17.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097092+0000", 2024-09-14T04:08:17.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029343899999999999 2024-09-14T04:08:17.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.104874+0000", 2024-09-14T04:08:17.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0077821729999999999 2024-09-14T04:08:17.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.104924+0000", 2024-09-14T04:08:17.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0223999999999999e-05 2024-09-14T04:08:17.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.104967+0000", 2024-09-14T04:08:17.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2629000000000001e-05 2024-09-14T04:08:17.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.785 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.786 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220936 9.f 9:f88a3439:::benchmark_data_smithi114_153893_object91475:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.786 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.252191+0000", 2024-09-14T04:08:17.786 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.408414457999996, 2024-09-14T04:08:17.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85289873000000005, 2024-09-14T04:08:17.786 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.787 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.787 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.787 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.787 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.787 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220936 2024-09-14T04:08:17.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.787 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252191+0000", 2024-09-14T04:08:17.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252191+0000", 2024-09-14T04:08:17.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252189+0000", 2024-09-14T04:08:17.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:08:17.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252196+0000", 2024-09-14T04:08:17.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4999999999999996e-06 2024-09-14T04:08:17.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252209+0000", 2024-09-14T04:08:17.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3634e-05 2024-09-14T04:08:17.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252215+0000", 2024-09-14T04:08:17.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9569999999999999e-06 2024-09-14T04:08:17.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252764+0000", 2024-09-14T04:08:17.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00054909399999999997 2024-09-14T04:08:17.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252773+0000", 2024-09-14T04:08:17.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1589999999999993e-06 2024-09-14T04:08:17.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064770+0000", 2024-09-14T04:08:17.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81199766200000001 2024-09-14T04:08:17.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064844+0000", 2024-09-14T04:08:17.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3677000000000006e-05 2024-09-14T04:08:17.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064910+0000", 2024-09-14T04:08:17.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6523e-05 2024-09-14T04:08:17.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.075180+0000", 2024-09-14T04:08:17.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010269967 2024-09-14T04:08:17.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097135+0000", 2024-09-14T04:08:17.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021954497 2024-09-14T04:08:17.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.105017+0000", 2024-09-14T04:08:17.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0078818959999999993 2024-09-14T04:08:17.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.105058+0000", 2024-09-14T04:08:17.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0750000000000001e-05 2024-09-14T04:08:17.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.105090+0000", 2024-09-14T04:08:17.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2270000000000001e-05 2024-09-14T04:08:17.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.801 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.802 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220943 9.f 9:f08a27fb:::benchmark_data_smithi114_153893_object91482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.802 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.260673+0000", 2024-09-14T04:08:17.802 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.399932480000004, 2024-09-14T04:08:17.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85680367899999998, 2024-09-14T04:08:17.802 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.803 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.803 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.803 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.803 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.803 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220943 2024-09-14T04:08:17.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.804 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260673+0000", 2024-09-14T04:08:17.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260673+0000", 2024-09-14T04:08:17.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260671+0000", 2024-09-14T04:08:17.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:08:17.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260678+0000", 2024-09-14T04:08:17.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9519999999999997e-06 2024-09-14T04:08:17.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260694+0000", 2024-09-14T04:08:17.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6223e-05 2024-09-14T04:08:17.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260700+0000", 2024-09-14T04:08:17.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9699999999999996e-06 2024-09-14T04:08:17.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260765+0000", 2024-09-14T04:08:17.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4989999999999999e-05 2024-09-14T04:08:17.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.810 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.810 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260773+0000", 2024-09-14T04:08:17.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7840000000000004e-06 2024-09-14T04:08:17.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067803+0000", 2024-09-14T04:08:17.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.807030256 2024-09-14T04:08:17.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067880+0000", 2024-09-14T04:08:17.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6699999999999994e-05 2024-09-14T04:08:17.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067970+0000", 2024-09-14T04:08:17.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9563999999999998e-05 2024-09-14T04:08:17.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.083713+0000", 2024-09-14T04:08:17.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015743183000000001 2024-09-14T04:08:17.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097183+0000", 2024-09-14T04:08:17.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013470355999999999 2024-09-14T04:08:17.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117401+0000", 2024-09-14T04:08:17.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020217386 2024-09-14T04:08:17.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117442+0000", 2024-09-14T04:08:17.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1891999999999998e-05 2024-09-14T04:08:17.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117477+0000", 2024-09-14T04:08:17.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4264999999999998e-05 2024-09-14T04:08:17.817 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.818 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.818 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.818 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220947 9.f 9:fe0d007e:::benchmark_data_smithi114_153893_object91486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.819 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.263647+0000", 2024-09-14T04:08:17.819 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.396958161000001, 2024-09-14T04:08:17.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85404936600000003, 2024-09-14T04:08:17.819 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.819 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.819 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.820 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.820 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.820 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220947 2024-09-14T04:08:17.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.820 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263647+0000", 2024-09-14T04:08:17.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263647+0000", 2024-09-14T04:08:17.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263646+0000", 2024-09-14T04:08:17.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:08:17.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263652+0000", 2024-09-14T04:08:17.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4670000000000003e-06 2024-09-14T04:08:17.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.824 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.824 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263662+0000", 2024-09-14T04:08:17.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6139999999999998e-06 2024-09-14T04:08:17.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.825 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263667+0000", 2024-09-14T04:08:17.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4859999999999997e-06 2024-09-14T04:08:17.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263714+0000", 2024-09-14T04:08:17.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7447000000000002e-05 2024-09-14T04:08:17.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263721+0000", 2024-09-14T04:08:17.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.387e-06 2024-09-14T04:08:17.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.071744+0000", 2024-09-14T04:08:17.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80802356200000003 2024-09-14T04:08:17.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.071828+0000", 2024-09-14T04:08:17.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3418999999999995e-05 2024-09-14T04:08:17.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.829 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.071899+0000", 2024-09-14T04:08:17.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0732e-05 2024-09-14T04:08:17.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096601+0000", 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.024702867 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097216+0000", 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000614085 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117634+0000", 2024-09-14T04:08:17.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020418546999999999 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117668+0000", 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3683999999999998e-05 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117697+0000", 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8969e-05 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220953 9.f 9:fc60803b:::benchmark_data_smithi114_153893_object91492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.270585+0000", 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.390020946000007, 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84728676199999997, 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220953 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.832 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270585+0000", 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270585+0000", 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270584+0000", 2024-09-14T04:08:17.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270589+0000", 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7080000000000002e-06 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270605+0000", 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5421e-05 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270611+0000", 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8029999999999999e-06 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270701+0000", 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9697999999999994e-05 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270708+0000", 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1269999999999997e-06 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.075221+0000", 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80451222499999997 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.075322+0000", 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101952 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.075437+0000", 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011485900000000001 2024-09-14T04:08:17.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096610+0000", 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021172291999999999 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097251+0000", 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00064100699999999997 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117821+0000", 2024-09-14T04:08:17.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020570082999999999 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117844+0000", 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2796e-05 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117871+0000", 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7840000000000001e-05 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.839 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220961 9.f 9:f1118824:::benchmark_data_smithi114_153893_object91500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.839 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.280922+0000", 2024-09-14T04:08:17.839 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.379683071000002, 2024-09-14T04:08:17.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83707822399999998, 2024-09-14T04:08:17.839 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.839 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.839 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.839 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220961 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280922+0000", 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280922+0000", 2024-09-14T04:08:17.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280921+0000", 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280927+0000", 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7919999999999998e-06 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280938+0000", 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1375e-05 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280945+0000", 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.652e-06 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.281000+0000", 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5179999999999997e-05 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.281009+0000", 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8030000000000003e-06 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.079375+0000", 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.79836644999999995 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.079493+0000", 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117242 2024-09-14T04:08:17.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.079610+0000", 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117735 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096761+0000", 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017151079999999999 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097273+0000", 2024-09-14T04:08:17.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00051192499999999999 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117917+0000", 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020643812000000001 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117962+0000", 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4338999999999997e-05 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.118001+0000", 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9165999999999997e-05 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220963 9.f 9:fa967625:::benchmark_data_smithi114_153893_object91502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.283564+0000", 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.377041930000004, 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83455748200000002, 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.846 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220963 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283564+0000", 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283564+0000", 2024-09-14T04:08:17.847 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283563+0000", 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283572+0000", 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3070000000000008e-06 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283582+0000", 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4790000000000002e-06 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283586+0000", 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1899999999999997e-06 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283615+0000", 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8847e-05 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.850 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283621+0000", 2024-09-14T04:08:17.850 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2149999999999998e-06 2024-09-14T04:08:17.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.850 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.850 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:17.851 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.083752+0000", 2024-09-14T04:08:17.851 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80013120199999999 2024-09-14T04:08:17.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.083870+0000", 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117824 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.083994+0000", 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012407999999999999 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096768+0000", 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012773746000000001 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097306+0000", 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00053861299999999996 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.118040+0000", 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020733445999999999 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.853 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.118083+0000", 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3464999999999998e-05 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.118121+0000", 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7966000000000002e-05 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220968 9.f 9:f83ac14a:::benchmark_data_smithi114_153893_object91507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.854 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.290423+0000", 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.370182600000007, 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83063764699999998, 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220968 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.855 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290423+0000", 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290423+0000", 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290422+0000", 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290426+0000", 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9559999999999999e-06 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290435+0000", 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9630000000000004e-06 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290440+0000", 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7489999999999998e-06 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.291044+0000", 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00060378600000000004 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.291051+0000", 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1350000000000003e-06 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.087791+0000", 2024-09-14T04:08:17.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.79674028699999999 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.087905+0000", 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011346200000000001 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.088019+0000", 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011462499999999999 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096775+0000", 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0087554710000000008 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097356+0000", 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00058162199999999996 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.120960+0000", 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.023603678999999999 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.121005+0000", 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4607000000000002e-05 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.121061+0000", 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6013000000000001e-05 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.861 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220980 9.f 9:f3c12ea5:::benchmark_data_smithi114_153893_object91519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.311552+0000", 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 97.349053057000006, 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80967589200000001, 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220980 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:17.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311552+0000", 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311552+0000", 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311551+0000", 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311557+0000", 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3519999999999997e-06 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311565+0000", 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6689999999999995e-06 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311569+0000", 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1860000000000002e-06 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311601+0000", 2024-09-14T04:08:17.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1862000000000001e-05 2024-09-14T04:08:17.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:17.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311608+0000", 2024-09-14T04:08:17.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9149999999999999e-06 2024-09-14T04:08:17.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.091990+0000", 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78038170600000001 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.092117+0000", 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012667200000000001 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.092253+0000", 2024-09-14T04:08:17.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013666899999999999 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097073+0000", 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048195069999999998 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.104825+0000", 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0077518580000000004 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:17.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.121153+0000", 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016328667000000002 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.121192+0000", 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9079999999999999e-05 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.121228+0000", 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6010000000000003e-05 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.868 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.869 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:17.869 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:17.869 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:08:17.887 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:17.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:17.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:18.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:18.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:18.083 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:17 smithi160 ceph-mon[88407]: pgmap v714: 89 pgs: 89 active+clean; 1.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-09-14T04:08:18.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:18.151 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:18.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:18.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 36319 36303 151.235 115.375 0.00622242 0.00660676 2024-09-14T04:08:18.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:18.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:18.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:18.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:18.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:17 smithi114 ceph-mon[106969]: pgmap v714: 89 pgs: 89 active+clean; 1.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-09-14T04:08:18.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:17 smithi114 ceph-mon[103029]: pgmap v714: 89 pgs: 89 active+clean; 1.8 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-09-14T04:08:18.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:18.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:18.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:18.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:18.541 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:18.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:18.643 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:18.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:18.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:18.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:18.790 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:18.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:18.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:18.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:18.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:19.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:19.055 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:19.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:19.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:19.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:19.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:19.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 37549 37533 146.588 76.875 0.0152896 0.00681626 2024-09-14T04:08:19.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:19.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:19.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:19.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:19.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:19.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:19.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:19.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:19.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:19.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:19.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:19.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:19.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:19.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:19.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:19.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:20.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:20.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:20.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:20.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:19 smithi114 ceph-mon[103029]: pgmap v715: 89 pgs: 89 active+clean; 1.8 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-14T04:08:20.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:19 smithi114 ceph-mon[106969]: pgmap v715: 89 pgs: 89 active+clean; 1.8 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-14T04:08:20.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:20.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:20.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:19 smithi160 ceph-mon[88407]: pgmap v715: 89 pgs: 89 active+clean; 1.8 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-14T04:08:20.230 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 39218 39202 144.1 104.312 0.00761339 0.006935 2024-09-14T04:08:20.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:20.273 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:20.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:20.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:20.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:20.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:20.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:20.532 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:20.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:20.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:08:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:08:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:08:20.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:20.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:20.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:20.787 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:20.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:20.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:20.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:20.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:20.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:21.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:21.057 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:21.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:21.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:21.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:21.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:21.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:21.230 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 40884 40868 141.879 104.125 0.00206209 0.00702902 2024-09-14T04:08:21.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:21.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:21.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:21.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-14T04:08:21.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:21.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:21.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:21.567 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:21.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:21.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:21.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:21.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:21.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:21.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:21.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:08:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:08:21] "GET /metrics HTTP/1.1" 200 36473 "" "Prometheus/2.51.0" 2024-09-14T04:08:21.895 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:21.940 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:21.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:21.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:22.031 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:21 smithi160 ceph-mon[88407]: pgmap v716: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-14T04:08:22.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:22.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:22.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:22.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:22.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 42296 42280 139.046 88.25 0.00649505 0.00718693 2024-09-14T04:08:22.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:22.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:22.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:22.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:21 smithi114 ceph-mon[103029]: pgmap v716: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-14T04:08:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:21 smithi114 ceph-mon[106969]: pgmap v716: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-14T04:08:22.386 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:22.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:22.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:22.426 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-14T04:08:22.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:08:22.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:22.546 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:22.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:22.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:22.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:22.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:22.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:22.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:22.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:22.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:22.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:22.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:22.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:23.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:23.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:23.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:23.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:23.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:23.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:23.192 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:23.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:23.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:23.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:08:23.231153+0000 min lat: 0.00109236 max lat: 0.268444 avg lat: 0.00729112 2024-09-14T04:08:23.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:08:23.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 43887 43871 137.066 99.4375 0.00742701 0.00729112 2024-09-14T04:08:23.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:23.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:23.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:23.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:23.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:23.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:23.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:23.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:23.624 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:23.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:23.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:23.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:23.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:23.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:23.921 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:23.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:24.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:24.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:24.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:24.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:23 smithi114 ceph-mon[103029]: pgmap v717: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-14T04:08:24.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:23 smithi114 ceph-mon[106969]: pgmap v717: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-14T04:08:24.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:24.181 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:24.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:24.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 45516 45500 135.387 101.812 0.00269675 0.00737977 2024-09-14T04:08:24.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:24.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:24.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:23 smithi160 ceph-mon[88407]: pgmap v717: 89 pgs: 89 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-14T04:08:24.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:24.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:24.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:24.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:24.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:24.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:24.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:24.706 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:24.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:24.808 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:24.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:24.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:24.940 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:24.961 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:24.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:25.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:25.096 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:25.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:25.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:25.233 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 46983 46967 133.393 91.6875 0.00767586 0.00749068 2024-09-14T04:08:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:25.236 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:25.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:25.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:25.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:25.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:25.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:25.533 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:25.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:25.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:25.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:25.684 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:25.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:25.759 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:25.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:25.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:25.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:25.825 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:08:25.825 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:08:25.859 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:08:25.859 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:08:25.896 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:08:25.896 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:08:25.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:25.934 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:08:25.942 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:08:25.942 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:08:25.956 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:25.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:25.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:25.973 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:08:25.973 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:08:26.008 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:08:26.009 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:08:26.050 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:08:26.051 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:08:26.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:26.082 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:08:26.082 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:08:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:26.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:26.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:26.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:26.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:26.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:26.233 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 48521 48505 131.773 96.125 0.00658741 0.00758355 2024-09-14T04:08:26.303 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:25 smithi160 ceph-mon[88407]: pgmap v718: 89 pgs: 89 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T04:08:26.304 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/357879501' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:08:26.304 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:25 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:08:26.304 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1451247157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:26.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:26.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:26.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:26.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:25 smithi114 ceph-mon[106969]: pgmap v718: 89 pgs: 89 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T04:08:26.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/357879501' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:08:26.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:25 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:08:26.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1451247157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:26.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:25 smithi114 ceph-mon[103029]: pgmap v718: 89 pgs: 89 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T04:08:26.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/357879501' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:08:26.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:25 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:08:26.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1451247157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:26.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:26.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:26.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:26.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:26.634 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:26.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:26.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:26.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:26.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:26.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:26.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:26.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:27.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:27.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:27.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:27.066 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:26 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:08:27.066 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:26 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:08:27.066 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:26 smithi160 ceph-mon[88407]: osdmap e206: 8 total, 8 up, 7 in 2024-09-14T04:08:27.067 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:26 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:27.067 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:26 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:26 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:08:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:26 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:08:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:26 smithi114 ceph-mon[106969]: osdmap e206: 8 total, 8 up, 7 in 2024-09-14T04:08:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:26 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:26 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:26 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:08:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:26 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:08:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:26 smithi114 ceph-mon[103029]: osdmap e206: 8 total, 8 up, 7 in 2024-09-14T04:08:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:26 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:26 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:27.141 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:27.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:27.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:27.215 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:08:27.216 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:08:27.233 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 50250 50234 130.784 108.062 0.0137266 0.00764111 2024-09-14T04:08:27.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:27.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:27.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:27.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:27.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:27.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:27.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:27.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:27.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:27.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:27.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:27.716 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:27.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:27.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:27.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:27.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:27.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:27.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:27.959 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-14T04:08:28.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:28.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:28.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:28.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:28.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:28.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:28.195 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:27 smithi114 ceph-mon[103029]: pgmap v720: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-14T04:08:28.195 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:27 smithi114 ceph-mon[106969]: pgmap v720: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-14T04:08:28.233 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 51871 51855 129.605 101.312 0.014582 0.00770926 2024-09-14T04:08:28.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:28.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:28.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:28.323 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:28.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:27 smithi160 ceph-mon[88407]: pgmap v720: 89 pgs: 89 active+clean; 2.8 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-14T04:08:28.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:28.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:28.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:28.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:28.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:28.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:28.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:28.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:28.839 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:28.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:28.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:08:28.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:28.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:28.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:29.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:29.123 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:29.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:29.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:29.233 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 53284 53268 128.017 88.3125 0.0110182 0.00780727 2024-09-14T04:08:29.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:29.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:29.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:29.392 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:29.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:29.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:29.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:29.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:29.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:29.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:29.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:29.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:29.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:29.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:29.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:29.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:30.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:30.099 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:30.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:30.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:29 smithi114 ceph-mon[106969]: pgmap v721: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-14T04:08:30.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:29 smithi114 ceph-mon[103029]: pgmap v721: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-14T04:08:30.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:08:30.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:30.215 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:29 smithi160 ceph-mon[88407]: pgmap v721: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-14T04:08:30.233 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 54799 54783 126.782 94.6875 0.00507179 0.00788161 2024-09-14T04:08:30.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:30.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:30.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:30.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:30.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:30.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:30.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:30.559 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:30.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:08:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:08:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:08:30.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:30.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:30.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:30.701 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:30.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:30.826 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:30.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:30.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:30.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:30.955 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:30.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:30.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:31.038 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:08:31.040 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:08:31.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:31.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:31.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:31.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:30 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:31.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:30 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:31.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:30 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:08:31.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:30 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:08:31.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:30 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:31.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:30 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:31.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:30 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:31.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:30 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:08:31.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:30 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:08:31.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:30 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:31.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:31.233 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 55861 55845 124.625 66.375 0.0140722 0.008019 2024-09-14T04:08:31.237 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:30 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:31.237 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:30 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:31.237 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:30 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:08:31.237 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:30 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:08:31.237 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:30 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:31.240 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:31.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:31.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:31.373 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:31.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:31.400 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:08:31.400 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:08:31.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:31.511 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:31.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:31.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:31.652 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:31.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:31.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:31.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:31.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:31.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:08:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:08:31] "GET /metrics HTTP/1.1" 200 36479 "" "Prometheus/2.51.0" 2024-09-14T04:08:31.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:31.928 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:31.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:32.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:32.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:31 smithi160 ceph-mon[88407]: pgmap v722: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-14T04:08:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:31 smithi160 ceph-mon[88407]: from='client.45935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:08:32.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:32.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:32.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:32.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:32.234 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 57182 57166 123.172 82.5625 0.0132491 0.00811317 2024-09-14T04:08:32.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:31 smithi114 ceph-mon[103029]: pgmap v722: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-14T04:08:32.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:31 smithi114 ceph-mon[103029]: from='client.45935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:08:32.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:31 smithi114 ceph-mon[106969]: pgmap v722: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-14T04:08:32.243 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:31 smithi114 ceph-mon[106969]: from='client.45935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:08:32.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-14T04:08:32.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:32.337 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:32.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:32.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:32.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:32.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:32.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:32.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:32.611 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:32.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:32.641 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:32.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:32.745 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:32.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:32.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:32.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:32.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:32.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:32.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:33.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:33.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:33.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:33.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:33.158 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:33.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:33.234 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 58602 58586 122.025 88.75 0.00787894 0.00818977 2024-09-14T04:08:33.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:33.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:33.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:33.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:33.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:33.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:33.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:33.548 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:33.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:33.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:33.681 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:33.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:33.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:33.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:33.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:33.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:33.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:33.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:34.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:34.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:34.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:34.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:34.111 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:33 smithi160 ceph-mon[88407]: pgmap v723: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-14T04:08:34.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:34.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:34.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:34.234 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 59874 59858 120.653 79.5 0.00551862 0.00828348 2024-09-14T04:08:34.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:34.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:34.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:34.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:33 smithi114 ceph-mon[106969]: pgmap v723: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-14T04:08:34.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:33 smithi114 ceph-mon[103029]: pgmap v723: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-14T04:08:34.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:34.493 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:34.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:34.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:34.623 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:34.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:34.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:34.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:34.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:34.806 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:08:34.806 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:08:34.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:34.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:34.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:34.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:35.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:35.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:35.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:35.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:35.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:35.234 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 61086 61070 119.25 75.75 0.00665543 0.00838091 2024-09-14T04:08:35.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:35.265 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:35.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:35.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:35.405 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:35.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:35.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:35.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:35.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:35.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-14T04:08:35.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:35.711 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:35.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:35.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:35.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:35.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:35.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:35.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:35.975 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:35.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:35.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:35.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:36.059 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:36.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:36.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:36.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:36.234 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 62453 62437 118.225 85.4375 0.00959624 0.00845355 2024-09-14T04:08:36.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:36.284 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:36.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:36.341 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:35 smithi160 ceph-mon[88407]: pgmap v724: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-14T04:08:36.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:35 smithi114 ceph-mon[106969]: pgmap v724: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-14T04:08:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:35 smithi114 ceph-mon[103029]: pgmap v724: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-14T04:08:36.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:36.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:36.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:36.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:36.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:36.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:36.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:36.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:36.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:36.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:36.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:36.842 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:36.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:36.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:36.975 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:36.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:37.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:37.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:37.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:37.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:37.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:37.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:37.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 63869 63853 117.351 88.5 0.0072452 0.00851682 2024-09-14T04:08:37.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:37.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:37.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:37.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:37.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:37.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:37.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:37.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:37.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:37.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:37.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:37.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:37.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:37.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:37.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:38.012 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:38.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:38.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:38.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:38.179 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:38.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:38.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:38.234 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 65327 65311 116.601 91.125 0.00308826 0.00856831 2024-09-14T04:08:38.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:38.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:38.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:38.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:38 smithi114 ceph-mon[106969]: pgmap v725: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-14T04:08:38.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:37 smithi114 ceph-mon[103029]: pgmap v725: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-14T04:08:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:38 smithi160 ceph-mon[88407]: pgmap v725: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-14T04:08:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:38.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:38.445 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2024-09-14T04:08:30.732872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3423410125,"num_objects":52254,"num_object_clones":0,"num_object_copies":156762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52254,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":52440,"num_write_kb":3344251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":595258,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52386,"ondisk_log_size":52386,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14353492,"kb_used_data":9987928,"kb_used_omap":27,"kb_used_meta":4365412,"kb_avail":641862572,"statfs":{"total":671965249536,"available":657267273728,"internally_reserved":0,"allocated":10227638272,"data_stored":10220878930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28483,"internal_metadata":4470182077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1038614528,"num_objects":15848,"num_object_clones":0,"num_object_copies":47544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15848,"num_write_kb":1014272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.008821"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":219,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944367+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:08:25.944367+0000","last_peered":"2024-09-14T04:08:25.944367+0000","last_clean":"2024-09-14T04:08:25.944367+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:08:25.944367+0000","last_undegraded":"2024-09-14T04:08:25.944367+0000","last_fullsized":"2024-09-14T04:08:25.944367+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":189,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894375+0000","last_change":"2024-09-14T04:05:55.701296+0000","last_active":"2024-09-14T04:08:14.894375+0000","last_peered":"2024-09-14T04:08:14.894375+0000","last_clean":"2024-09-14T04:08:14.894375+0000","last_became_active":"2024-09-14T04:05:55.676634+0000","last_became_peered":"2024-09-14T04:05:55.676634+0000","last_unstale":"2024-09-14T04:08:14.894375+0000","last_undegraded":"2024-09-14T04:08:14.894375+0000","last_fullsized":"2024-09-14T04:08:14.894375+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:44.187804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":171,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888168+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:08:14.888168+0000","last_peered":"2024-09-14T04:08:14.888168+0000","last_clean":"2024-09-14T04:08:14.888168+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:08:14.888168+0000","last_undegraded":"2024-09-14T04:08:14.888168+0000","last_fullsized":"2024-09-14T04:08:14.888168+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":303,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.937038+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:08:25.937038+0000","last_peered":"2024-09-14T04:08:25.937038+0000","last_clean":"2024-09-14T04:08:25.937038+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:08:25.937038+0000","last_undegraded":"2024-09-14T04:08:25.937038+0000","last_fullsized":"2024-09-14T04:08:25.937038+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":289,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.938315+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:08:25.938315+0000","last_peered":"2024-09-14T04:08:25.938315+0000","last_clean":"2024-09-14T04:08:25.938315+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:08:25.938315+0000","last_undegraded":"2024-09-14T04:08:25.938315+0000","last_fullsized":"2024-09-14T04:08:25.938315+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":164,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.896040+0000","last_change":"2024-09-14T04:05:55.676850+0000","last_active":"2024-09-14T04:08:14.896040+0000","last_peered":"2024-09-14T04:08:14.896040+0000","last_clean":"2024-09-14T04:08:14.896040+0000","last_became_active":"2024-09-14T04:05:55.676337+0000","last_became_peered":"2024-09-14T04:05:55.676337+0000","last_unstale":"2024-09-14T04:08:14.896040+0000","last_undegraded":"2024-09-14T04:08:14.896040+0000","last_fullsized":"2024-09-14T04:08:14.896040+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:12.377665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.945978+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:08:25.945978+0000","last_peered":"2024-09-14T04:08:25.945978+0000","last_clean":"2024-09-14T04:08:25.945978+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:08:25.945978+0000","last_undegraded":"2024-09-14T04:08:25.945978+0000","last_fullsized":"2024-09-14T04:08:25.945978+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":180,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.891988+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:08:14.891988+0000","last_peered":"2024-09-14T04:08:14.891988+0000","last_clean":"2024-09-14T04:08:14.891988+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:08:14.891988+0000","last_undegraded":"2024-09-14T04:08:14.891988+0000","last_fullsized":"2024-09-14T04:08:14.891988+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"206'3290","reported_seq":3305,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:29.249784+0000","last_change":"2024-09-14T04:07:55.368334+0000","last_active":"2024-09-14T04:08:29.249784+0000","last_peered":"2024-09-14T04:08:29.249784+0000","last_clean":"2024-09-14T04:08:29.249784+0000","last_became_active":"2024-09-14T04:07:55.368016+0000","last_became_peered":"2024-09-14T04:07:55.368016+0000","last_unstale":"2024-09-14T04:08:29.249784+0000","last_undegraded":"2024-09-14T04:08:29.249784+0000","last_fullsized":"2024-09-14T04:08:29.249784+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3290,"log_dups_size":0,"ondisk_log_size":3290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:13.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215613440,"num_objects":3290,"num_object_clones":0,"num_object_copies":9870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3290,"num_write_kb":210560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":186,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.895972+0000","last_change":"2024-09-14T04:05:55.704897+0000","last_active":"2024-09-14T04:08:14.895972+0000","last_peered":"2024-09-14T04:08:14.895972+0000","last_clean":"2024-09-14T04:08:14.895972+0000","last_became_active":"2024-09-14T04:05:55.681093+0000","last_became_peered":"2024-09-14T04:05:55.681093+0000","last_unstale":"2024-09-14T04:08:14.895972+0000","last_undegraded":"2024-09-14T04:08:14.895972+0000","last_fullsized":"2024-09-14T04:08:14.895972+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:49.647801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":154,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.887865+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:08:14.887865+0000","last_peered":"2024-09-14T04:08:14.887865+0000","last_clean":"2024-09-14T04:08:14.887865+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:08:14.887865+0000","last_undegraded":"2024-09-14T04:08:14.887865+0000","last_fullsized":"2024-09-14T04:08:14.887865+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"206'3360","reported_seq":3375,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:28.720574+0000","last_change":"2024-09-14T04:07:55.607517+0000","last_active":"2024-09-14T04:08:28.720574+0000","last_peered":"2024-09-14T04:08:28.720574+0000","last_clean":"2024-09-14T04:08:28.720574+0000","last_became_active":"2024-09-14T04:07:55.607208+0000","last_became_peered":"2024-09-14T04:07:55.607208+0000","last_unstale":"2024-09-14T04:08:28.720574+0000","last_undegraded":"2024-09-14T04:08:28.720574+0000","last_fullsized":"2024-09-14T04:08:28.720574+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3360,"log_dups_size":0,"ondisk_log_size":3360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:32:20.123209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220200960,"num_objects":3360,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3360,"num_write_kb":215040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":187,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.896126+0000","last_change":"2024-09-14T04:05:55.702981+0000","last_active":"2024-09-14T04:08:14.896126+0000","last_peered":"2024-09-14T04:08:14.896126+0000","last_clean":"2024-09-14T04:08:14.896126+0000","last_became_active":"2024-09-14T04:05:55.679789+0000","last_became_peered":"2024-09-14T04:05:55.679789+0000","last_unstale":"2024-09-14T04:08:14.896126+0000","last_undegraded":"2024-09-14T04:08:14.896126+0000","last_fullsized":"2024-09-14T04:08:14.896126+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:34:24.358644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":209,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.930854+0000","last_change":"2024-09-14T04:07:44.340689+0000","last_active":"2024-09-14T04:08:25.930854+0000","last_peered":"2024-09-14T04:08:25.930854+0000","last_clean":"2024-09-14T04:08:25.930854+0000","last_became_active":"2024-09-14T04:07:44.340112+0000","last_became_peered":"2024-09-14T04:07:44.340112+0000","last_unstale":"2024-09-14T04:08:25.930854+0000","last_undegraded":"2024-09-14T04:08:25.930854+0000","last_fullsized":"2024-09-14T04:08:25.930854+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:04.263233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"206'3224","reported_seq":3249,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:27.812361+0000","last_change":"2024-09-14T04:07:55.608111+0000","last_active":"2024-09-14T04:08:27.812361+0000","last_peered":"2024-09-14T04:08:27.812361+0000","last_clean":"2024-09-14T04:08:27.812361+0000","last_became_active":"2024-09-14T04:07:55.607646+0000","last_became_peered":"2024-09-14T04:07:55.607646+0000","last_unstale":"2024-09-14T04:08:27.812361+0000","last_undegraded":"2024-09-14T04:08:27.812361+0000","last_fullsized":"2024-09-14T04:08:27.812361+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3224,"log_dups_size":0,"ondisk_log_size":3224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:02:33.076539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211288064,"num_objects":3224,"num_object_clones":0,"num_object_copies":9672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3224,"num_write_kb":206336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":155,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894734+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:08:14.894734+0000","last_peered":"2024-09-14T04:08:14.894734+0000","last_clean":"2024-09-14T04:08:14.894734+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:08:14.894734+0000","last_undegraded":"2024-09-14T04:08:14.894734+0000","last_fullsized":"2024-09-14T04:08:14.894734+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":181,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.892174+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:08:14.892174+0000","last_peered":"2024-09-14T04:08:14.892174+0000","last_clean":"2024-09-14T04:08:14.892174+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:08:14.892174+0000","last_undegraded":"2024-09-14T04:08:14.892174+0000","last_fullsized":"2024-09-14T04:08:14.892174+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"206'3032","reported_seq":3056,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:26.420176+0000","last_change":"2024-09-14T04:07:55.607545+0000","last_active":"2024-09-14T04:08:26.420176+0000","last_peered":"2024-09-14T04:08:26.420176+0000","last_clean":"2024-09-14T04:08:26.420176+0000","last_became_active":"2024-09-14T04:07:55.607263+0000","last_became_peered":"2024-09-14T04:07:55.607263+0000","last_unstale":"2024-09-14T04:08:26.420176+0000","last_undegraded":"2024-09-14T04:08:26.420176+0000","last_fullsized":"2024-09-14T04:08:26.420176+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3032,"log_dups_size":0,"ondisk_log_size":3032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:38:52.910168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198705152,"num_objects":3032,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3032,"num_write_kb":194048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":192,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888174+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:08:14.888174+0000","last_peered":"2024-09-14T04:08:14.888174+0000","last_clean":"2024-09-14T04:08:14.888174+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:08:14.888174+0000","last_undegraded":"2024-09-14T04:08:14.888174+0000","last_fullsized":"2024-09-14T04:08:14.888174+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":318,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933574+0000","last_change":"2024-09-14T04:07:54.363033+0000","last_active":"2024-09-14T04:08:25.933574+0000","last_peered":"2024-09-14T04:08:25.933574+0000","last_clean":"2024-09-14T04:08:25.933574+0000","last_became_active":"2024-09-14T04:07:54.362706+0000","last_became_peered":"2024-09-14T04:07:54.362706+0000","last_unstale":"2024-09-14T04:08:25.933574+0000","last_undegraded":"2024-09-14T04:08:25.933574+0000","last_fullsized":"2024-09-14T04:08:25.933574+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:31.193597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"206'3375","reported_seq":3399,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:29.440993+0000","last_change":"2024-09-14T04:07:55.607327+0000","last_active":"2024-09-14T04:08:29.440993+0000","last_peered":"2024-09-14T04:08:29.440993+0000","last_clean":"2024-09-14T04:08:29.440993+0000","last_became_active":"2024-09-14T04:07:55.606967+0000","last_became_peered":"2024-09-14T04:07:55.606967+0000","last_unstale":"2024-09-14T04:08:29.440993+0000","last_undegraded":"2024-09-14T04:08:29.440993+0000","last_fullsized":"2024-09-14T04:08:29.440993+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3375,"log_dups_size":0,"ondisk_log_size":3375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:36.618823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221184000,"num_objects":3375,"num_object_clones":0,"num_object_copies":10125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3375,"num_write_kb":216000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":258,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944030+0000","last_change":"2024-09-14T04:05:55.695825+0000","last_active":"2024-09-14T04:08:25.944030+0000","last_peered":"2024-09-14T04:08:25.944030+0000","last_clean":"2024-09-14T04:08:25.944030+0000","last_became_active":"2024-09-14T04:05:55.680668+0000","last_became_peered":"2024-09-14T04:05:55.680668+0000","last_unstale":"2024-09-14T04:08:25.944030+0000","last_undegraded":"2024-09-14T04:08:25.944030+0000","last_fullsized":"2024-09-14T04:08:25.944030+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:30.262947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":178,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944437+0000","last_change":"2024-09-14T04:05:55.677723+0000","last_active":"2024-09-14T04:08:25.944437+0000","last_peered":"2024-09-14T04:08:25.944437+0000","last_clean":"2024-09-14T04:08:25.944437+0000","last_became_active":"2024-09-14T04:05:55.677254+0000","last_became_peered":"2024-09-14T04:05:55.677254+0000","last_unstale":"2024-09-14T04:08:25.944437+0000","last_undegraded":"2024-09-14T04:08:25.944437+0000","last_fullsized":"2024-09-14T04:08:25.944437+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:53.163120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"206'3052","reported_seq":3077,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:26.415536+0000","last_change":"2024-09-14T04:07:55.607260+0000","last_active":"2024-09-14T04:08:26.415536+0000","last_peered":"2024-09-14T04:08:26.415536+0000","last_clean":"2024-09-14T04:08:26.415536+0000","last_became_active":"2024-09-14T04:07:55.606973+0000","last_became_peered":"2024-09-14T04:07:55.606973+0000","last_unstale":"2024-09-14T04:08:26.415536+0000","last_undegraded":"2024-09-14T04:08:26.415536+0000","last_fullsized":"2024-09-14T04:08:26.415536+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3052,"log_dups_size":0,"ondisk_log_size":3052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:59:15.715573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200015872,"num_objects":3052,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":195328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":182,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.892291+0000","last_change":"2024-09-14T04:05:55.679201+0000","last_active":"2024-09-14T04:08:14.892291+0000","last_peered":"2024-09-14T04:08:14.892291+0000","last_clean":"2024-09-14T04:08:14.892291+0000","last_became_active":"2024-09-14T04:05:55.678843+0000","last_became_peered":"2024-09-14T04:05:55.678843+0000","last_unstale":"2024-09-14T04:08:14.892291+0000","last_undegraded":"2024-09-14T04:08:14.892291+0000","last_fullsized":"2024-09-14T04:08:14.892291+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:12:57.418514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":169,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.951780+0000","last_change":"2024-09-14T04:05:55.673650+0000","last_active":"2024-09-14T04:08:25.951780+0000","last_peered":"2024-09-14T04:08:25.951780+0000","last_clean":"2024-09-14T04:08:25.951780+0000","last_became_active":"2024-09-14T04:05:55.673258+0000","last_became_peered":"2024-09-14T04:05:55.673258+0000","last_unstale":"2024-09-14T04:08:25.951780+0000","last_undegraded":"2024-09-14T04:08:25.951780+0000","last_fullsized":"2024-09-14T04:08:25.951780+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:21.089797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"206'3313","reported_seq":3328,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:28.707535+0000","last_change":"2024-09-14T04:07:55.365272+0000","last_active":"2024-09-14T04:08:28.707535+0000","last_peered":"2024-09-14T04:08:28.707535+0000","last_clean":"2024-09-14T04:08:28.707535+0000","last_became_active":"2024-09-14T04:07:55.364722+0000","last_became_peered":"2024-09-14T04:07:55.364722+0000","last_unstale":"2024-09-14T04:08:28.707535+0000","last_undegraded":"2024-09-14T04:08:28.707535+0000","last_fullsized":"2024-09-14T04:08:28.707535+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3313,"log_dups_size":0,"ondisk_log_size":3313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:40:55.470085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217120768,"num_objects":3313,"num_object_clones":0,"num_object_copies":9939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":212032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":153,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944277+0000","last_change":"2024-09-14T04:05:55.671329+0000","last_active":"2024-09-14T04:08:25.944277+0000","last_peered":"2024-09-14T04:08:25.944277+0000","last_clean":"2024-09-14T04:08:25.944277+0000","last_became_active":"2024-09-14T04:05:55.671032+0000","last_became_peered":"2024-09-14T04:05:55.671032+0000","last_unstale":"2024-09-14T04:08:25.944277+0000","last_undegraded":"2024-09-14T04:08:25.944277+0000","last_fullsized":"2024-09-14T04:08:25.944277+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:58:10.779452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":103,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894352+0000","last_change":"2024-09-14T04:05:55.679870+0000","last_active":"2024-09-14T04:08:14.894352+0000","last_peered":"2024-09-14T04:08:14.894352+0000","last_clean":"2024-09-14T04:08:14.894352+0000","last_became_active":"2024-09-14T04:05:55.679574+0000","last_became_peered":"2024-09-14T04:05:55.679574+0000","last_unstale":"2024-09-14T04:08:14.894352+0000","last_undegraded":"2024-09-14T04:08:14.894352+0000","last_fullsized":"2024-09-14T04:08:14.894352+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:51:09.931574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"206'3289","reported_seq":3304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:29.037533+0000","last_change":"2024-09-14T04:07:55.367126+0000","last_active":"2024-09-14T04:08:29.037533+0000","last_peered":"2024-09-14T04:08:29.037533+0000","last_clean":"2024-09-14T04:08:29.037533+0000","last_became_active":"2024-09-14T04:07:55.366798+0000","last_became_peered":"2024-09-14T04:07:55.366798+0000","last_unstale":"2024-09-14T04:08:29.037533+0000","last_undegraded":"2024-09-14T04:08:29.037533+0000","last_fullsized":"2024-09-14T04:08:29.037533+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3289,"log_dups_size":0,"ondisk_log_size":3289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:24:51.791784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215547904,"num_objects":3289,"num_object_clones":0,"num_object_copies":9867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3289,"num_write_kb":210496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943605+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:08:25.943605+0000","last_peered":"2024-09-14T04:08:25.943605+0000","last_clean":"2024-09-14T04:08:25.943605+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:08:25.943605+0000","last_undegraded":"2024-09-14T04:08:25.943605+0000","last_fullsized":"2024-09-14T04:08:25.943605+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":262,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.931383+0000","last_change":"2024-09-14T04:05:55.678520+0000","last_active":"2024-09-14T04:08:25.931383+0000","last_peered":"2024-09-14T04:08:25.931383+0000","last_clean":"2024-09-14T04:08:25.931383+0000","last_became_active":"2024-09-14T04:05:55.678008+0000","last_became_peered":"2024-09-14T04:05:55.678008+0000","last_unstale":"2024-09-14T04:08:25.931383+0000","last_undegraded":"2024-09-14T04:08:25.931383+0000","last_fullsized":"2024-09-14T04:08:25.931383+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:40:26.778212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"206'3195","reported_seq":3210,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:28.721697+0000","last_change":"2024-09-14T04:07:55.367659+0000","last_active":"2024-09-14T04:08:28.721697+0000","last_peered":"2024-09-14T04:08:28.721697+0000","last_clean":"2024-09-14T04:08:28.721697+0000","last_became_active":"2024-09-14T04:07:55.367247+0000","last_became_peered":"2024-09-14T04:07:55.367247+0000","last_unstale":"2024-09-14T04:08:28.721697+0000","last_undegraded":"2024-09-14T04:08:28.721697+0000","last_fullsized":"2024-09-14T04:08:28.721697+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:58:08.638370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209387520,"num_objects":3195,"num_object_clones":0,"num_object_copies":9585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":204480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":288,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.899168+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:08:14.899168+0000","last_peered":"2024-09-14T04:08:14.899168+0000","last_clean":"2024-09-14T04:08:14.899168+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:08:14.899168+0000","last_undegraded":"2024-09-14T04:08:14.899168+0000","last_fullsized":"2024-09-14T04:08:14.899168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":185,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888092+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:08:14.888092+0000","last_peered":"2024-09-14T04:08:14.888092+0000","last_clean":"2024-09-14T04:08:14.888092+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:08:14.888092+0000","last_undegraded":"2024-09-14T04:08:14.888092+0000","last_fullsized":"2024-09-14T04:08:14.888092+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.946536+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:08:25.946536+0000","last_peered":"2024-09-14T04:08:25.946536+0000","last_clean":"2024-09-14T04:08:25.946536+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:08:25.946536+0000","last_undegraded":"2024-09-14T04:08:25.946536+0000","last_fullsized":"2024-09-14T04:08:25.946536+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":164,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894091+0000","last_change":"2024-09-14T04:05:55.676445+0000","last_active":"2024-09-14T04:08:14.894091+0000","last_peered":"2024-09-14T04:08:14.894091+0000","last_clean":"2024-09-14T04:08:14.894091+0000","last_became_active":"2024-09-14T04:05:55.674141+0000","last_became_peered":"2024-09-14T04:05:55.674141+0000","last_unstale":"2024-09-14T04:08:14.894091+0000","last_undegraded":"2024-09-14T04:08:14.894091+0000","last_fullsized":"2024-09-14T04:08:14.894091+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:28:50.348309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"206'3243","reported_seq":3267,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:27.824259+0000","last_change":"2024-09-14T04:07:55.366975+0000","last_active":"2024-09-14T04:08:27.824259+0000","last_peered":"2024-09-14T04:08:27.824259+0000","last_clean":"2024-09-14T04:08:27.824259+0000","last_became_active":"2024-09-14T04:07:55.366645+0000","last_became_peered":"2024-09-14T04:07:55.366645+0000","last_unstale":"2024-09-14T04:08:27.824259+0000","last_undegraded":"2024-09-14T04:08:27.824259+0000","last_fullsized":"2024-09-14T04:08:27.824259+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:39:56.637518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212533248,"num_objects":3243,"num_object_clones":0,"num_object_copies":9729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3243,"num_write_kb":207552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":200,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.899179+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:08:14.899179+0000","last_peered":"2024-09-14T04:08:14.899179+0000","last_clean":"2024-09-14T04:08:14.899179+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:08:14.899179+0000","last_undegraded":"2024-09-14T04:08:14.899179+0000","last_fullsized":"2024-09-14T04:08:14.899179+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":311,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.934219+0000","last_change":"2024-09-14T04:07:44.337314+0000","last_active":"2024-09-14T04:08:25.934219+0000","last_peered":"2024-09-14T04:08:25.934219+0000","last_clean":"2024-09-14T04:08:25.934219+0000","last_became_active":"2024-09-14T04:07:44.336916+0000","last_became_peered":"2024-09-14T04:07:44.336916+0000","last_unstale":"2024-09-14T04:08:25.934219+0000","last_undegraded":"2024-09-14T04:08:25.934219+0000","last_fullsized":"2024-09-14T04:08:25.934219+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:23.153578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":301,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.938064+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:08:25.938064+0000","last_peered":"2024-09-14T04:08:25.938064+0000","last_clean":"2024-09-14T04:08:25.938064+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:08:25.938064+0000","last_undegraded":"2024-09-14T04:08:25.938064+0000","last_fullsized":"2024-09-14T04:08:25.938064+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"206'3308","reported_seq":3323,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:29.219637+0000","last_change":"2024-09-14T04:07:55.365448+0000","last_active":"2024-09-14T04:08:29.219637+0000","last_peered":"2024-09-14T04:08:29.219637+0000","last_clean":"2024-09-14T04:08:29.219637+0000","last_became_active":"2024-09-14T04:07:55.365168+0000","last_became_peered":"2024-09-14T04:07:55.365168+0000","last_unstale":"2024-09-14T04:08:29.219637+0000","last_undegraded":"2024-09-14T04:08:29.219637+0000","last_fullsized":"2024-09-14T04:08:29.219637+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:25.997258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216793088,"num_objects":3308,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3308,"num_write_kb":211712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":295,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944770+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:08:25.944770+0000","last_peered":"2024-09-14T04:08:25.944770+0000","last_clean":"2024-09-14T04:08:25.944770+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:08:25.944770+0000","last_undegraded":"2024-09-14T04:08:25.944770+0000","last_fullsized":"2024-09-14T04:08:25.944770+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":256,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933187+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:08:25.933187+0000","last_peered":"2024-09-14T04:08:25.933187+0000","last_clean":"2024-09-14T04:08:25.933187+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:08:25.933187+0000","last_undegraded":"2024-09-14T04:08:25.933187+0000","last_fullsized":"2024-09-14T04:08:25.933187+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943194+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:08:25.943194+0000","last_peered":"2024-09-14T04:08:25.943194+0000","last_clean":"2024-09-14T04:08:25.943194+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:08:25.943194+0000","last_undegraded":"2024-09-14T04:08:25.943194+0000","last_fullsized":"2024-09-14T04:08:25.943194+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"206'3296","reported_seq":3310,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:28.726872+0000","last_change":"2024-09-14T04:07:55.367851+0000","last_active":"2024-09-14T04:08:28.726872+0000","last_peered":"2024-09-14T04:08:28.726872+0000","last_clean":"2024-09-14T04:08:28.726872+0000","last_became_active":"2024-09-14T04:07:55.367146+0000","last_became_peered":"2024-09-14T04:07:55.367146+0000","last_unstale":"2024-09-14T04:08:28.726872+0000","last_undegraded":"2024-09-14T04:08:28.726872+0000","last_fullsized":"2024-09-14T04:08:28.726872+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:26:46.495508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216006656,"num_objects":3296,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":210944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":186,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888404+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:08:14.888404+0000","last_peered":"2024-09-14T04:08:14.888404+0000","last_clean":"2024-09-14T04:08:14.888404+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:08:14.888404+0000","last_undegraded":"2024-09-14T04:08:14.888404+0000","last_fullsized":"2024-09-14T04:08:14.888404+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":124,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933876+0000","last_change":"2024-09-14T04:05:55.676693+0000","last_active":"2024-09-14T04:08:25.933876+0000","last_peered":"2024-09-14T04:08:25.933876+0000","last_clean":"2024-09-14T04:08:25.933876+0000","last_became_active":"2024-09-14T04:05:55.676335+0000","last_became_peered":"2024-09-14T04:05:55.676335+0000","last_unstale":"2024-09-14T04:08:25.933876+0000","last_undegraded":"2024-09-14T04:08:25.933876+0000","last_fullsized":"2024-09-14T04:08:25.933876+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:19:43.626016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":280,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.932785+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:08:25.932785+0000","last_peered":"2024-09-14T04:08:25.932785+0000","last_clean":"2024-09-14T04:08:25.932785+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:08:25.932785+0000","last_undegraded":"2024-09-14T04:08:25.932785+0000","last_fullsized":"2024-09-14T04:08:25.932785+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"206'3313","reported_seq":3322,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:29.073220+0000","last_change":"2024-09-14T04:07:55.607796+0000","last_active":"2024-09-14T04:08:29.073220+0000","last_peered":"2024-09-14T04:08:29.073220+0000","last_clean":"2024-09-14T04:08:29.073220+0000","last_became_active":"2024-09-14T04:07:55.607447+0000","last_became_peered":"2024-09-14T04:07:55.607447+0000","last_unstale":"2024-09-14T04:08:29.073220+0000","last_undegraded":"2024-09-14T04:08:29.073220+0000","last_fullsized":"2024-09-14T04:08:29.073220+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3313,"log_dups_size":0,"ondisk_log_size":3313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:09:44.307466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217120768,"num_objects":3313,"num_object_clones":0,"num_object_copies":9939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":212032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":209,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888302+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:08:14.888302+0000","last_peered":"2024-09-14T04:08:14.888302+0000","last_clean":"2024-09-14T04:08:14.888302+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:08:14.888302+0000","last_undegraded":"2024-09-14T04:08:14.888302+0000","last_fullsized":"2024-09-14T04:08:14.888302+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":294,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.932884+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:08:25.932884+0000","last_peered":"2024-09-14T04:08:25.932884+0000","last_clean":"2024-09-14T04:08:25.932884+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:08:25.932884+0000","last_undegraded":"2024-09-14T04:08:25.932884+0000","last_fullsized":"2024-09-14T04:08:25.932884+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":251,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.931142+0000","last_change":"2024-09-14T04:05:55.674511+0000","last_active":"2024-09-14T04:08:25.931142+0000","last_peered":"2024-09-14T04:08:25.931142+0000","last_clean":"2024-09-14T04:08:25.931142+0000","last_became_active":"2024-09-14T04:05:55.674182+0000","last_became_peered":"2024-09-14T04:05:55.674182+0000","last_unstale":"2024-09-14T04:08:25.931142+0000","last_undegraded":"2024-09-14T04:08:25.931142+0000","last_fullsized":"2024-09-14T04:08:25.931142+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:12:34.453647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"206'3288","reported_seq":3301,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:28.699757+0000","last_change":"2024-09-14T04:07:55.607679+0000","last_active":"2024-09-14T04:08:28.699757+0000","last_peered":"2024-09-14T04:08:28.699757+0000","last_clean":"2024-09-14T04:08:28.699757+0000","last_became_active":"2024-09-14T04:07:55.607338+0000","last_became_peered":"2024-09-14T04:07:55.607338+0000","last_unstale":"2024-09-14T04:08:28.699757+0000","last_undegraded":"2024-09-14T04:08:28.699757+0000","last_fullsized":"2024-09-14T04:08:28.699757+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3288,"log_dups_size":0,"ondisk_log_size":3288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:02:20.945987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215482368,"num_objects":3288,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3288,"num_write_kb":210432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":255,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933016+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:08:25.933016+0000","last_peered":"2024-09-14T04:08:25.933016+0000","last_clean":"2024-09-14T04:08:25.933016+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:08:25.933016+0000","last_undegraded":"2024-09-14T04:08:25.933016+0000","last_fullsized":"2024-09-14T04:08:25.933016+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":193,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894945+0000","last_change":"2024-09-14T04:05:55.681238+0000","last_active":"2024-09-14T04:08:14.894945+0000","last_peered":"2024-09-14T04:08:14.894945+0000","last_clean":"2024-09-14T04:08:14.894945+0000","last_became_active":"2024-09-14T04:05:55.680946+0000","last_became_peered":"2024-09-14T04:05:55.680946+0000","last_unstale":"2024-09-14T04:08:14.894945+0000","last_undegraded":"2024-09-14T04:08:14.894945+0000","last_fullsized":"2024-09-14T04:08:14.894945+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:13:39.006545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":279,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933651+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:08:25.933651+0000","last_peered":"2024-09-14T04:08:25.933651+0000","last_clean":"2024-09-14T04:08:25.933651+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:08:25.933651+0000","last_undegraded":"2024-09-14T04:08:25.933651+0000","last_fullsized":"2024-09-14T04:08:25.933651+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"206'3340","reported_seq":3355,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:28.721244+0000","last_change":"2024-09-14T04:07:55.366077+0000","last_active":"2024-09-14T04:08:28.721244+0000","last_peered":"2024-09-14T04:08:28.721244+0000","last_clean":"2024-09-14T04:08:28.721244+0000","last_became_active":"2024-09-14T04:07:55.365753+0000","last_became_peered":"2024-09-14T04:07:55.365753+0000","last_unstale":"2024-09-14T04:08:28.721244+0000","last_undegraded":"2024-09-14T04:08:28.721244+0000","last_fullsized":"2024-09-14T04:08:28.721244+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3340,"log_dups_size":0,"ondisk_log_size":3340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:04:18.784725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218890240,"num_objects":3340,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3340,"num_write_kb":213760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":247,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.946122+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:08:25.946122+0000","last_peered":"2024-09-14T04:08:25.946122+0000","last_clean":"2024-09-14T04:08:25.946122+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:08:25.946122+0000","last_undegraded":"2024-09-14T04:08:25.946122+0000","last_fullsized":"2024-09-14T04:08:25.946122+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":262,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.930955+0000","last_change":"2024-09-14T04:05:55.676617+0000","last_active":"2024-09-14T04:08:25.930955+0000","last_peered":"2024-09-14T04:08:25.930955+0000","last_clean":"2024-09-14T04:08:25.930955+0000","last_became_active":"2024-09-14T04:05:55.676318+0000","last_became_peered":"2024-09-14T04:05:55.676318+0000","last_unstale":"2024-09-14T04:08:25.930955+0000","last_undegraded":"2024-09-14T04:08:25.930955+0000","last_fullsized":"2024-09-14T04:08:25.930955+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:38.529611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.887796+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:08:14.887796+0000","last_peered":"2024-09-14T04:08:14.887796+0000","last_clean":"2024-09-14T04:08:14.887796+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:08:14.887796+0000","last_undegraded":"2024-09-14T04:08:14.887796+0000","last_fullsized":"2024-09-14T04:08:14.887796+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.f","version":"206'3310","reported_seq":3325,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:28.698955+0000","last_change":"2024-09-14T04:07:55.607496+0000","last_active":"2024-09-14T04:08:28.698955+0000","last_peered":"2024-09-14T04:08:28.698955+0000","last_clean":"2024-09-14T04:08:28.698955+0000","last_became_active":"2024-09-14T04:07:55.607124+0000","last_became_peered":"2024-09-14T04:07:55.607124+0000","last_unstale":"2024-09-14T04:08:28.698955+0000","last_undegraded":"2024-09-14T04:08:28.698955+0000","last_fullsized":"2024-09-14T04:08:28.698955+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":3310,"log_dups_size":0,"ondisk_log_size":3310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:15:10.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216924160,"num_objects":3310,"num_object_clones":0,"num_object_copies":9930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3310,"num_write_kb":211840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.938400+0000","last_change":"2024-09-14T04:05:55.690010+0000","last_active":"2024-09-14T04:08:25.938400+0000","last_peered":"2024-09-14T04:08:25.938400+0000","last_clean":"2024-09-14T04:08:25.938400+0000","last_became_active":"2024-09-14T04:05:55.670892+0000","last_became_peered":"2024-09-14T04:05:55.670892+0000","last_unstale":"2024-09-14T04:08:25.938400+0000","last_undegraded":"2024-09-14T04:08:25.938400+0000","last_fullsized":"2024-09-14T04:08:25.938400+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:40:33.024617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":184,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.892909+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:08:14.892909+0000","last_peered":"2024-09-14T04:08:14.892909+0000","last_clean":"2024-09-14T04:08:14.892909+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:08:14.892909+0000","last_undegraded":"2024-09-14T04:08:14.892909+0000","last_fullsized":"2024-09-14T04:08:14.892909+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":49,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888232+0000","last_change":"2024-09-14T04:05:55.674629+0000","last_active":"2024-09-14T04:08:14.888232+0000","last_peered":"2024-09-14T04:08:14.888232+0000","last_clean":"2024-09-14T04:08:14.888232+0000","last_became_active":"2024-09-14T04:05:55.674242+0000","last_became_peered":"2024-09-14T04:05:55.674242+0000","last_unstale":"2024-09-14T04:08:14.888232+0000","last_undegraded":"2024-09-14T04:08:14.888232+0000","last_fullsized":"2024-09-14T04:08:14.888232+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:06:16.827114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":62,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.895407+0000","last_change":"2024-09-14T04:05:55.677636+0000","last_active":"2024-09-14T04:08:14.895407+0000","last_peered":"2024-09-14T04:08:14.895407+0000","last_clean":"2024-09-14T04:08:14.895407+0000","last_became_active":"2024-09-14T04:05:55.677313+0000","last_became_peered":"2024-09-14T04:05:55.677313+0000","last_unstale":"2024-09-14T04:08:14.895407+0000","last_undegraded":"2024-09-14T04:08:14.895407+0000","last_fullsized":"2024-09-14T04:08:14.895407+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:00:13.489641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":102,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894690+0000","last_change":"2024-09-14T04:05:55.686083+0000","last_active":"2024-09-14T04:08:14.894690+0000","last_peered":"2024-09-14T04:08:14.894690+0000","last_clean":"2024-09-14T04:08:14.894690+0000","last_became_active":"2024-09-14T04:05:55.673710+0000","last_became_peered":"2024-09-14T04:05:55.673710+0000","last_unstale":"2024-09-14T04:08:14.894690+0000","last_undegraded":"2024-09-14T04:08:14.894690+0000","last_fullsized":"2024-09-14T04:08:14.894690+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:25.169218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":255,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.930500+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:08:25.930500+0000","last_peered":"2024-09-14T04:08:25.930500+0000","last_clean":"2024-09-14T04:08:25.930500+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:08:25.930500+0000","last_undegraded":"2024-09-14T04:08:25.930500+0000","last_fullsized":"2024-09-14T04:08:25.930500+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":190,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894602+0000","last_change":"2024-09-14T04:05:55.698766+0000","last_active":"2024-09-14T04:08:14.894602+0000","last_peered":"2024-09-14T04:08:14.894602+0000","last_clean":"2024-09-14T04:08:14.894602+0000","last_became_active":"2024-09-14T04:05:55.674923+0000","last_became_peered":"2024-09-14T04:05:55.674923+0000","last_unstale":"2024-09-14T04:08:14.894602+0000","last_undegraded":"2024-09-14T04:08:14.894602+0000","last_fullsized":"2024-09-14T04:08:14.894602+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:52:37.679728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":172,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888424+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:08:14.888424+0000","last_peered":"2024-09-14T04:08:14.888424+0000","last_clean":"2024-09-14T04:08:14.888424+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:08:14.888424+0000","last_undegraded":"2024-09-14T04:08:14.888424+0000","last_fullsized":"2024-09-14T04:08:14.888424+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":207,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944295+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:08:25.944295+0000","last_peered":"2024-09-14T04:08:25.944295+0000","last_clean":"2024-09-14T04:08:25.944295+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:08:25.944295+0000","last_undegraded":"2024-09-14T04:08:25.944295+0000","last_fullsized":"2024-09-14T04:08:25.944295+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":303,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.934110+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:08:25.934110+0000","last_peered":"2024-09-14T04:08:25.934110+0000","last_clean":"2024-09-14T04:08:25.934110+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:08:25.934110+0000","last_undegraded":"2024-09-14T04:08:25.934110+0000","last_fullsized":"2024-09-14T04:08:25.934110+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":219,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943629+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:08:25.943629+0000","last_peered":"2024-09-14T04:08:25.943629+0000","last_clean":"2024-09-14T04:08:25.943629+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:08:25.943629+0000","last_undegraded":"2024-09-14T04:08:25.943629+0000","last_fullsized":"2024-09-14T04:08:25.943629+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":181,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.892755+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:08:14.892755+0000","last_peered":"2024-09-14T04:08:14.892755+0000","last_clean":"2024-09-14T04:08:14.892755+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:08:14.892755+0000","last_undegraded":"2024-09-14T04:08:14.892755+0000","last_fullsized":"2024-09-14T04:08:14.892755+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943499+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:08:25.943499+0000","last_peered":"2024-09-14T04:08:25.943499+0000","last_clean":"2024-09-14T04:08:25.943499+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:08:25.943499+0000","last_undegraded":"2024-09-14T04:08:25.943499+0000","last_fullsized":"2024-09-14T04:08:25.943499+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":305,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.938268+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:08:25.938268+0000","last_peered":"2024-09-14T04:08:25.938268+0000","last_clean":"2024-09-14T04:08:25.938268+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:08:25.938268+0000","last_undegraded":"2024-09-14T04:08:25.938268+0000","last_fullsized":"2024-09-14T04:08:25.938268+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":261,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.930937+0000","last_change":"2024-09-14T04:05:55.677962+0000","last_active":"2024-09-14T04:08:25.930937+0000","last_peered":"2024-09-14T04:08:25.930937+0000","last_clean":"2024-09-14T04:08:25.930937+0000","last_became_active":"2024-09-14T04:05:55.677277+0000","last_became_peered":"2024-09-14T04:05:55.677277+0000","last_unstale":"2024-09-14T04:08:25.930937+0000","last_undegraded":"2024-09-14T04:08:25.930937+0000","last_fullsized":"2024-09-14T04:08:25.930937+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:24.908568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.951034+0000","last_change":"2024-09-14T04:07:44.338546+0000","last_active":"2024-09-14T04:08:25.951034+0000","last_peered":"2024-09-14T04:08:25.951034+0000","last_clean":"2024-09-14T04:08:25.951034+0000","last_became_active":"2024-09-14T04:07:44.338210+0000","last_became_peered":"2024-09-14T04:07:44.338210+0000","last_unstale":"2024-09-14T04:08:25.951034+0000","last_undegraded":"2024-09-14T04:08:25.951034+0000","last_fullsized":"2024-09-14T04:08:25.951034+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:56.100013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":86,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.931270+0000","last_change":"2024-09-14T04:05:55.671091+0000","last_active":"2024-09-14T04:08:25.931270+0000","last_peered":"2024-09-14T04:08:25.931270+0000","last_clean":"2024-09-14T04:08:25.931270+0000","last_became_active":"2024-09-14T04:05:55.670180+0000","last_became_peered":"2024-09-14T04:05:55.670180+0000","last_unstale":"2024-09-14T04:08:25.931270+0000","last_undegraded":"2024-09-14T04:08:25.931270+0000","last_fullsized":"2024-09-14T04:08:25.931270+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:33.489272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":298,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943601+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:08:25.943601+0000","last_peered":"2024-09-14T04:08:25.943601+0000","last_clean":"2024-09-14T04:08:25.943601+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:08:25.943601+0000","last_undegraded":"2024-09-14T04:08:25.943601+0000","last_fullsized":"2024-09-14T04:08:25.943601+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":288,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933722+0000","last_change":"2024-09-14T04:05:55.671536+0000","last_active":"2024-09-14T04:08:25.933722+0000","last_peered":"2024-09-14T04:08:25.933722+0000","last_clean":"2024-09-14T04:08:25.933722+0000","last_became_active":"2024-09-14T04:05:55.671029+0000","last_became_peered":"2024-09-14T04:05:55.671029+0000","last_unstale":"2024-09-14T04:08:25.933722+0000","last_undegraded":"2024-09-14T04:08:25.933722+0000","last_fullsized":"2024-09-14T04:08:25.933722+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:35.745580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":195,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943973+0000","last_change":"2024-09-14T04:05:55.681242+0000","last_active":"2024-09-14T04:08:25.943973+0000","last_peered":"2024-09-14T04:08:25.943973+0000","last_clean":"2024-09-14T04:08:25.943973+0000","last_became_active":"2024-09-14T04:05:55.680502+0000","last_became_peered":"2024-09-14T04:05:55.680502+0000","last_unstale":"2024-09-14T04:08:25.943973+0000","last_undegraded":"2024-09-14T04:08:25.943973+0000","last_fullsized":"2024-09-14T04:08:25.943973+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:23.760294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":188,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888097+0000","last_change":"2024-09-14T04:05:55.687662+0000","last_active":"2024-09-14T04:08:14.888097+0000","last_peered":"2024-09-14T04:08:14.888097+0000","last_clean":"2024-09-14T04:08:14.888097+0000","last_became_active":"2024-09-14T04:05:55.673561+0000","last_became_peered":"2024-09-14T04:05:55.673561+0000","last_unstale":"2024-09-14T04:08:14.888097+0000","last_undegraded":"2024-09-14T04:08:14.888097+0000","last_fullsized":"2024-09-14T04:08:14.888097+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:14:51.708356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":195,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.946763+0000","last_change":"2024-09-14T04:05:55.679885+0000","last_active":"2024-09-14T04:08:25.946763+0000","last_peered":"2024-09-14T04:08:25.946763+0000","last_clean":"2024-09-14T04:08:25.946763+0000","last_became_active":"2024-09-14T04:05:55.678589+0000","last_became_peered":"2024-09-14T04:05:55.678589+0000","last_unstale":"2024-09-14T04:08:25.946763+0000","last_undegraded":"2024-09-14T04:08:25.946763+0000","last_fullsized":"2024-09-14T04:08:25.946763+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:51:32.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":171,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888255+0000","last_change":"2024-09-14T04:05:55.697832+0000","last_active":"2024-09-14T04:08:14.888255+0000","last_peered":"2024-09-14T04:08:14.888255+0000","last_clean":"2024-09-14T04:08:14.888255+0000","last_became_active":"2024-09-14T04:05:55.673923+0000","last_became_peered":"2024-09-14T04:05:55.673923+0000","last_unstale":"2024-09-14T04:08:14.888255+0000","last_undegraded":"2024-09-14T04:08:14.888255+0000","last_fullsized":"2024-09-14T04:08:14.888255+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:44.697302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944212+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:08:25.944212+0000","last_peered":"2024-09-14T04:08:25.944212+0000","last_clean":"2024-09-14T04:08:25.944212+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:08:25.944212+0000","last_undegraded":"2024-09-14T04:08:25.944212+0000","last_fullsized":"2024-09-14T04:08:25.944212+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943947+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:08:25.943947+0000","last_peered":"2024-09-14T04:08:25.943947+0000","last_clean":"2024-09-14T04:08:25.943947+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:08:25.943947+0000","last_undegraded":"2024-09-14T04:08:25.943947+0000","last_fullsized":"2024-09-14T04:08:25.943947+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":123,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.891772+0000","last_change":"2024-09-14T04:07:44.338957+0000","last_active":"2024-09-14T04:08:14.891772+0000","last_peered":"2024-09-14T04:08:14.891772+0000","last_clean":"2024-09-14T04:08:14.891772+0000","last_became_active":"2024-09-14T04:07:44.338190+0000","last_became_peered":"2024-09-14T04:07:44.338190+0000","last_unstale":"2024-09-14T04:08:14.891772+0000","last_undegraded":"2024-09-14T04:08:14.891772+0000","last_fullsized":"2024-09-14T04:08:14.891772+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:36:27.488115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":293,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933806+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:08:25.933806+0000","last_peered":"2024-09-14T04:08:25.933806+0000","last_clean":"2024-09-14T04:08:25.933806+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:08:25.933806+0000","last_undegraded":"2024-09-14T04:08:25.933806+0000","last_fullsized":"2024-09-14T04:08:25.933806+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3422814208,"num_objects":52228,"num_object_clones":0,"num_object_copies":156684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52228,"num_write_kb":3342592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10276896768,"data_stored":10276896768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52228,"ondisk_log_size":52228,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4890,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430537,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1958260,"kb_used_data":1046888,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":91786892,"statfs":{"total":95995035648,"available":93989777408,"internally_reserved":0,"allocated":1072013312,"data_stored":1071032235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":933230936},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626766,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2074408,"kb_used_data":1476192,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":91670744,"statfs":{"total":95995035648,"available":93870841856,"internally_reserved":0,"allocated":1511620608,"data_stored":1510647315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822994,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2166996,"kb_used_data":1270028,"kb_used_omap":1,"kb_used_meta":896958,"kb_avail":91578156,"statfs":{"total":95995035648,"available":93776031744,"internally_reserved":0,"allocated":1300508672,"data_stored":1299550792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918485336},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":798863917076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552375,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2124756,"kb_used_data":1711180,"kb_used_omap":4,"kb_used_meta":413563,"kb_avail":91620396,"statfs":{"total":95995035648,"available":93819285504,"internally_reserved":0,"allocated":1752248320,"data_stored":1751290521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":423488957},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435416,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1525852,"kb_used_data":1166164,"kb_used_omap":6,"kb_used_meta":359673,"kb_avail":92219300,"statfs":{"total":95995035648,"available":94432563200,"internally_reserved":0,"allocated":1194151936,"data_stored":1193189741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":368305613},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827886,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1857368,"kb_used_data":1275920,"kb_used_omap":5,"kb_used_meta":581434,"kb_avail":91887784,"statfs":{"total":95995035648,"available":94093090816,"internally_reserved":0,"allocated":1306542080,"data_stored":1305580136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":595388655},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383874,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2645852,"kb_used_data":2041556,"kb_used_omap":5,"kb_used_meta":604218,"kb_avail":91099300,"statfs":{"total":95995035648,"available":93285683200,"internally_reserved":0,"allocated":2090553344,"data_stored":2089588190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":618719292},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2094661632,"data_stored":2094661632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1310785536,"data_stored":1310785536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1203830784,"data_stored":1203830784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1754071040,"data_stored":1754071040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1311047680,"data_stored":1311047680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1514078208,"data_stored":1514078208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1088421888,"data_stored":1088421888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:08:38.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:08:38.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:38.451 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:38.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:38.517 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:08:38.518 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:08:38.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:38.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:38.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:38.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:38.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:38.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:38.724 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:38.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:38.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:38.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:38.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:38.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:39.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:39.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:39.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:39.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:39.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:39.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:39.234 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 66451 66435 115.314 70.25 0.0188733 0.00866661 2024-09-14T04:08:39.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:39.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:39.298 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:39.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:39.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:39.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:39.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:39.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:39.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:39.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:39.586 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:08:39.586 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:08:39.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-14T04:08:39.621 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:08:39.621 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:08:39.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:39.657 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:08:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:08:39.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:39.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:39.698 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:08:39.698 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:08:39.729 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:08:39.729 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:08:39.759 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:08:39.760 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:08:39.793 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:08:39.794 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:08:39.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:39.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:39.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:39.837 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:08:39.837 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:08:39.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:39.938 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:39.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:39.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:40.009 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:08:40.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:40.092 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:40.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:40.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:40.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:40.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:40.234 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 67662 67646 114.243 75.6875 0.0163314 0.00874809 2024-09-14T04:08:40.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:40.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:40.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:40 smithi114 ceph-mon[106969]: pgmap v726: 89 pgs: 89 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-14T04:08:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2282198100' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:40 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3788717451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:40.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:40 smithi114 ceph-mon[103029]: pgmap v726: 89 pgs: 89 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-14T04:08:40.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2282198100' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:40.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:40 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:40.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3788717451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:40.402 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:08:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:08:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:08:40.403 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:40 smithi160 ceph-mon[88407]: pgmap v726: 89 pgs: 89 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-14T04:08:40.403 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2282198100' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:40.403 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:40 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:08:40.403 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3788717451' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:40.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:40.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:40.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:40.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:40.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:40.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:40.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:40.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:40.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:40.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:40.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:40.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:40.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:40.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:41.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:41.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:41.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:41.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:41.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:41 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:08:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:41 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:08:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:41 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:08:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:41 smithi160 ceph-mon[88407]: osdmap e207: 8 total, 8 up, 7 in 2024-09-14T04:08:41.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:41.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 68718 68702 112.973 66 0.0143641 0.00884687 2024-09-14T04:08:41.273 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:41.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:41.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:41.325 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:41 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:08:41.325 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:41 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:08:41.326 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:41 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:08:41.326 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:41 smithi114 ceph-mon[106969]: osdmap e207: 8 total, 8 up, 7 in 2024-09-14T04:08:41.326 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:41 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:08:41.326 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:41 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:08:41.326 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:41 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:08:41.326 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:41 smithi114 ceph-mon[103029]: osdmap e207: 8 total, 8 up, 7 in 2024-09-14T04:08:41.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:41.411 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:41.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:41.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:41.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:41.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:41.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:08:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:08:41] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-09-14T04:08:41.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:41.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:41.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:41.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:41.822 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:41.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:41.841 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:08:41.842 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:08:41.842 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:08:41.842 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:08:41.842 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:08:41.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:41.968 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:41.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:42.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:42.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:42.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:42 smithi160 ceph-mon[88407]: pgmap v728: 89 pgs: 89 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-14T04:08:42.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:42.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:42.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:42.234 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 70082 70066 112.262 85.25 0.00764193 0.00890273 2024-09-14T04:08:42.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:42.357 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:42 smithi114 ceph-mon[106969]: pgmap v728: 89 pgs: 89 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-14T04:08:42.360 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:42.360 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:42 smithi114 ceph-mon[103029]: pgmap v728: 89 pgs: 89 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-14T04:08:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:42.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:42.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:42.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:42.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:42.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:42.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-14T04:08:42.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:42.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:42.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:42.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:42.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:42.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:42.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:42.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:42.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:42.962 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:43.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:43.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:43.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:43.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:43.171 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:43.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:43.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:43.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:43.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:08:43.234750+0000 min lat: 0.00109236 max lat: 0.268444 avg lat: 0.00896097 2024-09-14T04:08:43.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:08:43.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 71409 71393 111.529 82.9375 0.0109726 0.00896097 2024-09-14T04:08:43.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:43.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:43.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:43.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:43.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:43.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:43.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:43.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:43.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:43.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:43.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:43.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:43.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:43.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:43.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:43.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:43.986 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:44.025 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:44.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:44.113 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:08:44.114 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:08:44.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:44.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:44.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:44.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 72604 72588 110.63 74.6875 0.00593709 0.00903413 2024-09-14T04:08:44.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:44.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:44.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:44.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:44 smithi114 ceph-mon[103029]: pgmap v729: 89 pgs: 89 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-14T04:08:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:44 smithi114 ceph-mon[106969]: pgmap v729: 89 pgs: 89 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-14T04:08:44.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:44.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:44 smithi160 ceph-mon[88407]: pgmap v729: 89 pgs: 89 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-14T04:08:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:44.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:44.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:44.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:44.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:44.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:44.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:44.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:44.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:44.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:44.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:44.830 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:44.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:44.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:44.969 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2024-09-14T04:08:42.736272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4345173965,"num_objects":66319,"num_object_clones":0,"num_object_copies":198957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66319,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":66505,"num_write_kb":4244411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":595258,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66451,"ondisk_log_size":66451,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17098464,"kb_used_data":12698276,"kb_used_omap":27,"kb_used_meta":4400100,"kb_avail":639117600,"statfs":{"total":671965249536,"available":654456422400,"internally_reserved":0,"allocated":13003034624,"data_stored":12996271297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28483,"internal_metadata":4505702589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":861995008,"num_objects":13153,"num_object_clones":0,"num_object_copies":39459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":841792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002800"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":219,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944367+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:08:25.944367+0000","last_peered":"2024-09-14T04:08:25.944367+0000","last_clean":"2024-09-14T04:08:25.944367+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:08:25.944367+0000","last_undegraded":"2024-09-14T04:08:25.944367+0000","last_fullsized":"2024-09-14T04:08:25.944367+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":189,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894375+0000","last_change":"2024-09-14T04:05:55.701296+0000","last_active":"2024-09-14T04:08:14.894375+0000","last_peered":"2024-09-14T04:08:14.894375+0000","last_clean":"2024-09-14T04:08:14.894375+0000","last_became_active":"2024-09-14T04:05:55.676634+0000","last_became_peered":"2024-09-14T04:05:55.676634+0000","last_unstale":"2024-09-14T04:08:14.894375+0000","last_undegraded":"2024-09-14T04:08:14.894375+0000","last_fullsized":"2024-09-14T04:08:14.894375+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:44.187804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":171,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888168+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:08:14.888168+0000","last_peered":"2024-09-14T04:08:14.888168+0000","last_clean":"2024-09-14T04:08:14.888168+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:08:14.888168+0000","last_undegraded":"2024-09-14T04:08:14.888168+0000","last_fullsized":"2024-09-14T04:08:14.888168+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":303,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.937038+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:08:25.937038+0000","last_peered":"2024-09-14T04:08:25.937038+0000","last_clean":"2024-09-14T04:08:25.937038+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:08:25.937038+0000","last_undegraded":"2024-09-14T04:08:25.937038+0000","last_fullsized":"2024-09-14T04:08:25.937038+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":289,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.938315+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:08:25.938315+0000","last_peered":"2024-09-14T04:08:25.938315+0000","last_clean":"2024-09-14T04:08:25.938315+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:08:25.938315+0000","last_undegraded":"2024-09-14T04:08:25.938315+0000","last_fullsized":"2024-09-14T04:08:25.938315+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":164,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.896040+0000","last_change":"2024-09-14T04:05:55.676850+0000","last_active":"2024-09-14T04:08:14.896040+0000","last_peered":"2024-09-14T04:08:14.896040+0000","last_clean":"2024-09-14T04:08:14.896040+0000","last_became_active":"2024-09-14T04:05:55.676337+0000","last_became_peered":"2024-09-14T04:05:55.676337+0000","last_unstale":"2024-09-14T04:08:14.896040+0000","last_undegraded":"2024-09-14T04:08:14.896040+0000","last_fullsized":"2024-09-14T04:08:14.896040+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:12.377665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.945978+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:08:25.945978+0000","last_peered":"2024-09-14T04:08:25.945978+0000","last_clean":"2024-09-14T04:08:25.945978+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:08:25.945978+0000","last_undegraded":"2024-09-14T04:08:25.945978+0000","last_fullsized":"2024-09-14T04:08:25.945978+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":180,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.891988+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:08:14.891988+0000","last_peered":"2024-09-14T04:08:14.891988+0000","last_clean":"2024-09-14T04:08:14.891988+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:08:14.891988+0000","last_undegraded":"2024-09-14T04:08:14.891988+0000","last_fullsized":"2024-09-14T04:08:14.891988+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"206'4109","reported_seq":4124,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:39.248022+0000","last_change":"2024-09-14T04:07:55.368334+0000","last_active":"2024-09-14T04:08:39.248022+0000","last_peered":"2024-09-14T04:08:39.248022+0000","last_clean":"2024-09-14T04:08:39.248022+0000","last_became_active":"2024-09-14T04:07:55.368016+0000","last_became_peered":"2024-09-14T04:07:55.368016+0000","last_unstale":"2024-09-14T04:08:39.248022+0000","last_undegraded":"2024-09-14T04:08:39.248022+0000","last_fullsized":"2024-09-14T04:08:39.248022+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:13.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269287424,"num_objects":4109,"num_object_clones":0,"num_object_copies":12327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4109,"num_write_kb":262976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":186,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.895972+0000","last_change":"2024-09-14T04:05:55.704897+0000","last_active":"2024-09-14T04:08:14.895972+0000","last_peered":"2024-09-14T04:08:14.895972+0000","last_clean":"2024-09-14T04:08:14.895972+0000","last_became_active":"2024-09-14T04:05:55.681093+0000","last_became_peered":"2024-09-14T04:05:55.681093+0000","last_unstale":"2024-09-14T04:08:14.895972+0000","last_undegraded":"2024-09-14T04:08:14.895972+0000","last_fullsized":"2024-09-14T04:08:14.895972+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:49.647801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":154,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.887865+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:08:14.887865+0000","last_peered":"2024-09-14T04:08:14.887865+0000","last_clean":"2024-09-14T04:08:14.887865+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:08:14.887865+0000","last_undegraded":"2024-09-14T04:08:14.887865+0000","last_fullsized":"2024-09-14T04:08:14.887865+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"206'4178","reported_seq":4193,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:38.670985+0000","last_change":"2024-09-14T04:07:55.607517+0000","last_active":"2024-09-14T04:08:38.670985+0000","last_peered":"2024-09-14T04:08:38.670985+0000","last_clean":"2024-09-14T04:08:38.670985+0000","last_became_active":"2024-09-14T04:07:55.607208+0000","last_became_peered":"2024-09-14T04:07:55.607208+0000","last_unstale":"2024-09-14T04:08:38.670985+0000","last_undegraded":"2024-09-14T04:08:38.670985+0000","last_fullsized":"2024-09-14T04:08:38.670985+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4178,"log_dups_size":0,"ondisk_log_size":4178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:32:20.123209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273809408,"num_objects":4178,"num_object_clones":0,"num_object_copies":12534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4178,"num_write_kb":267392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":187,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.896126+0000","last_change":"2024-09-14T04:05:55.702981+0000","last_active":"2024-09-14T04:08:14.896126+0000","last_peered":"2024-09-14T04:08:14.896126+0000","last_clean":"2024-09-14T04:08:14.896126+0000","last_became_active":"2024-09-14T04:05:55.679789+0000","last_became_peered":"2024-09-14T04:05:55.679789+0000","last_unstale":"2024-09-14T04:08:14.896126+0000","last_undegraded":"2024-09-14T04:08:14.896126+0000","last_fullsized":"2024-09-14T04:08:14.896126+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:34:24.358644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":209,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.930854+0000","last_change":"2024-09-14T04:07:44.340689+0000","last_active":"2024-09-14T04:08:25.930854+0000","last_peered":"2024-09-14T04:08:25.930854+0000","last_clean":"2024-09-14T04:08:25.930854+0000","last_became_active":"2024-09-14T04:07:44.340112+0000","last_became_peered":"2024-09-14T04:07:44.340112+0000","last_unstale":"2024-09-14T04:08:25.930854+0000","last_undegraded":"2024-09-14T04:08:25.930854+0000","last_fullsized":"2024-09-14T04:08:25.930854+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:04.263233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"206'4023","reported_seq":4048,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:37.855697+0000","last_change":"2024-09-14T04:07:55.608111+0000","last_active":"2024-09-14T04:08:37.855697+0000","last_peered":"2024-09-14T04:08:37.855697+0000","last_clean":"2024-09-14T04:08:37.855697+0000","last_became_active":"2024-09-14T04:07:55.607646+0000","last_became_peered":"2024-09-14T04:07:55.607646+0000","last_unstale":"2024-09-14T04:08:37.855697+0000","last_undegraded":"2024-09-14T04:08:37.855697+0000","last_fullsized":"2024-09-14T04:08:37.855697+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4023,"log_dups_size":0,"ondisk_log_size":4023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:02:33.076539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263651328,"num_objects":4023,"num_object_clones":0,"num_object_copies":12069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4023,"num_write_kb":257472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":155,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894734+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:08:14.894734+0000","last_peered":"2024-09-14T04:08:14.894734+0000","last_clean":"2024-09-14T04:08:14.894734+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:08:14.894734+0000","last_undegraded":"2024-09-14T04:08:14.894734+0000","last_fullsized":"2024-09-14T04:08:14.894734+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":181,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.892174+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:08:14.892174+0000","last_peered":"2024-09-14T04:08:14.892174+0000","last_clean":"2024-09-14T04:08:14.892174+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:08:14.892174+0000","last_undegraded":"2024-09-14T04:08:14.892174+0000","last_fullsized":"2024-09-14T04:08:14.892174+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"207'4315","reported_seq":4342,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-14T04:08:41.379128+0000","last_change":"2024-09-14T04:07:55.607545+0000","last_active":"2024-09-14T04:08:41.379128+0000","last_peered":"2024-09-14T04:08:41.379128+0000","last_clean":"2024-09-14T04:08:41.379128+0000","last_became_active":"2024-09-14T04:07:55.607263+0000","last_became_peered":"2024-09-14T04:07:55.607263+0000","last_unstale":"2024-09-14T04:08:41.379128+0000","last_undegraded":"2024-09-14T04:08:41.379128+0000","last_fullsized":"2024-09-14T04:08:41.379128+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:38:52.910168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282787840,"num_objects":4315,"num_object_clones":0,"num_object_copies":12945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4315,"num_write_kb":276160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":192,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888174+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:08:14.888174+0000","last_peered":"2024-09-14T04:08:14.888174+0000","last_clean":"2024-09-14T04:08:14.888174+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:08:14.888174+0000","last_undegraded":"2024-09-14T04:08:14.888174+0000","last_fullsized":"2024-09-14T04:08:14.888174+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":318,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933574+0000","last_change":"2024-09-14T04:07:54.363033+0000","last_active":"2024-09-14T04:08:25.933574+0000","last_peered":"2024-09-14T04:08:25.933574+0000","last_clean":"2024-09-14T04:08:25.933574+0000","last_became_active":"2024-09-14T04:07:54.362706+0000","last_became_peered":"2024-09-14T04:07:54.362706+0000","last_unstale":"2024-09-14T04:08:25.933574+0000","last_undegraded":"2024-09-14T04:08:25.933574+0000","last_fullsized":"2024-09-14T04:08:25.933574+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:31.193597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"206'4201","reported_seq":4226,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:39.421354+0000","last_change":"2024-09-14T04:07:55.607327+0000","last_active":"2024-09-14T04:08:39.421354+0000","last_peered":"2024-09-14T04:08:39.421354+0000","last_clean":"2024-09-14T04:08:39.421354+0000","last_became_active":"2024-09-14T04:07:55.606967+0000","last_became_peered":"2024-09-14T04:07:55.606967+0000","last_unstale":"2024-09-14T04:08:39.421354+0000","last_undegraded":"2024-09-14T04:08:39.421354+0000","last_fullsized":"2024-09-14T04:08:39.421354+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:36.618823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275316736,"num_objects":4201,"num_object_clones":0,"num_object_copies":12603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4201,"num_write_kb":268864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":258,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944030+0000","last_change":"2024-09-14T04:05:55.695825+0000","last_active":"2024-09-14T04:08:25.944030+0000","last_peered":"2024-09-14T04:08:25.944030+0000","last_clean":"2024-09-14T04:08:25.944030+0000","last_became_active":"2024-09-14T04:05:55.680668+0000","last_became_peered":"2024-09-14T04:05:55.680668+0000","last_unstale":"2024-09-14T04:08:25.944030+0000","last_undegraded":"2024-09-14T04:08:25.944030+0000","last_fullsized":"2024-09-14T04:08:25.944030+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:30.262947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":178,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944437+0000","last_change":"2024-09-14T04:05:55.677723+0000","last_active":"2024-09-14T04:08:25.944437+0000","last_peered":"2024-09-14T04:08:25.944437+0000","last_clean":"2024-09-14T04:08:25.944437+0000","last_became_active":"2024-09-14T04:05:55.677254+0000","last_became_peered":"2024-09-14T04:05:55.677254+0000","last_unstale":"2024-09-14T04:08:25.944437+0000","last_undegraded":"2024-09-14T04:08:25.944437+0000","last_fullsized":"2024-09-14T04:08:25.944437+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:53.163120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"207'4304","reported_seq":4331,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-14T04:08:41.407882+0000","last_change":"2024-09-14T04:07:55.607260+0000","last_active":"2024-09-14T04:08:41.407882+0000","last_peered":"2024-09-14T04:08:41.407882+0000","last_clean":"2024-09-14T04:08:41.407882+0000","last_became_active":"2024-09-14T04:07:55.606973+0000","last_became_peered":"2024-09-14T04:07:55.606973+0000","last_unstale":"2024-09-14T04:08:41.407882+0000","last_undegraded":"2024-09-14T04:08:41.407882+0000","last_fullsized":"2024-09-14T04:08:41.407882+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:59:15.715573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282066944,"num_objects":4304,"num_object_clones":0,"num_object_copies":12912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4304,"num_write_kb":275456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":182,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.892291+0000","last_change":"2024-09-14T04:05:55.679201+0000","last_active":"2024-09-14T04:08:14.892291+0000","last_peered":"2024-09-14T04:08:14.892291+0000","last_clean":"2024-09-14T04:08:14.892291+0000","last_became_active":"2024-09-14T04:05:55.678843+0000","last_became_peered":"2024-09-14T04:05:55.678843+0000","last_unstale":"2024-09-14T04:08:14.892291+0000","last_undegraded":"2024-09-14T04:08:14.892291+0000","last_fullsized":"2024-09-14T04:08:14.892291+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:12:57.418514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":171,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-14T04:08:40.017673+0000","last_change":"2024-09-14T04:05:55.673650+0000","last_active":"2024-09-14T04:08:40.017673+0000","last_peered":"2024-09-14T04:08:40.017673+0000","last_clean":"2024-09-14T04:08:40.017673+0000","last_became_active":"2024-09-14T04:05:55.673258+0000","last_became_peered":"2024-09-14T04:05:55.673258+0000","last_unstale":"2024-09-14T04:08:40.017673+0000","last_undegraded":"2024-09-14T04:08:40.017673+0000","last_fullsized":"2024-09-14T04:08:40.017673+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:21.089797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"206'4140","reported_seq":4155,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:38.710515+0000","last_change":"2024-09-14T04:07:55.365272+0000","last_active":"2024-09-14T04:08:38.710515+0000","last_peered":"2024-09-14T04:08:38.710515+0000","last_clean":"2024-09-14T04:08:38.710515+0000","last_became_active":"2024-09-14T04:07:55.364722+0000","last_became_peered":"2024-09-14T04:07:55.364722+0000","last_unstale":"2024-09-14T04:08:38.710515+0000","last_undegraded":"2024-09-14T04:08:38.710515+0000","last_fullsized":"2024-09-14T04:08:38.710515+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:40:55.470085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271319040,"num_objects":4140,"num_object_clones":0,"num_object_copies":12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4140,"num_write_kb":264960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":155,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-14T04:08:40.017209+0000","last_change":"2024-09-14T04:05:55.671329+0000","last_active":"2024-09-14T04:08:40.017209+0000","last_peered":"2024-09-14T04:08:40.017209+0000","last_clean":"2024-09-14T04:08:40.017209+0000","last_became_active":"2024-09-14T04:05:55.671032+0000","last_became_peered":"2024-09-14T04:05:55.671032+0000","last_unstale":"2024-09-14T04:08:40.017209+0000","last_undegraded":"2024-09-14T04:08:40.017209+0000","last_fullsized":"2024-09-14T04:08:40.017209+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:58:10.779452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":103,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894352+0000","last_change":"2024-09-14T04:05:55.679870+0000","last_active":"2024-09-14T04:08:14.894352+0000","last_peered":"2024-09-14T04:08:14.894352+0000","last_clean":"2024-09-14T04:08:14.894352+0000","last_became_active":"2024-09-14T04:05:55.679574+0000","last_became_peered":"2024-09-14T04:05:55.679574+0000","last_unstale":"2024-09-14T04:08:14.894352+0000","last_undegraded":"2024-09-14T04:08:14.894352+0000","last_fullsized":"2024-09-14T04:08:14.894352+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:51:09.931574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"206'4075","reported_seq":4089,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:39.061624+0000","last_change":"2024-09-14T04:07:55.367126+0000","last_active":"2024-09-14T04:08:39.061624+0000","last_peered":"2024-09-14T04:08:39.061624+0000","last_clean":"2024-09-14T04:08:39.061624+0000","last_became_active":"2024-09-14T04:07:55.366798+0000","last_became_peered":"2024-09-14T04:07:55.366798+0000","last_unstale":"2024-09-14T04:08:39.061624+0000","last_undegraded":"2024-09-14T04:08:39.061624+0000","last_fullsized":"2024-09-14T04:08:39.061624+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:24:51.791784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267059200,"num_objects":4075,"num_object_clones":0,"num_object_copies":12225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4075,"num_write_kb":260800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943605+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:08:25.943605+0000","last_peered":"2024-09-14T04:08:25.943605+0000","last_clean":"2024-09-14T04:08:25.943605+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:08:25.943605+0000","last_undegraded":"2024-09-14T04:08:25.943605+0000","last_fullsized":"2024-09-14T04:08:25.943605+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":262,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.931383+0000","last_change":"2024-09-14T04:05:55.678520+0000","last_active":"2024-09-14T04:08:25.931383+0000","last_peered":"2024-09-14T04:08:25.931383+0000","last_clean":"2024-09-14T04:08:25.931383+0000","last_became_active":"2024-09-14T04:05:55.678008+0000","last_became_peered":"2024-09-14T04:05:55.678008+0000","last_unstale":"2024-09-14T04:08:25.931383+0000","last_undegraded":"2024-09-14T04:08:25.931383+0000","last_fullsized":"2024-09-14T04:08:25.931383+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:40:26.778212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"206'4004","reported_seq":4019,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:38.704884+0000","last_change":"2024-09-14T04:07:55.367659+0000","last_active":"2024-09-14T04:08:38.704884+0000","last_peered":"2024-09-14T04:08:38.704884+0000","last_clean":"2024-09-14T04:08:38.704884+0000","last_became_active":"2024-09-14T04:07:55.367247+0000","last_became_peered":"2024-09-14T04:07:55.367247+0000","last_unstale":"2024-09-14T04:08:38.704884+0000","last_undegraded":"2024-09-14T04:08:38.704884+0000","last_fullsized":"2024-09-14T04:08:38.704884+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:58:08.638370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262406144,"num_objects":4004,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4004,"num_write_kb":256256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":288,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.899168+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:08:14.899168+0000","last_peered":"2024-09-14T04:08:14.899168+0000","last_clean":"2024-09-14T04:08:14.899168+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:08:14.899168+0000","last_undegraded":"2024-09-14T04:08:14.899168+0000","last_fullsized":"2024-09-14T04:08:14.899168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":185,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888092+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:08:14.888092+0000","last_peered":"2024-09-14T04:08:14.888092+0000","last_clean":"2024-09-14T04:08:14.888092+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:08:14.888092+0000","last_undegraded":"2024-09-14T04:08:14.888092+0000","last_fullsized":"2024-09-14T04:08:14.888092+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.946536+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:08:25.946536+0000","last_peered":"2024-09-14T04:08:25.946536+0000","last_clean":"2024-09-14T04:08:25.946536+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:08:25.946536+0000","last_undegraded":"2024-09-14T04:08:25.946536+0000","last_fullsized":"2024-09-14T04:08:25.946536+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":164,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894091+0000","last_change":"2024-09-14T04:05:55.676445+0000","last_active":"2024-09-14T04:08:14.894091+0000","last_peered":"2024-09-14T04:08:14.894091+0000","last_clean":"2024-09-14T04:08:14.894091+0000","last_became_active":"2024-09-14T04:05:55.674141+0000","last_became_peered":"2024-09-14T04:05:55.674141+0000","last_unstale":"2024-09-14T04:08:14.894091+0000","last_undegraded":"2024-09-14T04:08:14.894091+0000","last_fullsized":"2024-09-14T04:08:14.894091+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:28:50.348309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"206'4108","reported_seq":4132,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:37.853659+0000","last_change":"2024-09-14T04:07:55.366975+0000","last_active":"2024-09-14T04:08:37.853659+0000","last_peered":"2024-09-14T04:08:37.853659+0000","last_clean":"2024-09-14T04:08:37.853659+0000","last_became_active":"2024-09-14T04:07:55.366645+0000","last_became_peered":"2024-09-14T04:07:55.366645+0000","last_unstale":"2024-09-14T04:08:37.853659+0000","last_undegraded":"2024-09-14T04:08:37.853659+0000","last_fullsized":"2024-09-14T04:08:37.853659+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4108,"log_dups_size":0,"ondisk_log_size":4108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:39:56.637518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269221888,"num_objects":4108,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4108,"num_write_kb":262912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":200,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.899179+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:08:14.899179+0000","last_peered":"2024-09-14T04:08:14.899179+0000","last_clean":"2024-09-14T04:08:14.899179+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:08:14.899179+0000","last_undegraded":"2024-09-14T04:08:14.899179+0000","last_fullsized":"2024-09-14T04:08:14.899179+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":311,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.934219+0000","last_change":"2024-09-14T04:07:44.337314+0000","last_active":"2024-09-14T04:08:25.934219+0000","last_peered":"2024-09-14T04:08:25.934219+0000","last_clean":"2024-09-14T04:08:25.934219+0000","last_became_active":"2024-09-14T04:07:44.336916+0000","last_became_peered":"2024-09-14T04:07:44.336916+0000","last_unstale":"2024-09-14T04:08:25.934219+0000","last_undegraded":"2024-09-14T04:08:25.934219+0000","last_fullsized":"2024-09-14T04:08:25.934219+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:23.153578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":301,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.938064+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:08:25.938064+0000","last_peered":"2024-09-14T04:08:25.938064+0000","last_clean":"2024-09-14T04:08:25.938064+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:08:25.938064+0000","last_undegraded":"2024-09-14T04:08:25.938064+0000","last_fullsized":"2024-09-14T04:08:25.938064+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"206'4171","reported_seq":4186,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:39.273836+0000","last_change":"2024-09-14T04:07:55.365448+0000","last_active":"2024-09-14T04:08:39.273836+0000","last_peered":"2024-09-14T04:08:39.273836+0000","last_clean":"2024-09-14T04:08:39.273836+0000","last_became_active":"2024-09-14T04:07:55.365168+0000","last_became_peered":"2024-09-14T04:07:55.365168+0000","last_unstale":"2024-09-14T04:08:39.273836+0000","last_undegraded":"2024-09-14T04:08:39.273836+0000","last_fullsized":"2024-09-14T04:08:39.273836+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:25.997258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273350656,"num_objects":4171,"num_object_clones":0,"num_object_copies":12513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4171,"num_write_kb":266944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":297,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-14T04:08:40.019192+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:08:40.019192+0000","last_peered":"2024-09-14T04:08:40.019192+0000","last_clean":"2024-09-14T04:08:40.019192+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:08:40.019192+0000","last_undegraded":"2024-09-14T04:08:40.019192+0000","last_fullsized":"2024-09-14T04:08:40.019192+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":256,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933187+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:08:25.933187+0000","last_peered":"2024-09-14T04:08:25.933187+0000","last_clean":"2024-09-14T04:08:25.933187+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:08:25.933187+0000","last_undegraded":"2024-09-14T04:08:25.933187+0000","last_fullsized":"2024-09-14T04:08:25.933187+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":28,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-14T04:08:40.016605+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:08:40.016605+0000","last_peered":"2024-09-14T04:08:40.016605+0000","last_clean":"2024-09-14T04:08:40.016605+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:08:40.016605+0000","last_undegraded":"2024-09-14T04:08:40.016605+0000","last_fullsized":"2024-09-14T04:08:40.016605+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"206'4151","reported_seq":4166,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:38.663742+0000","last_change":"2024-09-14T04:07:55.367851+0000","last_active":"2024-09-14T04:08:38.663742+0000","last_peered":"2024-09-14T04:08:38.663742+0000","last_clean":"2024-09-14T04:08:38.663742+0000","last_became_active":"2024-09-14T04:07:55.367146+0000","last_became_peered":"2024-09-14T04:07:55.367146+0000","last_unstale":"2024-09-14T04:08:38.663742+0000","last_undegraded":"2024-09-14T04:08:38.663742+0000","last_fullsized":"2024-09-14T04:08:38.663742+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4151,"log_dups_size":0,"ondisk_log_size":4151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:26:46.495508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272039936,"num_objects":4151,"num_object_clones":0,"num_object_copies":12453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4151,"num_write_kb":265664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":186,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888404+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:08:14.888404+0000","last_peered":"2024-09-14T04:08:14.888404+0000","last_clean":"2024-09-14T04:08:14.888404+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:08:14.888404+0000","last_undegraded":"2024-09-14T04:08:14.888404+0000","last_fullsized":"2024-09-14T04:08:14.888404+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":124,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933876+0000","last_change":"2024-09-14T04:05:55.676693+0000","last_active":"2024-09-14T04:08:25.933876+0000","last_peered":"2024-09-14T04:08:25.933876+0000","last_clean":"2024-09-14T04:08:25.933876+0000","last_became_active":"2024-09-14T04:05:55.676335+0000","last_became_peered":"2024-09-14T04:05:55.676335+0000","last_unstale":"2024-09-14T04:08:25.933876+0000","last_undegraded":"2024-09-14T04:08:25.933876+0000","last_fullsized":"2024-09-14T04:08:25.933876+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:19:43.626016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":280,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.932785+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:08:25.932785+0000","last_peered":"2024-09-14T04:08:25.932785+0000","last_clean":"2024-09-14T04:08:25.932785+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:08:25.932785+0000","last_undegraded":"2024-09-14T04:08:25.932785+0000","last_fullsized":"2024-09-14T04:08:25.932785+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"206'4135","reported_seq":4150,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:39.057836+0000","last_change":"2024-09-14T04:07:55.607796+0000","last_active":"2024-09-14T04:08:39.057836+0000","last_peered":"2024-09-14T04:08:39.057836+0000","last_clean":"2024-09-14T04:08:39.057836+0000","last_became_active":"2024-09-14T04:07:55.607447+0000","last_became_peered":"2024-09-14T04:07:55.607447+0000","last_unstale":"2024-09-14T04:08:39.057836+0000","last_undegraded":"2024-09-14T04:08:39.057836+0000","last_fullsized":"2024-09-14T04:08:39.057836+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:09:44.307466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270991360,"num_objects":4135,"num_object_clones":0,"num_object_copies":12405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4135,"num_write_kb":264640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":209,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888302+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:08:14.888302+0000","last_peered":"2024-09-14T04:08:14.888302+0000","last_clean":"2024-09-14T04:08:14.888302+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:08:14.888302+0000","last_undegraded":"2024-09-14T04:08:14.888302+0000","last_fullsized":"2024-09-14T04:08:14.888302+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":294,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.932884+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:08:25.932884+0000","last_peered":"2024-09-14T04:08:25.932884+0000","last_clean":"2024-09-14T04:08:25.932884+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:08:25.932884+0000","last_undegraded":"2024-09-14T04:08:25.932884+0000","last_fullsized":"2024-09-14T04:08:25.932884+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":251,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.931142+0000","last_change":"2024-09-14T04:05:55.674511+0000","last_active":"2024-09-14T04:08:25.931142+0000","last_peered":"2024-09-14T04:08:25.931142+0000","last_clean":"2024-09-14T04:08:25.931142+0000","last_became_active":"2024-09-14T04:05:55.674182+0000","last_became_peered":"2024-09-14T04:05:55.674182+0000","last_unstale":"2024-09-14T04:08:25.931142+0000","last_undegraded":"2024-09-14T04:08:25.931142+0000","last_fullsized":"2024-09-14T04:08:25.931142+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:12:34.453647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"206'4110","reported_seq":4125,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:38.724714+0000","last_change":"2024-09-14T04:07:55.607679+0000","last_active":"2024-09-14T04:08:38.724714+0000","last_peered":"2024-09-14T04:08:38.724714+0000","last_clean":"2024-09-14T04:08:38.724714+0000","last_became_active":"2024-09-14T04:07:55.607338+0000","last_became_peered":"2024-09-14T04:07:55.607338+0000","last_unstale":"2024-09-14T04:08:38.724714+0000","last_undegraded":"2024-09-14T04:08:38.724714+0000","last_fullsized":"2024-09-14T04:08:38.724714+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:02:20.945987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269352960,"num_objects":4110,"num_object_clones":0,"num_object_copies":12330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4110,"num_write_kb":263040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":255,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933016+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:08:25.933016+0000","last_peered":"2024-09-14T04:08:25.933016+0000","last_clean":"2024-09-14T04:08:25.933016+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:08:25.933016+0000","last_undegraded":"2024-09-14T04:08:25.933016+0000","last_fullsized":"2024-09-14T04:08:25.933016+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":193,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894945+0000","last_change":"2024-09-14T04:05:55.681238+0000","last_active":"2024-09-14T04:08:14.894945+0000","last_peered":"2024-09-14T04:08:14.894945+0000","last_clean":"2024-09-14T04:08:14.894945+0000","last_became_active":"2024-09-14T04:05:55.680946+0000","last_became_peered":"2024-09-14T04:05:55.680946+0000","last_unstale":"2024-09-14T04:08:14.894945+0000","last_undegraded":"2024-09-14T04:08:14.894945+0000","last_fullsized":"2024-09-14T04:08:14.894945+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:13:39.006545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":279,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933651+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:08:25.933651+0000","last_peered":"2024-09-14T04:08:25.933651+0000","last_clean":"2024-09-14T04:08:25.933651+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:08:25.933651+0000","last_undegraded":"2024-09-14T04:08:25.933651+0000","last_fullsized":"2024-09-14T04:08:25.933651+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"206'4124","reported_seq":4139,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:38.683642+0000","last_change":"2024-09-14T04:07:55.366077+0000","last_active":"2024-09-14T04:08:38.683642+0000","last_peered":"2024-09-14T04:08:38.683642+0000","last_clean":"2024-09-14T04:08:38.683642+0000","last_became_active":"2024-09-14T04:07:55.365753+0000","last_became_peered":"2024-09-14T04:07:55.365753+0000","last_unstale":"2024-09-14T04:08:38.683642+0000","last_undegraded":"2024-09-14T04:08:38.683642+0000","last_fullsized":"2024-09-14T04:08:38.683642+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4124,"log_dups_size":0,"ondisk_log_size":4124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:04:18.784725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270270464,"num_objects":4124,"num_object_clones":0,"num_object_copies":12372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4124,"num_write_kb":263936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":247,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.946122+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:08:25.946122+0000","last_peered":"2024-09-14T04:08:25.946122+0000","last_clean":"2024-09-14T04:08:25.946122+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:08:25.946122+0000","last_undegraded":"2024-09-14T04:08:25.946122+0000","last_fullsized":"2024-09-14T04:08:25.946122+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":262,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.930955+0000","last_change":"2024-09-14T04:05:55.676617+0000","last_active":"2024-09-14T04:08:25.930955+0000","last_peered":"2024-09-14T04:08:25.930955+0000","last_clean":"2024-09-14T04:08:25.930955+0000","last_became_active":"2024-09-14T04:05:55.676318+0000","last_became_peered":"2024-09-14T04:05:55.676318+0000","last_unstale":"2024-09-14T04:08:25.930955+0000","last_undegraded":"2024-09-14T04:08:25.930955+0000","last_fullsized":"2024-09-14T04:08:25.930955+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:38.529611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.887796+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:08:14.887796+0000","last_peered":"2024-09-14T04:08:14.887796+0000","last_clean":"2024-09-14T04:08:14.887796+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:08:14.887796+0000","last_undegraded":"2024-09-14T04:08:14.887796+0000","last_fullsized":"2024-09-14T04:08:14.887796+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.f","version":"206'4145","reported_seq":4160,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:38.699359+0000","last_change":"2024-09-14T04:07:55.607496+0000","last_active":"2024-09-14T04:08:38.699359+0000","last_peered":"2024-09-14T04:08:38.699359+0000","last_clean":"2024-09-14T04:08:38.699359+0000","last_became_active":"2024-09-14T04:07:55.607124+0000","last_became_peered":"2024-09-14T04:07:55.607124+0000","last_unstale":"2024-09-14T04:08:38.699359+0000","last_undegraded":"2024-09-14T04:08:38.699359+0000","last_fullsized":"2024-09-14T04:08:38.699359+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":4145,"log_dups_size":0,"ondisk_log_size":4145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:15:10.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271646720,"num_objects":4145,"num_object_clones":0,"num_object_copies":12435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4145,"num_write_kb":265280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":304,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.938400+0000","last_change":"2024-09-14T04:05:55.690010+0000","last_active":"2024-09-14T04:08:25.938400+0000","last_peered":"2024-09-14T04:08:25.938400+0000","last_clean":"2024-09-14T04:08:25.938400+0000","last_became_active":"2024-09-14T04:05:55.670892+0000","last_became_peered":"2024-09-14T04:05:55.670892+0000","last_unstale":"2024-09-14T04:08:25.938400+0000","last_undegraded":"2024-09-14T04:08:25.938400+0000","last_fullsized":"2024-09-14T04:08:25.938400+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:40:33.024617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":184,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.892909+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:08:14.892909+0000","last_peered":"2024-09-14T04:08:14.892909+0000","last_clean":"2024-09-14T04:08:14.892909+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:08:14.892909+0000","last_undegraded":"2024-09-14T04:08:14.892909+0000","last_fullsized":"2024-09-14T04:08:14.892909+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":49,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888232+0000","last_change":"2024-09-14T04:05:55.674629+0000","last_active":"2024-09-14T04:08:14.888232+0000","last_peered":"2024-09-14T04:08:14.888232+0000","last_clean":"2024-09-14T04:08:14.888232+0000","last_became_active":"2024-09-14T04:05:55.674242+0000","last_became_peered":"2024-09-14T04:05:55.674242+0000","last_unstale":"2024-09-14T04:08:14.888232+0000","last_undegraded":"2024-09-14T04:08:14.888232+0000","last_fullsized":"2024-09-14T04:08:14.888232+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:06:16.827114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":62,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.895407+0000","last_change":"2024-09-14T04:05:55.677636+0000","last_active":"2024-09-14T04:08:14.895407+0000","last_peered":"2024-09-14T04:08:14.895407+0000","last_clean":"2024-09-14T04:08:14.895407+0000","last_became_active":"2024-09-14T04:05:55.677313+0000","last_became_peered":"2024-09-14T04:05:55.677313+0000","last_unstale":"2024-09-14T04:08:14.895407+0000","last_undegraded":"2024-09-14T04:08:14.895407+0000","last_fullsized":"2024-09-14T04:08:14.895407+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:00:13.489641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":102,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894690+0000","last_change":"2024-09-14T04:05:55.686083+0000","last_active":"2024-09-14T04:08:14.894690+0000","last_peered":"2024-09-14T04:08:14.894690+0000","last_clean":"2024-09-14T04:08:14.894690+0000","last_became_active":"2024-09-14T04:05:55.673710+0000","last_became_peered":"2024-09-14T04:05:55.673710+0000","last_unstale":"2024-09-14T04:08:14.894690+0000","last_undegraded":"2024-09-14T04:08:14.894690+0000","last_fullsized":"2024-09-14T04:08:14.894690+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:25.169218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":255,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.930500+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:08:25.930500+0000","last_peered":"2024-09-14T04:08:25.930500+0000","last_clean":"2024-09-14T04:08:25.930500+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:08:25.930500+0000","last_undegraded":"2024-09-14T04:08:25.930500+0000","last_fullsized":"2024-09-14T04:08:25.930500+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":190,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.894602+0000","last_change":"2024-09-14T04:05:55.698766+0000","last_active":"2024-09-14T04:08:14.894602+0000","last_peered":"2024-09-14T04:08:14.894602+0000","last_clean":"2024-09-14T04:08:14.894602+0000","last_became_active":"2024-09-14T04:05:55.674923+0000","last_became_peered":"2024-09-14T04:05:55.674923+0000","last_unstale":"2024-09-14T04:08:14.894602+0000","last_undegraded":"2024-09-14T04:08:14.894602+0000","last_fullsized":"2024-09-14T04:08:14.894602+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:52:37.679728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":172,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888424+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:08:14.888424+0000","last_peered":"2024-09-14T04:08:14.888424+0000","last_clean":"2024-09-14T04:08:14.888424+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:08:14.888424+0000","last_undegraded":"2024-09-14T04:08:14.888424+0000","last_fullsized":"2024-09-14T04:08:14.888424+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":207,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944295+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:08:25.944295+0000","last_peered":"2024-09-14T04:08:25.944295+0000","last_clean":"2024-09-14T04:08:25.944295+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:08:25.944295+0000","last_undegraded":"2024-09-14T04:08:25.944295+0000","last_fullsized":"2024-09-14T04:08:25.944295+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":303,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.934110+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:08:25.934110+0000","last_peered":"2024-09-14T04:08:25.934110+0000","last_clean":"2024-09-14T04:08:25.934110+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:08:25.934110+0000","last_undegraded":"2024-09-14T04:08:25.934110+0000","last_fullsized":"2024-09-14T04:08:25.934110+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":219,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943629+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:08:25.943629+0000","last_peered":"2024-09-14T04:08:25.943629+0000","last_clean":"2024-09-14T04:08:25.943629+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:08:25.943629+0000","last_undegraded":"2024-09-14T04:08:25.943629+0000","last_fullsized":"2024-09-14T04:08:25.943629+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":181,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.892755+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:08:14.892755+0000","last_peered":"2024-09-14T04:08:14.892755+0000","last_clean":"2024-09-14T04:08:14.892755+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:08:14.892755+0000","last_undegraded":"2024-09-14T04:08:14.892755+0000","last_fullsized":"2024-09-14T04:08:14.892755+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943499+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:08:25.943499+0000","last_peered":"2024-09-14T04:08:25.943499+0000","last_clean":"2024-09-14T04:08:25.943499+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:08:25.943499+0000","last_undegraded":"2024-09-14T04:08:25.943499+0000","last_fullsized":"2024-09-14T04:08:25.943499+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":305,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.938268+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:08:25.938268+0000","last_peered":"2024-09-14T04:08:25.938268+0000","last_clean":"2024-09-14T04:08:25.938268+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:08:25.938268+0000","last_undegraded":"2024-09-14T04:08:25.938268+0000","last_fullsized":"2024-09-14T04:08:25.938268+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":262,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.930976+0000","last_change":"2024-09-14T04:05:55.677962+0000","last_active":"2024-09-14T04:08:25.930976+0000","last_peered":"2024-09-14T04:08:25.930976+0000","last_clean":"2024-09-14T04:08:25.930976+0000","last_became_active":"2024-09-14T04:05:55.677277+0000","last_became_peered":"2024-09-14T04:05:55.677277+0000","last_unstale":"2024-09-14T04:08:25.930976+0000","last_undegraded":"2024-09-14T04:08:25.930976+0000","last_fullsized":"2024-09-14T04:08:25.930976+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:24.908568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.951034+0000","last_change":"2024-09-14T04:07:44.338546+0000","last_active":"2024-09-14T04:08:25.951034+0000","last_peered":"2024-09-14T04:08:25.951034+0000","last_clean":"2024-09-14T04:08:25.951034+0000","last_became_active":"2024-09-14T04:07:44.338210+0000","last_became_peered":"2024-09-14T04:07:44.338210+0000","last_unstale":"2024-09-14T04:08:25.951034+0000","last_undegraded":"2024-09-14T04:08:25.951034+0000","last_fullsized":"2024-09-14T04:08:25.951034+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:56.100013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":86,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.931270+0000","last_change":"2024-09-14T04:05:55.671091+0000","last_active":"2024-09-14T04:08:25.931270+0000","last_peered":"2024-09-14T04:08:25.931270+0000","last_clean":"2024-09-14T04:08:25.931270+0000","last_became_active":"2024-09-14T04:05:55.670180+0000","last_became_peered":"2024-09-14T04:05:55.670180+0000","last_unstale":"2024-09-14T04:08:25.931270+0000","last_undegraded":"2024-09-14T04:08:25.931270+0000","last_fullsized":"2024-09-14T04:08:25.931270+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:33.489272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":300,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-14T04:08:40.016466+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:08:40.016466+0000","last_peered":"2024-09-14T04:08:40.016466+0000","last_clean":"2024-09-14T04:08:40.016466+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:08:40.016466+0000","last_undegraded":"2024-09-14T04:08:40.016466+0000","last_fullsized":"2024-09-14T04:08:40.016466+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":288,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933722+0000","last_change":"2024-09-14T04:05:55.671536+0000","last_active":"2024-09-14T04:08:25.933722+0000","last_peered":"2024-09-14T04:08:25.933722+0000","last_clean":"2024-09-14T04:08:25.933722+0000","last_became_active":"2024-09-14T04:05:55.671029+0000","last_became_peered":"2024-09-14T04:05:55.671029+0000","last_unstale":"2024-09-14T04:08:25.933722+0000","last_undegraded":"2024-09-14T04:08:25.933722+0000","last_fullsized":"2024-09-14T04:08:25.933722+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:35.745580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":195,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943973+0000","last_change":"2024-09-14T04:05:55.681242+0000","last_active":"2024-09-14T04:08:25.943973+0000","last_peered":"2024-09-14T04:08:25.943973+0000","last_clean":"2024-09-14T04:08:25.943973+0000","last_became_active":"2024-09-14T04:05:55.680502+0000","last_became_peered":"2024-09-14T04:05:55.680502+0000","last_unstale":"2024-09-14T04:08:25.943973+0000","last_undegraded":"2024-09-14T04:08:25.943973+0000","last_fullsized":"2024-09-14T04:08:25.943973+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:23.760294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":188,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888097+0000","last_change":"2024-09-14T04:05:55.687662+0000","last_active":"2024-09-14T04:08:14.888097+0000","last_peered":"2024-09-14T04:08:14.888097+0000","last_clean":"2024-09-14T04:08:14.888097+0000","last_became_active":"2024-09-14T04:05:55.673561+0000","last_became_peered":"2024-09-14T04:05:55.673561+0000","last_unstale":"2024-09-14T04:08:14.888097+0000","last_undegraded":"2024-09-14T04:08:14.888097+0000","last_fullsized":"2024-09-14T04:08:14.888097+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:14:51.708356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":195,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.946763+0000","last_change":"2024-09-14T04:05:55.679885+0000","last_active":"2024-09-14T04:08:25.946763+0000","last_peered":"2024-09-14T04:08:25.946763+0000","last_clean":"2024-09-14T04:08:25.946763+0000","last_became_active":"2024-09-14T04:05:55.678589+0000","last_became_peered":"2024-09-14T04:05:55.678589+0000","last_unstale":"2024-09-14T04:08:25.946763+0000","last_undegraded":"2024-09-14T04:08:25.946763+0000","last_fullsized":"2024-09-14T04:08:25.946763+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:51:32.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":171,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.888255+0000","last_change":"2024-09-14T04:05:55.697832+0000","last_active":"2024-09-14T04:08:14.888255+0000","last_peered":"2024-09-14T04:08:14.888255+0000","last_clean":"2024-09-14T04:08:14.888255+0000","last_became_active":"2024-09-14T04:05:55.673923+0000","last_became_peered":"2024-09-14T04:05:55.673923+0000","last_unstale":"2024-09-14T04:08:14.888255+0000","last_undegraded":"2024-09-14T04:08:14.888255+0000","last_fullsized":"2024-09-14T04:08:14.888255+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:44.697302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.944212+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:08:25.944212+0000","last_peered":"2024-09-14T04:08:25.944212+0000","last_clean":"2024-09-14T04:08:25.944212+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:08:25.944212+0000","last_undegraded":"2024-09-14T04:08:25.944212+0000","last_fullsized":"2024-09-14T04:08:25.944212+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":32,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.943947+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:08:25.943947+0000","last_peered":"2024-09-14T04:08:25.943947+0000","last_clean":"2024-09-14T04:08:25.943947+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:08:25.943947+0000","last_undegraded":"2024-09-14T04:08:25.943947+0000","last_fullsized":"2024-09-14T04:08:25.943947+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":123,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-14T04:08:14.891772+0000","last_change":"2024-09-14T04:07:44.338957+0000","last_active":"2024-09-14T04:08:14.891772+0000","last_peered":"2024-09-14T04:08:14.891772+0000","last_clean":"2024-09-14T04:08:14.891772+0000","last_became_active":"2024-09-14T04:07:44.338190+0000","last_became_peered":"2024-09-14T04:07:44.338190+0000","last_unstale":"2024-09-14T04:08:14.891772+0000","last_undegraded":"2024-09-14T04:08:14.891772+0000","last_fullsized":"2024-09-14T04:08:14.891772+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:36:27.488115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":293,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-14T04:08:25.933806+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:08:25.933806+0000","last_peered":"2024-09-14T04:08:25.933806+0000","last_clean":"2024-09-14T04:08:25.933806+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:08:25.933806+0000","last_undegraded":"2024-09-14T04:08:25.933806+0000","last_fullsized":"2024-09-14T04:08:25.933806+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4344578048,"num_objects":66293,"num_object_clones":0,"num_object_copies":198879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66293,"num_write_kb":4242752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13048741888,"data_stored":13048741888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66293,"ondisk_log_size":66293,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4890,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430539,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2228464,"kb_used_data":1317096,"kb_used_omap":1,"kb_used_meta":911358,"kb_avail":91516688,"statfs":{"total":95995035648,"available":93713088512,"internally_reserved":0,"allocated":1348706304,"data_stored":1347725227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":933230936},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626768,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2438568,"kb_used_data":1848672,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":91306584,"statfs":{"total":95995035648,"available":93497942016,"internally_reserved":0,"allocated":1893040128,"data_stored":1892066835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822996,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2487060,"kb_used_data":1585932,"kb_used_omap":1,"kb_used_meta":901118,"kb_avail":91258092,"statfs":{"total":95995035648,"available":93448286208,"internally_reserved":0,"allocated":1623994368,"data_stored":1623036488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922745176},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":798863917079,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552377,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2543956,"kb_used_data":2126220,"kb_used_omap":4,"kb_used_meta":417723,"kb_avail":91201196,"statfs":{"total":95995035648,"available":93390024704,"internally_reserved":0,"allocated":2177249280,"data_stored":2176291481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":427748797},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435419,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2040492,"kb_used_data":1654432,"kb_used_omap":6,"kb_used_meta":386041,"kb_avail":91704660,"statfs":{"total":95995035648,"available":93905571840,"internally_reserved":0,"allocated":1694138368,"data_stored":1693172188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":395306445},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827888,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2173208,"kb_used_data":1591760,"kb_used_omap":5,"kb_used_meta":581434,"kb_avail":91571944,"statfs":{"total":95995035648,"available":93769670656,"internally_reserved":0,"allocated":1629962240,"data_stored":1629000296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":595388655},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383876,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3186716,"kb_used_data":2574164,"kb_used_omap":5,"kb_used_meta":612538,"kb_avail":90558436,"statfs":{"total":95995035648,"available":92731838464,"internally_reserved":0,"allocated":2635943936,"data_stored":2634978782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":627238972},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2646933504,"data_stored":2646933504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1627848704,"data_stored":1627848704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1702690816,"data_stored":1702690816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2184249344,"data_stored":2184249344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1637548032,"data_stored":1637548032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1891237888,"data_stored":1891237888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1358233600,"data_stored":1358233600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:08:44.974 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-14T04:08:44.974 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:08:44.974 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-14T04:08:44.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-14T04:08:44.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:44.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:45.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:45.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:08:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:45.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:45.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:45.196 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:45 smithi114 ceph-mon[106969]: from='client.45944 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:08:45.197 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:45 smithi114 ceph-mon[103029]: from='client.45944 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:08:45.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:45.240 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 73660 73644 109.567 66 0.0254235 0.00912191 2024-09-14T04:08:45.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:45.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:45.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:45.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:45 smithi160 ceph-mon[88407]: from='client.45944 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:08:45.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:45.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:45.448 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:45.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:45.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:45.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:45.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:45.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:45.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:45.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:45.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:45.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:45.955 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:45.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:45.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:46.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:08:46.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:46.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:46.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:46.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:46.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 74584 74568 108.362 57.75 0.0218099 0.00922332 2024-09-14T04:08:46.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:46.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:46.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:46.333 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:46 smithi114 ceph-mon[106969]: pgmap v730: 89 pgs: 89 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-14T04:08:46.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:46 smithi114 ceph-mon[103029]: pgmap v730: 89 pgs: 89 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-14T04:08:46.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:46.400 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:46.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:46 smithi160 ceph-mon[88407]: pgmap v730: 89 pgs: 89 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-14T04:08:46.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:46.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:46.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:46.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:46.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:46.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:46.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:46.738 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:46.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:46.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:46.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:46.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:46.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:46.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:46.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:47.021 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:47.046 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:47.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:47.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:47.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:47.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:47.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 75701 75685 107.486 69.8125 0.00275451 0.00928241 2024-09-14T04:08:47.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:47.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:47.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:47.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:47.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:47.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:47.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:47.602 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:47.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:47.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:47.715 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:47.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:47.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:47.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:47.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:47.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:47.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:47.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:48.033 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:48.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:48.052 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:08:48.052 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:08:48.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:08:48.053 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:08:48.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.053 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:79163 9.d 9:b8bb6715:::benchmark_data_smithi114_153893_object79162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-14T04:08:48.053 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:04:02.322866+0000", 2024-09-14T04:08:48.053 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 285.72374170900002, 2024-09-14T04:08:48.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32682998800000002, 2024-09-14T04:08:48.053 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.053 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.053 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79163 2024-09-14T04:08:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322866+0000", 2024-09-14T04:08:48.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322866+0000", 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322865+0000", 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:48.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322920+0000", 2024-09-14T04:08:48.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5031999999999998e-05 2024-09-14T04:08:48.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322932+0000", 2024-09-14T04:08:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2268e-05 2024-09-14T04:08:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322941+0000", 2024-09-14T04:08:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6689999999999995e-06 2024-09-14T04:08:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.323046+0000", 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010484000000000001 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.323232+0000", 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018588599999999999 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,5", 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.323552+0000", 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00032002499999999999 2024-09-14T04:08:48.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.326534+0000", 2024-09-14T04:08:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029825390000000002 2024-09-14T04:08:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.328765+0000", 2024-09-14T04:08:48.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022310860000000002 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.649306+0000", 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32054082099999998 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.649661+0000", 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035479400000000002 2024-09-14T04:08:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.649696+0000", 2024-09-14T04:08:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5377999999999999e-05 2024-09-14T04:08:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:195981 9.e 9:741dedbf:::benchmark_data_smithi114_153893_object66520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:06.788523+0000", 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 161.258084575, 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.331278462, 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195981 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788523+0000", 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788523+0000", 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788522+0000", 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:48.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788528+0000", 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3659999999999997e-06 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788541+0000", 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2755000000000001e-05 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788546+0000", 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0239999999999997e-06 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788597+0000", 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1316000000000001e-05 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788726+0000", 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012843300000000001 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,4", 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788884+0000", 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015847600000000001 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.792615+0000", 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003730424 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.792730+0000", 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011485900000000001 2024-09-14T04:08:48.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.805843+0000", 2024-09-14T04:08:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013113855000000001 2024-09-14T04:08:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119607+0000", 2024-09-14T04:08:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31376335799999999 2024-09-14T04:08:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119669+0000", 2024-09-14T04:08:48.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2397000000000006e-05 2024-09-14T04:08:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119802+0000", 2024-09-14T04:08:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013257899999999999 2024-09-14T04:08:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.069 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:195983 9.e 9:76113637:::benchmark_data_smithi114_153893_object66522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:06.793113+0000", 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 161.25349537299999, 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32693720900000001, 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195983 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793113+0000", 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793113+0000", 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793111+0000", 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793117+0000", 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4919999999999998e-06 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793130+0000", 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2347999999999999e-05 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793136+0000", 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2310000000000001e-06 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793173+0000", 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7468000000000001e-05 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793290+0000", 2024-09-14T04:08:48.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116841 2024-09-14T04:08:48.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,4", 2024-09-14T04:08:48.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793372+0000", 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1866000000000005e-05 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.796363+0000", 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029914579999999998 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.797301+0000", 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000937089 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.805934+0000", 2024-09-14T04:08:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0086334329999999994 2024-09-14T04:08:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119867+0000", 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31393343200000001 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119922+0000", 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4187000000000001e-05 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.120050+0000", 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012817899999999999 2024-09-14T04:08:48.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.078 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.078 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:195999 9.e 9:7a76e1c2:::benchmark_data_smithi114_153893_object66538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-14T04:08:48.078 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:06.807061+0000", 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 161.23954680400001, 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31979472599999997, 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195999 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807061+0000", 2024-09-14T04:08:48.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807061+0000", 2024-09-14T04:08:48.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807060+0000", 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807064+0000", 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.912e-06 2024-09-14T04:08:48.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807075+0000", 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0807e-05 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807081+0000", 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9699999999999996e-06 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807122+0000", 2024-09-14T04:08:48.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1507000000000001e-05 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807247+0000", 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012494900000000001 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,4", 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807381+0000", 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000133621 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.810839+0000", 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034575180000000001 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.810906+0000", 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7020999999999994e-05 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.811227+0000", 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00032114300000000003 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.126723+0000", 2024-09-14T04:08:48.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31549654399999999 2024-09-14T04:08:48.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.126759+0000", 2024-09-14T04:08:48.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5809999999999998e-05 2024-09-14T04:08:48.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.126856+0000", 2024-09-14T04:08:48.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6773000000000003e-05 2024-09-14T04:08:48.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.086 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.087 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221222 9.e 9:71e3d4ca:::benchmark_data_smithi114_153893_object91761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.087 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.075396+0000", 2024-09-14T04:08:48.087 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.97121221499999, 2024-09-14T04:08:48.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98772043600000003, 2024-09-14T04:08:48.087 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.087 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.087 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.088 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.088 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.088 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221222 2024-09-14T04:08:48.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.088 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075396+0000", 2024-09-14T04:08:48.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075396+0000", 2024-09-14T04:08:48.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075394+0000", 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075399+0000", 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2059999999999998e-06 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075414+0000", 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4457999999999999e-05 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075419+0000", 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0590000000000002e-06 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075463+0000", 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4357000000000002e-05 2024-09-14T04:08:48.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075471+0000", 2024-09-14T04:08:48.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5900000000000002e-06 2024-09-14T04:08:48.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.054310+0000", 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97883944700000003 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.054405+0000", 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4752000000000004e-05 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.054480+0000", 2024-09-14T04:08:48.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4501999999999999e-05 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.062905+0000", 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0084256130000000002 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.062914+0000", 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8750000000000006e-06 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063021+0000", 2024-09-14T04:08:48.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106359 2024-09-14T04:08:48.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063077+0000", 2024-09-14T04:08:48.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6490999999999998e-05 2024-09-14T04:08:48.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063116+0000", 2024-09-14T04:08:48.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9150999999999998e-05 2024-09-14T04:08:48.100 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.100 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.100 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.100 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221279 9.e 9:7a0e57f8:::benchmark_data_smithi114_153893_object91818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.100 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.162273+0000", 2024-09-14T04:08:48.100 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.884334581, 2024-09-14T04:08:48.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90723786100000003, 2024-09-14T04:08:48.101 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.101 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.101 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.101 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.101 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.101 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221279 2024-09-14T04:08:48.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.101 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162273+0000", 2024-09-14T04:08:48.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162273+0000", 2024-09-14T04:08:48.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162271+0000", 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162279+0000", 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9810000000000003e-06 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162291+0000", 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1804000000000001e-05 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162297+0000", 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2790000000000004e-06 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162350+0000", 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2383999999999999e-05 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162358+0000", 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.057426+0000", 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89506750499999999 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.057490+0000", 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4579999999999995e-05 2024-09-14T04:08:48.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.106 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.106 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.057567+0000", 2024-09-14T04:08:48.106 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6237999999999998e-05 2024-09-14T04:08:48.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.106 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.106 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.062944+0000", 2024-09-14T04:08:48.106 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0053775539999999997 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063126+0000", 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000181977 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069407+0000", 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062807540000000004 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.107 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069473+0000", 2024-09-14T04:08:48.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6450000000000002e-05 2024-09-14T04:08:48.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069511+0000", 2024-09-14T04:08:48.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7894e-05 2024-09-14T04:08:48.108 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221287 9.e 9:7c5f7cde:::benchmark_data_smithi114_153893_object91826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.168058+0000", 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.878550416, 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90162981099999995, 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.109 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221287 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168058+0000", 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168058+0000", 2024-09-14T04:08:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168056+0000", 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168061+0000", 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8089999999999998e-06 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.111 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168075+0000", 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3091000000000001e-05 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168080+0000", 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7799999999999997e-06 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168154+0000", 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3906999999999996e-05 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168162+0000", 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4819999999999997e-06 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.060288+0000", 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89212670500000002 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.060357+0000", 2024-09-14T04:08:48.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8893999999999995e-05 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.060439+0000", 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.174e-05 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063132+0000", 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002692664 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.114 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069374+0000", 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062418459999999997 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069617+0000", 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024384999999999999 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069658+0000", 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0562000000000002e-05 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.115 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069687+0000", 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9346999999999999e-05 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221290 9.e 9:7e953b68:::benchmark_data_smithi114_153893_object91829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.169237+0000", 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.877371067, 2024-09-14T04:08:48.116 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90707892899999998, 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221290 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169237+0000", 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.117 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169237+0000", 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169236+0000", 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169242+0000", 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0349999999999998e-06 2024-09-14T04:08:48.118 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169254+0000", 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2427e-05 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169260+0000", 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9259999999999997e-06 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169302+0000", 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2283999999999997e-05 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169309+0000", 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0589999999999997e-06 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063175+0000", 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89386510500000005 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063285+0000", 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011004 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063400+0000", 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011540300000000001 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069380+0000", 2024-09-14T04:08:48.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059798780000000001 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069573+0000", 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000193191 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076249+0000", 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0066763270000000001 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076282+0000", 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2206999999999998e-05 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076316+0000", 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4171000000000002e-05 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.123 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221308 9.e 9:7b1cb916:::benchmark_data_smithi114_153893_object91847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.194179+0000", 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.852428625, 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88227606199999997, 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221308 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194179+0000", 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194179+0000", 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194178+0000", 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:08:48.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194184+0000", 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5339999999999999e-06 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194195+0000", 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1090000000000001e-05 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194200+0000", 2024-09-14T04:08:48.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4079999999999997e-06 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194253+0000", 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3186999999999997e-05 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194260+0000", 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8920000000000003e-06 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.065682+0000", 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87142183299999998 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.065750+0000", 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7461999999999999e-05 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.065824+0000", 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4950999999999994e-05 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069695+0000", 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0038707199999999998 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069743+0000", 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7444999999999997e-05 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076385+0000", 2024-09-14T04:08:48.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0066419080000000002 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076424+0000", 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9212000000000003e-05 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076455+0000", 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1605e-05 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221336 9.e 9:7fbc0f9f:::benchmark_data_smithi114_153893_object91875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.824771748, 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86189694800000005, 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221336 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.131 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:08:48.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221839+0000", 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8779999999999998e-06 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221844+0000", 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9030000000000001e-06 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221848+0000", 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.506e-06 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221883+0000", 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5491000000000003e-05 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221888+0000", 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7690000000000004e-06 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069784+0000", 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84789569300000001 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069903+0000", 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119528 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.070022+0000", 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118662 2024-09-14T04:08:48.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076341+0000", 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0063191180000000003 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076464+0000", 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012243599999999999 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083657+0000", 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0071937420000000004 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083700+0000", 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2095000000000003e-05 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083733+0000", 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3596000000000002e-05 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.137 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221353 9.e 9:7a884025:::benchmark_data_smithi114_153893_object91892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.247072+0000", 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.799535526, 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85085830500000004, 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221353 2024-09-14T04:08:48.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247072+0000", 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247072+0000", 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247071+0000", 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247078+0000", 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6290000000000004e-06 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247089+0000", 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1168999999999999e-05 2024-09-14T04:08:48.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247093+0000", 2024-09-14T04:08:48.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.578e-06 2024-09-14T04:08:48.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247140+0000", 2024-09-14T04:08:48.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6113999999999999e-05 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247148+0000", 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1839999999999999e-06 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.073043+0000", 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82589522500000001 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.073170+0000", 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000127541 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.073311+0000", 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014082 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083510+0000", 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010198239 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083573+0000", 2024-09-14T04:08:48.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3004999999999997e-05 2024-09-14T04:08:48.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097818+0000", 2024-09-14T04:08:48.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014245821000000001 2024-09-14T04:08:48.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097866+0000", 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8029999999999999e-05 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097931+0000", 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4312000000000004e-05 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221359 9.e 9:76f61d7d:::benchmark_data_smithi114_153893_object91898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.252688+0000", 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.793919598, 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84535683900000003, 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221359 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.146 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252688+0000", 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252688+0000", 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252687+0000", 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252696+0000", 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5210000000000008e-06 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252709+0000", 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2852e-05 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252716+0000", 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5749999999999997e-06 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252760+0000", 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.405e-05 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252768+0000", 2024-09-14T04:08:48.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0490000000000004e-06 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076490+0000", 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82372204199999999 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076560+0000", 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9900999999999994e-05 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076623+0000", 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3226999999999996e-05 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083562+0000", 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0069392339999999999 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083593+0000", 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0880999999999998e-05 2024-09-14T04:08:48.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097986+0000", 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014392922000000001 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098017+0000", 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1362000000000002e-05 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098045+0000", 2024-09-14T04:08:48.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7875999999999999e-05 2024-09-14T04:08:48.153 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.153 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.153 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.153 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221381 9.e 9:70816107:::benchmark_data_smithi114_153893_object91920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.153 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.278485+0000", 2024-09-14T04:08:48.153 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:48.155 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.768122449, 2024-09-14T04:08:48.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81964557199999999, 2024-09-14T04:08:48.155 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.155 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.155 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.155 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.155 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.155 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221381 2024-09-14T04:08:48.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278485+0000", 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278485+0000", 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278484+0000", 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278490+0000", 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5910000000000003e-06 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278502+0000", 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2756e-05 2024-09-14T04:08:48.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278508+0000", 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3410000000000002e-06 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278558+0000", 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9959000000000001e-05 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278564+0000", 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3319999999999997e-06 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.080018+0000", 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80145439200000002 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.080117+0000", 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8821999999999997e-05 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.080228+0000", 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111292 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083771+0000", 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003542842 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097698+0000", 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013926565 2024-09-14T04:08:48.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098075+0000", 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037682900000000002 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098104+0000", 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9519e-05 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098131+0000", 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6872999999999998e-05 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221383 9.e 9:78a714d0:::benchmark_data_smithi114_153893_object91922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.282793+0000", 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.763815024, 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82646461900000001, 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221383 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282793+0000", 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282793+0000", 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282791+0000", 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282797+0000", 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5149999999999997e-06 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282808+0000", 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1358e-05 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282820+0000", 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1313e-05 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282857+0000", 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6928000000000003e-05 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282862+0000", 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.677e-06 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083790+0000", 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80092809499999995 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083893+0000", 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103038 2024-09-14T04:08:48.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083996+0000", 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010259600000000001 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097702+0000", 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013706476 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097754+0000", 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1768000000000003e-05 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109094+0000", 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011340195000000001 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109221+0000", 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000127019 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109258+0000", 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6075000000000003e-05 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221421 9.e 9:7e5137ba:::benchmark_data_smithi114_153893_object91960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.359708+0000", 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.686900044, 2024-09-14T04:08:48.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.74967277499999996, 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221421 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359708+0000", 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359708+0000", 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359706+0000", 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359713+0000", 2024-09-14T04:08:48.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5989999999999998e-06 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359732+0000", 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9040000000000001e-05 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359738+0000", 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.404e-06 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359780+0000", 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1572000000000001e-05 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359787+0000", 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4490000000000002e-06 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.087250+0000", 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72746270599999996 2024-09-14T04:08:48.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.087357+0000", 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010766 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.087467+0000", 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109231 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097707+0000", 2024-09-14T04:08:48.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010240383 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097786+0000", 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9331000000000003e-05 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109308+0000", 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011522107 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109345+0000", 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6078999999999998e-05 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109381+0000", 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6149000000000002e-05 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221427 9.e 9:73b3c5ab:::benchmark_data_smithi114_153893_object91966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.371115+0000", 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.67549252400001, 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.73836333300000001, 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221427 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371115+0000", 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371115+0000", 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371114+0000", 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371120+0000", 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3069999999999999e-06 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371134+0000", 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4318000000000001e-05 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371141+0000", 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9890000000000003e-06 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371193+0000", 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1620999999999999e-05 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371201+0000", 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2719999999999997e-06 2024-09-14T04:08:48.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.090232+0000", 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71903041000000001 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.090303+0000", 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1452000000000006e-05 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.090372+0000", 2024-09-14T04:08:48.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8422000000000003e-05 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097760+0000", 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.007388309 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097961+0000", 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000200619 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109420+0000", 2024-09-14T04:08:48.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011458983000000001 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109451+0000", 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1189999999999998e-05 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109479+0000", 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8042e-05 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221428 9.e 9:7f963fb1:::benchmark_data_smithi114_153893_object91967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.372209+0000", 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.674399335, 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.73735522099999995, 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221428 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372209+0000", 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372209+0000", 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372208+0000", 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372213+0000", 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4999999999999999e-06 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372227+0000", 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3754000000000001e-05 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372233+0000", 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7749999999999998e-06 2024-09-14T04:08:48.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372281+0000", 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8371999999999997e-05 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372288+0000", 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.306e-06 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.093512+0000", 2024-09-14T04:08:48.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72122381700000004 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.093613+0000", 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100993 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.093719+0000", 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105808 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105360+0000", 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01164106 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105400+0000", 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0420999999999998e-05 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109507+0000", 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041070669999999998 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109537+0000", 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9247999999999998e-05 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109564+0000", 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7118e-05 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.190 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221441 9.e 9:79734224:::benchmark_data_smithi114_153893_object91980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.396588+0000", 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.650020383, 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71322003, 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221441 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396588+0000", 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396588+0000", 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396586+0000", 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396591+0000", 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9289999999999997e-06 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396602+0000", 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0547000000000001e-05 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396607+0000", 2024-09-14T04:08:48.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1000000000000003e-06 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396649+0000", 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1941999999999999e-05 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396656+0000", 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8369999999999999e-06 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098153+0000", 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70149738800000006 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098264+0000", 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011047100000000001 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098389+0000", 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000125053 2024-09-14T04:08:48.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109038+0000", 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010649614 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109276+0000", 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000237999 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109715+0000", 2024-09-14T04:08:48.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043909999999999999 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109776+0000", 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0977999999999998e-05 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109808+0000", 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1319000000000003e-05 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.197 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221442 9.e 9:7a9a1506:::benchmark_data_smithi114_153893_object91981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.396654+0000", 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.649954279, 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72359706499999998, 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221442 2024-09-14T04:08:48.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.199 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.199 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396654+0000", 2024-09-14T04:08:48.199 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.199 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396654+0000", 2024-09-14T04:08:48.199 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396653+0000", 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396656+0000", 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0419999999999998e-06 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396667+0000", 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1181e-05 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396671+0000", 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9879999999999998e-06 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396699+0000", 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7458e-05 2024-09-14T04:08:48.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.202 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396704+0000", 2024-09-14T04:08:48.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0039999999999999e-06 2024-09-14T04:08:48.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.202 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.101911+0000", 2024-09-14T04:08:48.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70520688399999998 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.102000+0000", 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9349000000000001e-05 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.102100+0000", 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9623999999999993e-05 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109387+0000", 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0072876240000000004 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.117933+0000", 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0085459390000000007 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120171+0000", 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022373919999999999 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120213+0000", 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2882e-05 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120251+0000", 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7323000000000002e-05 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221473 9.e 9:74ac708e:::benchmark_data_smithi114_153893_object92012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.530644+0000", 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.51596413999999, 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.58970111199999997, 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221473 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:08:48.206 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530644+0000", 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530644+0000", 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530643+0000", 2024-09-14T04:08:48.207 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:08:48.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.208 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:08:48.208 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530649+0000", 2024-09-14T04:08:48.208 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1480000000000003e-06 2024-09-14T04:08:48.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.208 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:08:48.208 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530660+0000", 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1765e-05 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530666+0000", 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7529999999999998e-06 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530701+0000", 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4487999999999999e-05 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.209 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530706+0000", 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4689999999999996e-06 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105426+0000", 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.57472014800000004 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105530+0000", 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103225 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105641+0000", 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011111399999999999 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109392+0000", 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037509470000000001 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.211 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.117959+0000", 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.008567346 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120281+0000", 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023216310000000002 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120314+0000", 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3865000000000002e-05 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:08:48.212 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:08:48.213 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120345+0000", 2024-09-14T04:08:48.213 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0487000000000001e-05 2024-09-14T04:08:48.213 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.213 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.213 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.213 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:08:48.213 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:08:48.213 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:08:48.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 76850 76834 106.693 71.8125 0.00497725 0.00936761 2024-09-14T04:08:48.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:48.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:48.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:48.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:48.369 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:48.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:48.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:48.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:48 smithi114 ceph-mon[106969]: pgmap v731: 89 pgs: 89 active+clean; 4.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-14T04:08:48.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:48 smithi114 ceph-mon[103029]: pgmap v731: 89 pgs: 89 active+clean; 4.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-14T04:08:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:48 smithi160 ceph-mon[88407]: pgmap v731: 89 pgs: 89 active+clean; 4.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-14T04:08:48.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:48.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:48.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:48.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:48.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:48.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:48.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:48.773 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:48.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:48.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:48.912 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:48.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:49.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:49.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:49.158 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:49.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:49.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:49.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:08:49.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 78096 78080 106.066 77.875 0.00492195 0.00942257 2024-09-14T04:08:49.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:49.334 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:49.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:49.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:49.472 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:49.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:49.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:49.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:49.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:49.622 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:49.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:49.748 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:49.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:49.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:49.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:49.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:49.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:50.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:50.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:50.138 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:50.172 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:50.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:50.236 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 79295 79279 105.404 74.9375 0.00605185 0.00948236 2024-09-14T04:08:50.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:50.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:50.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:50.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:50 smithi114 ceph-mon[103029]: pgmap v732: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-14T04:08:50.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:50 smithi114 ceph-mon[106969]: pgmap v732: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-14T04:08:50.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:08:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:08:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:08:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:50 smithi160 ceph-mon[88407]: pgmap v732: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-14T04:08:50.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:50.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:50.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:50.574 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:50.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:50.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:50.702 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:50.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:50.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:50.832 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:50.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:50.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:50.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-14T04:08:50.977 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:50.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:51.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:51.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:51.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:51.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:51.236 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 80507 80491 104.786 75.75 0.00926046 0.00953692 2024-09-14T04:08:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:51.264 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:51.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:51.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:51.400 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:51.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:51.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:51.530 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:51.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:51.579 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:08:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:08:51.579 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-14T04:08:51.579 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:08:51.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:51.682 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:51.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:51.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:51.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:51.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:51.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:08:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:08:51] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-09-14T04:08:51.925 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:51.955 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:51.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:52.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:52.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:52.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:52.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:52.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:52.236 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 81658 81642 104.116 71.9375 0.00839655 0.00959929 2024-09-14T04:08:52.256 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:52.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:52.276 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:52 smithi160 ceph-mon[88407]: pgmap v733: 89 pgs: 89 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-14T04:08:52.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:52 smithi114 ceph-mon[106969]: pgmap v733: 89 pgs: 89 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-14T04:08:52.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:52 smithi114 ceph-mon[103029]: pgmap v733: 89 pgs: 89 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-14T04:08:52.309 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-14T04:08:52.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:52.404 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:52.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:52.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:52.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:52.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:52.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:52.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:52.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:52.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:52.702 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:52.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:52.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:52.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:52.924 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:52.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:52.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:52.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:53.013 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:08:53.014 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:08:53.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:53.054 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:08:53.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:08:53.061 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:08:53.093 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:08:53.093 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:08:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:53.098 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:53.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:53.141 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:08:53.142 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:08:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:08:53.179 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:08:53.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:53.214 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:08:53.214 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:08:53.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:53.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:53.236 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 82674 82658 103.303 63.5 0.00648645 0.00967481 2024-09-14T04:08:53.252 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:08:53.252 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:08:53.291 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:08:53.291 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:08:53.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:53.379 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:53.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:53.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:53.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:53.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:53.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:53.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:53.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:53.659 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:08:53.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:53.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:53.788 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:53.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:53.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:53 smithi160 ceph-mon[88407]: pgmap v734: 89 pgs: 89 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-14T04:08:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2343579183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:53.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2638112949' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:08:53.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:53 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:08:53.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:53.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:54.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:54.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:54.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:54.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[103029]: pgmap v734: 89 pgs: 89 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-14T04:08:54.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2343579183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:54.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2638112949' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:08:54.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:08:54.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:08:54.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:08:54.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[106969]: pgmap v734: 89 pgs: 89 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-14T04:08:54.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2343579183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:08:54.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2638112949' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:08:54.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:53 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:08:54.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:54.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:54.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:54.236 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 83804 83788 102.662 70.625 0.0163881 0.00973552 2024-09-14T04:08:54.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:54.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:54.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:54.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:54.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:55.097 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:08:55.097 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:08:55.098 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:54 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:08:55.098 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:54 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:08:55.098 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:54 smithi160 ceph-mon[88407]: osdmap e208: 8 total, 8 up, 7 in 2024-09-14T04:08:55.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:54 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:08:55.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:54 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:08:55.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:54 smithi114 ceph-mon[106969]: osdmap e208: 8 total, 8 up, 7 in 2024-09-14T04:08:55.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:54 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:08:55.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:54 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:08:55.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:54 smithi114 ceph-mon[103029]: osdmap e208: 8 total, 8 up, 7 in 2024-09-14T04:08:55.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:55.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:55.236 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 84709 84693 101.776 56.5625 0.0324088 0.00981982 2024-09-14T04:08:55.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:55.279 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:55.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:55.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:55.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:55.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:55.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:55.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:55.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:55.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:55.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:55.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:55.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:55.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:55.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:55.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:55 smithi160 ceph-mon[88407]: pgmap v736: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-14T04:08:55.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:55.954 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:55.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:55.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:56.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:56.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-14T04:08:56.092 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:56.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:56.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:55 smithi114 ceph-mon[106969]: pgmap v736: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-14T04:08:56.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:55 smithi114 ceph-mon[103029]: pgmap v736: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-14T04:08:56.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:56.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:56.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:56.236 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 85817 85801 101.162 69.25 0.0123644 0.00988008 2024-09-14T04:08:56.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:56.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:56.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:56.419 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:08:56.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:56.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:56.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:56.595 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:56.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:56.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:56.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:56.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:56.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:56.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:56.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:56.901 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:56.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:57.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:57.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:57.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:57.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:57.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:57.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:57.236 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 86980 86964 100.635 72.6875 0.0116532 0.00993197 2024-09-14T04:08:57.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:57.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:57.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:57.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:57.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:57.460 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:57.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:57.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:08:57.589 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:57.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:08:57.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:08:57.723 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:57.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:08:57.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:57.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:57 smithi160 ceph-mon[88407]: pgmap v737: 89 pgs: 89 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-14T04:08:57.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:57.994 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:57.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:58.008 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:57 smithi114 ceph-mon[103029]: pgmap v737: 89 pgs: 89 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-14T04:08:58.009 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:57 smithi114 ceph-mon[106969]: pgmap v737: 89 pgs: 89 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-14T04:08:58.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:58.135 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:58.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:58.237 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 15 88041 88026 100.012 66.375 0.0279184 0.00999362 2024-09-14T04:08:58.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:58.278 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:58.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:58.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:58.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:58.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:58.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:58.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:58.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:58.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:58.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:58.684 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:58.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:58.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:58.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:58.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:58.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:58.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:58.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:59.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:08:59.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:59.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:08:59.195 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:08:59.195 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:08:59.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:08:59.237 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 89157 89141 99.4702 69.6875 0.00466305 0.0100479 2024-09-14T04:08:59.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:59.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:08:59.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:08:59.373 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:59.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:08:59.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:08:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:59.506 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:59.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:08:59.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:08:59.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-14T04:08:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:59.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:08:59.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:08:59.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:59.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:08:59.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:59.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:08:59.833 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:08:59 smithi114 ceph-mon[103029]: pgmap v738: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-14T04:08:59.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:08:59 smithi114 ceph-mon[106969]: pgmap v738: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-14T04:08:59.874 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:08:59 smithi160 ceph-mon[88407]: pgmap v738: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-14T04:08:59.887 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:08:59.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-14T04:08:59.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:08:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:08:59.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:00.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:00.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:00.046 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:00.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:00.114 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:00.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:00.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:00.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:00.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:00.237 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 90113 90097 98.772 59.75 0.0166822 0.0101191 2024-09-14T04:09:00.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:00.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:00.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:00.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:00.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:00.443 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:00.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:00.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:09:00.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:00.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:00.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:09:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:09:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:09:00.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:00.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:00.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:00.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:00.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:00.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:00.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:00.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:00.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:01.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:01.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:01.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:01.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:01.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:01.237 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 91026 91010 98.0529 57.0625 0.010282 0.0101938 2024-09-14T04:09:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:01.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:01.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:01.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:01.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:01.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:01.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:01.510 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:01.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:01.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:01.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:09:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:09:01] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-14T04:09:01.650 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:01.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:01.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:01.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:01.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:01.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:01.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:01 smithi160 ceph-mon[88407]: pgmap v739: 89 pgs: 89 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-14T04:09:01.931 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:01.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:01.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:02.033 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:02.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:02.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:02.110 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:01 smithi114 ceph-mon[106969]: pgmap v739: 89 pgs: 89 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-14T04:09:02.110 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:01 smithi114 ceph-mon[103029]: pgmap v739: 89 pgs: 89 active+clean; 5.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-14T04:09:02.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:02.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:02.232 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:02.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:02.242 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 92005 91989 97.4208 61.1875 0.0132719 0.0102586 2024-09-14T04:09:02.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:02.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:02.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:02.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:02.509 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:02.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:02.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:02.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:02.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:02.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:02.779 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:02.780 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:02.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:02.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:02.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:02.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:02.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:02.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:03.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:03.095 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:03.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:03.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:03.205 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:03.244 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:09:03.244274+0000 min lat: 0.00109236 max lat: 0.268444 avg lat: 0.0103327 2024-09-14T04:09:03.245 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:09:03.245 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 92895 92879 96.7205 55.625 0.0398855 0.0103327 2024-09-14T04:09:03.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:03.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:03.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:03.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:03.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:03.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:03.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:03.519 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:03.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:03.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:03.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:03.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:03.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:03.754 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:03.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:03.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:03.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:03.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:03.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:04.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:04.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:04.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:04.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:03 smithi114 ceph-mon[103029]: pgmap v740: 89 pgs: 89 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-14T04:09:04.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:03 smithi114 ceph-mon[106969]: pgmap v740: 89 pgs: 89 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-14T04:09:04.153 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:03 smithi160 ceph-mon[88407]: pgmap v740: 89 pgs: 89 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-14T04:09:04.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:04.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:04.244 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 93973 93957 96.2394 67.375 0.00924706 0.0103853 2024-09-14T04:09:04.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:04.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:04.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:04.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:04.443 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:04.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:04.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:04.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:04.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:04.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:04.709 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:04.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:04.811 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:04.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:04.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:04.944 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:04.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:04.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:05.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:05.112 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:05.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:05.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:05.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:05.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:05.244 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 95071 95055 95.794 68.625 0.0158659 0.0104335 2024-09-14T04:09:05.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:05.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:05.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:05.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:05.523 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:05.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:05.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:05.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:05.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:05.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:05.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:05.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:05.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:05.915 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:05.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:06.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:06.041 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:06.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:06.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:05 smithi114 ceph-mon[106969]: pgmap v741: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-14T04:09:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:05 smithi114 ceph-mon[103029]: pgmap v741: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-14T04:09:06.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:05 smithi160 ceph-mon[88407]: pgmap v741: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-14T04:09:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:06.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:06.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:06.245 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 96003 95987 95.1981 58.25 0.0101863 0.0104986 2024-09-14T04:09:06.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:06.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:06.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:06.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:06.440 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:06.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:06.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:06.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:06.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:06.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:06.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:06.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:06.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:06.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:06.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:06.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:06.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:06.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:07.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:07.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:07.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:07.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:07.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:07.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:07.245 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 96734 96718 94.4246 45.6875 0.00933872 0.0105853 2024-09-14T04:09:07.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:07.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:07.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:07.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:07.457 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:07.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:07.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:07.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:07.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:07.697 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:07.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:07.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:07.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:07.861 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:07.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:07.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:08.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:08.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:08.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:07 smithi114 ceph-mon[106969]: pgmap v742: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-14T04:09:08.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:08.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:08.099 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:07 smithi114 ceph-mon[103029]: pgmap v742: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-14T04:09:08.099 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:08.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:08.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:08.144 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:08.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:07 smithi160 ceph-mon[88407]: pgmap v742: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-14T04:09:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:08.245 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 97676 97660 93.8777 58.875 0.0100959 0.0106469 2024-09-14T04:09:08.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:08.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:08.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:08.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:08.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:08.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:08.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:08.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:08.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:08.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:08.700 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:08.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:08.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:08.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:08.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:08.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:08.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:08.850 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:09:08.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:08.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:08.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:09.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:09.087 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4119558078' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:09:09.087 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4119558078' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:09:09.126 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:09.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:09.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-14T04:09:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4119558078' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:09:09.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:09.245 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 98691 98675 93.4164 63.4375 0.0247807 0.0106991 2024-09-14T04:09:09.285 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:09.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:09.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:09.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:09.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:09.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:09.591 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:09.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:09.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:09.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:09.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:09.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:09.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:09.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:09.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:09.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:09.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:09.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:10.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:10.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:10.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:10.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:10.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:09 smithi114 ceph-mon[106969]: pgmap v743: 89 pgs: 89 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-14T04:09:10.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:09 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:09:10.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:09 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4119558078' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:09:10.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:09 smithi114 ceph-mon[106969]: osdmap e209: 8 total, 8 up, 7 in 2024-09-14T04:09:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:09 smithi114 ceph-mon[103029]: pgmap v743: 89 pgs: 89 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-14T04:09:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:09 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:09:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:09 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4119558078' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:09:10.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:09 smithi114 ceph-mon[103029]: osdmap e209: 8 total, 8 up, 7 in 2024-09-14T04:09:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:09 smithi160 ceph-mon[88407]: pgmap v743: 89 pgs: 89 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-14T04:09:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:09 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:09:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:09 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4119558078' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:09:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:09 smithi160 ceph-mon[88407]: osdmap e209: 8 total, 8 up, 7 in 2024-09-14T04:09:10.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:10.246 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 99694 99678 92.9565 62.6875 0.00962872 0.0107523 2024-09-14T04:09:10.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:10.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:10.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:10.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:10.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:10.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:10.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:10.536 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:10.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:10.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:10.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:09:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:09:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:09:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:10.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:10.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:10.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:10.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:10.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:10.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:10.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:10.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:10.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:11.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:11.110 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:11.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:11.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:11.246 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 100661 100645 92.4783 60.4375 0.0144377 0.0108079 2024-09-14T04:09:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:11.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:11.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:11.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:11.373 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:11.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:11.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:11.497 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:11.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:11.509 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:09:11.509 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:09:11.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:11.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:11.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:11.698 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:09:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:09:11] "GET /metrics HTTP/1.1" 200 36476 "" "Prometheus/2.51.0" 2024-09-14T04:09:11.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:11.770 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:11.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:11.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:09:11.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:09:11.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:09:11.852 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:09:11.852 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:09:11.852 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:09:11.852 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:09:11.852 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:09:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:09:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:09:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:09:11.853 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:09:11.854 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:09:11.854 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:09:11.854 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:09:11.859 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:11.867 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:09:11.867 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:09:11.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:11.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:11.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:11.970 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:11 smithi114 ceph-mon[103029]: pgmap v745: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 956 op/s 2024-09-14T04:09:11.970 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:11 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1818444081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:11.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:11 smithi114 ceph-mon[106969]: pgmap v745: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 956 op/s 2024-09-14T04:09:11.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:11 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1818444081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:11.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:12.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:12.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:12.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:11 smithi160 ceph-mon[88407]: pgmap v745: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 60 MiB/s wr, 956 op/s 2024-09-14T04:09:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:11 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1818444081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:12.180 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:12.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:12.246 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 101555 101539 91.9479 55.875 0.0136469 0.0108709 2024-09-14T04:09:12.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:12.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:12.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:12.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-14T04:09:12.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:12.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:12.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:12.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:12.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:12.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:12.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:12.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:12.857 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:12.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:12.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:12.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:12.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:12.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:12.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:13.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:13.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:13.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:13.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:13.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 102485 102469 91.4637 58.125 0.0105513 0.0109275 2024-09-14T04:09:13.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:13.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:13.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:13.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:13.400 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:13.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:13.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:13.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:13.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:13.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:13.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:13.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:13.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:13.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:13.836 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:13.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:13.938 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:13.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:14.037 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:13 smithi114 ceph-mon[103029]: pgmap v746: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 61 MiB/s wr, 979 op/s 2024-09-14T04:09:14.038 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:13 smithi114 ceph-mon[106969]: pgmap v746: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 61 MiB/s wr, 979 op/s 2024-09-14T04:09:14.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:14.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:14.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:13 smithi160 ceph-mon[88407]: pgmap v746: 89 pgs: 89 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 61 MiB/s wr, 979 op/s 2024-09-14T04:09:14.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:14.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:14.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:14.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 103431 103415 91.0083 59.125 0.00984409 0.0109826 2024-09-14T04:09:14.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:14.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:14.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:14.478 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:14.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:14.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:14.538 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:14.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:14.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:14.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:14.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:14.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:14.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:14.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:14.921 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:14.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:15.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:15.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:15.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:15.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:15.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:15.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:15.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 104430 104414 90.6115 62.4375 0.0105596 0.0110304 2024-09-14T04:09:15.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:15.329 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:15.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:15.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:15.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:15.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:15.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:15.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:15.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:15.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:15.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:09:15.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:15.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:15.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:15.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:15.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:15.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:16.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:16.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:16.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:16.076 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:15 smithi114 ceph-mon[106969]: pgmap v747: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 942 op/s 2024-09-14T04:09:16.076 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:15 smithi114 ceph-mon[103029]: pgmap v747: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 942 op/s 2024-09-14T04:09:16.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:16.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:15 smithi160 ceph-mon[88407]: pgmap v747: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 942 op/s 2024-09-14T04:09:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:16.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:16.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:16.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 105354 105338 90.1613 57.75 0.0143983 0.0110858 2024-09-14T04:09:16.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:16.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:16.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:16.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:16.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:16.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:16.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:16.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:16.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:16.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:16.704 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:16.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:16.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:16.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:16.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:16.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:16.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:16.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:17.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:17.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:17.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:17.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:17.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:17.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:17.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:17.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 106308 106292 89.7487 59.625 0.00797926 0.0111369 2024-09-14T04:09:17.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:17.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:17.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:17.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:17.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:17.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:17.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:17.671 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:17.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:17.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:17.810 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:17.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:17.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:17.950 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:17.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:18.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:18.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:18.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:17 smithi160 ceph-mon[88407]: pgmap v748: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 940 op/s 2024-09-14T04:09:18.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:18.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:18.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:18.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:18.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 107269 107253 89.3529 60.0625 0.0225877 0.0111862 2024-09-14T04:09:18.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:18.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:18.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:18.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:17 smithi114 ceph-mon[106969]: pgmap v748: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 940 op/s 2024-09-14T04:09:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:17 smithi114 ceph-mon[103029]: pgmap v748: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 59 MiB/s wr, 940 op/s 2024-09-14T04:09:18.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:18.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:18.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:18.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-14T04:09:18.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:18.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:18.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:18.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:18.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:18.814 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:18.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:18.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:18.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:18.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:18.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:19.024 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:19.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:19.144 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:19.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:19.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:19.249 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 108179 108163 88.9255 56.875 0.0149742 0.0112398 2024-09-14T04:09:19.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:19.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:19.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:19.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:19.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:19.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:19.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:19.574 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:19.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:19.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:19.700 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:19.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:19.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:19.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:19.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:19.925 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:09:19.925 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:09:19.925 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:09:19.925 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:09:19.925 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:09:19.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:19.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:19.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:20.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:20.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:20.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:20.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:19 smithi114 ceph-mon[103029]: pgmap v749: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 59 MiB/s wr, 951 op/s 2024-09-14T04:09:20.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:19 smithi114 ceph-mon[106969]: pgmap v749: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 59 MiB/s wr, 951 op/s 2024-09-14T04:09:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:19 smithi160 ceph-mon[88407]: pgmap v749: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 59 MiB/s wr, 951 op/s 2024-09-14T04:09:20.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:20.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 109096 109080 88.5146 57.3125 0.0139255 0.0112923 2024-09-14T04:09:20.252 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:20.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:20.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:20.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:20.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:20.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:20.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:20.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:20.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:20.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:20.645 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:20.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:20.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:09:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:09:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:09:20.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:20.788 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:20.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:20.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:20.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:20.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:20.933 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:20.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:21.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:21.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:21.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:21.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:21.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:21.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:21.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 110032 110016 88.1298 58.5 0.0173364 0.0113418 2024-09-14T04:09:21.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:21.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:21.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:21.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:21.460 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:21.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:21.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:21.590 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:21.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:21.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:21.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:09:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:09:21] "GET /metrics HTTP/1.1" 200 36476 "" "Prometheus/2.51.0" 2024-09-14T04:09:21.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:21.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:21.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:21.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:21.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-14T04:09:21.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:21.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:21.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:21.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:22.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:22.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:22.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:21 smithi114 ceph-mon[106969]: pgmap v750: 89 pgs: 89 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 56 MiB/s wr, 897 op/s 2024-09-14T04:09:22.061 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:21 smithi114 ceph-mon[103029]: pgmap v750: 89 pgs: 89 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 56 MiB/s wr, 897 op/s 2024-09-14T04:09:22.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:22.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:21 smithi160 ceph-mon[88407]: pgmap v750: 89 pgs: 89 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 56 MiB/s wr, 897 op/s 2024-09-14T04:09:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:22.179 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:22.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:22.249 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 110922 110906 87.7174 55.625 0.0194562 0.0113941 2024-09-14T04:09:22.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:22.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:22.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:22.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:22.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:22.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:22.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:22.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:22.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:22.616 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:09:22.617 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:09:22.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:09:22.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:09:22.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:22.704 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:09:22.705 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:09:22.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:22.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:22.755 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:09:22.756 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:09:22.789 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:09:22.789 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:09:22.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:22.825 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:09:22.826 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:09:22.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:22.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:22.860 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:09:22.860 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:09:22.895 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:09:22.895 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:09:22.915 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:22 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1735323183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:22.915 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:22.915 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:22 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2712262133' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:09:22.915 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:22 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:09:22.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:22 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1735323183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:22 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2712262133' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:09:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:22 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:09:22.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:22.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:22.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:23.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:23.117 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:23.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:22 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1735323183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:22 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2712262133' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:09:23.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:22 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:09:23.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:23.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:23.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:23.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:23.249 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:09:23.248961+0000 min lat: 0.00109236 max lat: 0.357961 avg lat: 0.0114561 2024-09-14T04:09:23.249 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:09:23.249 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 111727 111711 87.2498 50.3125 0.0183691 0.0114561 2024-09-14T04:09:23.263 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:09:23.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:23.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:23.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:23.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:23.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:23.511 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:23.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:23.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:23.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:23.644 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:23.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:23.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:23.788 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:23.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:23.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:23.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:23.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:24.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:24.073 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:24.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:23 smithi160 ceph-mon[88407]: pgmap v751: 89 pgs: 89 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 54 MiB/s wr, 864 op/s 2024-09-14T04:09:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:23 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:09:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:23 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:09:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:23 smithi160 ceph-mon[88407]: osdmap e210: 8 total, 8 up, 7 in 2024-09-14T04:09:24.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:24.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:24.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:24.249 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 112618 112602 86.8602 55.6875 0.0112452 0.0115077 2024-09-14T04:09:24.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:24.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:24.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:24.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:23 smithi114 ceph-mon[103029]: pgmap v751: 89 pgs: 89 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 54 MiB/s wr, 864 op/s 2024-09-14T04:09:24.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:23 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:09:24.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:23 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:09:24.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:23 smithi114 ceph-mon[103029]: osdmap e210: 8 total, 8 up, 7 in 2024-09-14T04:09:24.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:23 smithi114 ceph-mon[106969]: pgmap v751: 89 pgs: 89 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 54 MiB/s wr, 864 op/s 2024-09-14T04:09:24.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:23 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:09:24.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:23 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:09:24.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:23 smithi114 ceph-mon[106969]: osdmap e210: 8 total, 8 up, 7 in 2024-09-14T04:09:24.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:24.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:24.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:24.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:24.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:24.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:24.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:24.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:24.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:24.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:24.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:24.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:24.863 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:24.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:24.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:24.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:24.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:24.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:25.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:25.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:25.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:25.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:25.252 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 113512 113496 86.4816 55.875 0.00840586 0.0115577 2024-09-14T04:09:25.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:25.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:25.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:25.396 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:25.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:25.498 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:25.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:25.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:25.578 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:25.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:25.708 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:25.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:25.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:25.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:25.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:25.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:25.986 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:25.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:26.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:26.109 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:25 smithi114 ceph-mon[103029]: pgmap v753: 89 pgs: 89 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 58 MiB/s wr, 921 op/s 2024-09-14T04:09:26.110 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:25 smithi114 ceph-mon[106969]: pgmap v753: 89 pgs: 89 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 58 MiB/s wr, 921 op/s 2024-09-14T04:09:26.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:26.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:25 smithi160 ceph-mon[88407]: pgmap v753: 89 pgs: 89 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 58 MiB/s wr, 921 op/s 2024-09-14T04:09:26.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:26.250 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 15 114448 114433 86.1452 58.5625 0.013785 0.0116027 2024-09-14T04:09:26.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:26.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:26.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:26.392 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:26.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:26.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:26.529 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:26.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:26.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:26.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:26.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:26.804 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:26.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:26.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:26.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:26.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:26.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:27.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:27.101 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:27.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:27.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:27.253 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:27.253 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 115366 115350 85.8004 57.3125 0.0343514 0.0116495 2024-09-14T04:09:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:27.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:27.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:27.394 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:27.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:27.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:27.532 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:27.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:27.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:27.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:27.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:27.709 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:27.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:27.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:27.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:27.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:27.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:27.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:27.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:28.021 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:28.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:28.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:28.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:28.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:09:28.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:28.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:28.191 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:27 smithi114 ceph-mon[106969]: pgmap v754: 89 pgs: 89 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-14T04:09:28.192 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:27 smithi114 ceph-mon[103029]: pgmap v754: 89 pgs: 89 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-14T04:09:28.252 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 116285 116269 85.4667 57.4375 0.0226032 0.011695 2024-09-14T04:09:28.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:28.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:27 smithi160 ceph-mon[88407]: pgmap v754: 89 pgs: 89 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-14T04:09:28.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:28.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:28.475 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:28.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:28.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:28.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:28.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:28.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:28.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:28.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:28.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:09:28.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:28.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:28.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:28.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:29.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:29.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:29.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:29.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:29.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:29.252 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 117106 117090 85.0696 51.3125 0.0232434 0.0117503 2024-09-14T04:09:29.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:29.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:29.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:29.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:29.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:29.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:29.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:29.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:29.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:29.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:29.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:29.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:29.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:29.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:29.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:29.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:29.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:29.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:29.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:30.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:30.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:30.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:30.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:30.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:29 smithi114 ceph-mon[103029]: pgmap v755: 89 pgs: 89 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 44 MiB/s wr, 708 op/s 2024-09-14T04:09:30.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:29 smithi114 ceph-mon[106969]: pgmap v755: 89 pgs: 89 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 44 MiB/s wr, 708 op/s 2024-09-14T04:09:30.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:30.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:30.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:30.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:29 smithi160 ceph-mon[88407]: pgmap v755: 89 pgs: 89 active+clean; 6.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 44 MiB/s wr, 708 op/s 2024-09-14T04:09:30.252 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 117808 117792 84.5962 43.875 0.00999549 0.0118156 2024-09-14T04:09:30.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:30.329 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:30.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:30.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:30.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:30.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:30.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:30.618 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:30.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:30.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:30.660 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:09:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:09:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:09:30.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:30.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:30.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:30.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:30.897 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:30.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:30.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:30.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:31.055 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:31.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:31.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:31.162 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:30 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:09:31.198 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:31.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:31.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:31.211 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:30 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:09:31.211 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:30 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:09:31.262 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 118653 118637 84.2255 52.8125 0.0228398 0.0118675 2024-09-14T04:09:31.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:31.343 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:31.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:31.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:31.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:31.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:31.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:31.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:31.536 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:09:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:09:31] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-09-14T04:09:31.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:31.625 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:31.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:31.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:31.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:31.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:31.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:31.888 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:31.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:31.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:32.043 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:32.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:32.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:31 smithi160 ceph-mon[88407]: pgmap v756: 89 pgs: 89 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 56 MiB/s wr, 897 op/s 2024-09-14T04:09:32.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:32.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:32.212 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:31 smithi114 ceph-mon[106969]: pgmap v756: 89 pgs: 89 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 56 MiB/s wr, 897 op/s 2024-09-14T04:09:32.212 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:31 smithi114 ceph-mon[103029]: pgmap v756: 89 pgs: 89 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 56 MiB/s wr, 897 op/s 2024-09-14T04:09:32.225 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:09:32.225 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:09:32.262 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 119424 119408 83.8207 48.1875 0.0297639 0.0119238 2024-09-14T04:09:32.284 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:32.320 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:32.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:32.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:32.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:32.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:32.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:32.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:32.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:32.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:32.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:32.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:32.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:32.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:32.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:32.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:32.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:33.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:33.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:33.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:33.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-14T04:09:33.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:33.143 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:33.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:33.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:33.262 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0079 2024-09-14T04:09:33.262 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 120193 2024-09-14T04:09:33.262 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-14T04:09:33.262 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-14T04:09:33.262 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 83.46 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 37.2231 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 228.438 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 43.875 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1335 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 595.569 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3655 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 702 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0119775 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0115765 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 0.357961 2024-09-14T04:09:33.263 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00109236 2024-09-14T04:09:33.268 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-14T04:09:33.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:33.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:33.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:33.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:33.489 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:33.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:33.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-14T04:09:33.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:33.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:33.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:33.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:33.651 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:33.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:33.756 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:33.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:33.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:33.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:33.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:33.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:33.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:34.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:34.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:34.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:33 smithi160 ceph-mon[88407]: pgmap v757: 89 pgs: 89 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s 2024-09-14T04:09:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:34.179 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:34.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:34.235 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:33 smithi114 ceph-mon[106969]: pgmap v757: 89 pgs: 89 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s 2024-09-14T04:09:34.235 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:33 smithi114 ceph-mon[103029]: pgmap v757: 89 pgs: 89 active+clean; 7.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s 2024-09-14T04:09:34.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:34.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:34.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:34.435 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:34.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:34.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:34.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:34.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:34.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:34.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:34.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:34.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:34.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:34.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:34.884 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:34.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:34.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:35.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:35.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:35.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:35.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:35.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:35.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:35.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:35.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:35.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:35.434 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:35.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:35.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:35.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:35.591 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:35.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:35.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:35.747 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:35.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:35.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:35.892 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:35.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:35.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:36.040 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:36.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:36.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:36.177 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:36.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:36.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:36.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:36.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:36.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:36.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:36.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:36.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:36.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:36 smithi114 ceph-mon[106969]: pgmap v758: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 86 B/s rd, 62 MiB/s wr, 1.01k op/s 2024-09-14T04:09:36.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:36 smithi114 ceph-mon[103029]: pgmap v758: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 86 B/s rd, 62 MiB/s wr, 1.01k op/s 2024-09-14T04:09:36.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:36.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:36 smithi160 ceph-mon[88407]: pgmap v758: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 86 B/s rd, 62 MiB/s wr, 1.01k op/s 2024-09-14T04:09:36.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:36.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:36.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:36.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:36.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:36.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:36.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:37.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:37.071 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:37.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:37.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:37.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:37.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:37.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:37.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:37.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:37.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:37.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:37.484 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:37.484 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:37.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:37.484 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:37.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:37.596 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:37.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:37.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:37.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:37.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:37.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:37.827 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:09:37.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:09:37.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:09:37.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:09:37.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:09:37.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:09:37.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:09:37.828 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:09:37.828 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:09:37.828 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:09:37.828 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:09:37.829 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:09:37.829 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:09:37.829 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:09:37.829 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:09:37.829 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:09:37.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:37.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:37.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:38.008 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:38.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:38.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:38.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:38.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:38.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:38.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:38.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:38.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:38.400 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:38.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:38.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:38.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:38.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:38.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:38.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:38.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:38.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:38.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:38.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:38 smithi114 ceph-mon[106969]: pgmap v759: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 43 MiB/s wr, 746 op/s 2024-09-14T04:09:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/160348861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:38.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:38 smithi114 ceph-mon[103029]: pgmap v759: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 43 MiB/s wr, 746 op/s 2024-09-14T04:09:38.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/160348861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:38.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:38 smithi160 ceph-mon[88407]: pgmap v759: 89 pgs: 89 active+clean; 7.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 43 MiB/s wr, 746 op/s 2024-09-14T04:09:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/160348861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:38.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:38.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:38.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:39.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:39.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:39.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:39.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:39.187 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:39.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:39.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:39.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:39.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:39.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:39.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:39.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:39.530 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:39.556 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:39.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:39.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:39.689 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:39.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:39.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:39.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:39.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:39.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:39.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:39.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:39.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:39.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:39.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:40.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:40.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:40.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:40.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:40.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:40.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:40.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:40.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:40.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:40.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:40.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:40.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:40.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:40 smithi114 ceph-mon[103029]: pgmap v760: 89 pgs: 89 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 40 MiB/s wr, 923 op/s 2024-09-14T04:09:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:40 smithi114 ceph-mon[106969]: pgmap v760: 89 pgs: 89 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 40 MiB/s wr, 923 op/s 2024-09-14T04:09:40.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:09:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:09:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:09:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:40 smithi160 ceph-mon[88407]: pgmap v760: 89 pgs: 89 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 40 MiB/s wr, 923 op/s 2024-09-14T04:09:40.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:40.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:40.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:40.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:40.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:40.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:40.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:40.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:41.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:41.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:41.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:41.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:41.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:41.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:41.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:41.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:41.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:41.460 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:09:41.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:41.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:41.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:41.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:41.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:41.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:41.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:41.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:41.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:41 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2886426985' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:09:41.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:41 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:09:41.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:41 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2886426985' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:09:41.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:41 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:09:41.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:09:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:09:41] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-09-14T04:09:41.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:41.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-14T04:09:41.765 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:41.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:41.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:41.895 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:41.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:41.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:41 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2886426985' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:09:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:41 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:09:41.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:42.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:42.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:42.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:09:42.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:42.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:42.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:42.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:42.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:42.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:42.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:42.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:42.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:42.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:42.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:42.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:42.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:42.677 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:42.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:42.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:42.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:42.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:42.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:42 smithi114 ceph-mon[106969]: pgmap v761: 89 pgs: 89 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 36 MiB/s wr, 948 op/s 2024-09-14T04:09:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:42 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:09:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:42 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:09:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:42 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:09:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:42 smithi114 ceph-mon[106969]: osdmap e211: 8 total, 8 up, 7 in 2024-09-14T04:09:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:42 smithi114 ceph-mon[103029]: pgmap v761: 89 pgs: 89 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 36 MiB/s wr, 948 op/s 2024-09-14T04:09:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:42 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:09:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:42 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:09:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:42 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:09:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:42 smithi114 ceph-mon[103029]: osdmap e211: 8 total, 8 up, 7 in 2024-09-14T04:09:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:42 smithi160 ceph-mon[88407]: pgmap v761: 89 pgs: 89 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 36 MiB/s wr, 948 op/s 2024-09-14T04:09:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:42 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:09:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:42 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:09:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:42 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:09:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:42 smithi160 ceph-mon[88407]: osdmap e211: 8 total, 8 up, 7 in 2024-09-14T04:09:42.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:42.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:42.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:43.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:43.088 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:43.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:43.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:43.220 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:43.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:43.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:43.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:43.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:43.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:43.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:43.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:43.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:43.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:43.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:43.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:43.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:43.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:43.748 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:43.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:43.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:43.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:43.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:43.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:43.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:43.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:44.021 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:44.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:44.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:44.145 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:44.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:44.218 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:09:44.218 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:09:44.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.218 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.35997.0:129687 10.2 10:47cca2f0:::benchmark_data_smithi114_188657_object9491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-14T04:09:44.218 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:09:44.129014+0000", 2024-09-14T04:09:44.218 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.014167338, 2024-09-14T04:09:44.218 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014197670000000001, 2024-09-14T04:09:44.218 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:09:44.218 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.35997", 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 129687 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.129014+0000", 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:09:44.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.129014+0000", 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.129016+0000", 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3030000000000001e-06 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:09:44.220 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.129020+0000", 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4859999999999999e-06 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.129041+0000", 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0296e-05 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.129048+0000", 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5240000000000003e-06 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.130514+0000", 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0014661100000000001 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.130631+0000", 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000116335 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.130758+0000", 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012781199999999999 2024-09-14T04:09:44.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.140583+0000", 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0098244960000000003 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.143139+0000", 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025560159999999999 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:129684 10.4 e211/199 10:246c3e24:::benchmark_data_smithi114_188657_object9488:head v 211'8050, mlcod=211'8048)", 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:09:44.139731+0000", 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0034509110000000001, 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035555819999999998, 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.139731+0000", 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:09:44.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.139731+0000", 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.139732+0000", 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9199999999999999e-07 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:09:44.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.139738+0000", 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4790000000000001e-06 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.139755+0000", 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6650000000000002e-05 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.139763+0000", 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8679999999999999e-06 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.139950+0000", 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018688799999999999 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.139994+0000", 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4709000000000002e-05 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:09:44.227 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:09:44.228 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 2 2024-09-14T04:09:44.228 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:09:44.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:44.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:44.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:44.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:44.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:44.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:44.582 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:44.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:44.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:44.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:44.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:44.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:44.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:44.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:44.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:44.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:44 smithi114 ceph-mon[103029]: pgmap v763: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 19 MiB/s wr, 839 op/s 2024-09-14T04:09:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:44 smithi114 ceph-mon[106969]: pgmap v763: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 19 MiB/s wr, 839 op/s 2024-09-14T04:09:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:44 smithi160 ceph-mon[88407]: pgmap v763: 89 pgs: 89 active+clean; 7.0 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 19 MiB/s wr, 839 op/s 2024-09-14T04:09:44.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:44.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:44.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:45.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-14T04:09:45.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:45.131 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:45.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:45.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:45.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:45.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:45.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:45.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:45.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:45.365 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:45.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:09:45.372 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:45.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:45.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:45.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:45.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:45.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:45.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:45.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:45.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:45.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:45.868 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:45.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:45.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:45 smithi114 ceph-mon[103029]: pgmap v764: 89 pgs: 89 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.4 MiB/s wr, 876 op/s 2024-09-14T04:09:45.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:45 smithi114 ceph-mon[106969]: pgmap v764: 89 pgs: 89 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.4 MiB/s wr, 876 op/s 2024-09-14T04:09:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:45 smithi160 ceph-mon[88407]: pgmap v764: 89 pgs: 89 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.4 MiB/s wr, 876 op/s 2024-09-14T04:09:45.970 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:45.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:45.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:46.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:46.134 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:46.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:46.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:46.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:46.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:46.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:46.401 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:46.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:46.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:46.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:46.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:46.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:46.672 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:46.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:46.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:46.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:46.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:46.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:46.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:46.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:47.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:47.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:47.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:47.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:47.208 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:47.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:47.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:47.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:47.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:47.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:47.489 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:47.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:47.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:47.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:47.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:47.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:47.771 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:47.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:47.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:47.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:47.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:47.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:48.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:48.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:48.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:48.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:47 smithi114 ceph-mon[106969]: pgmap v765: 89 pgs: 89 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 371 KiB/s wr, 876 op/s 2024-09-14T04:09:48.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:47 smithi114 ceph-mon[103029]: pgmap v765: 89 pgs: 89 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 371 KiB/s wr, 876 op/s 2024-09-14T04:09:48.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:48.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:47 smithi160 ceph-mon[88407]: pgmap v765: 89 pgs: 89 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 371 KiB/s wr, 876 op/s 2024-09-14T04:09:48.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:48.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:48.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:48.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:48.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:48.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:48.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:48.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:48.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:48.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:48.574 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:48.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:48.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:48.720 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:48.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:48.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:48.773 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:48.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:48.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:48.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:49.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:49.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:49.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:49.151 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:49.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:49.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:49.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:49.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:49.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:49.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:49.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:49.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:49.536 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:49.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:49.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:49.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:49.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:49.711 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:49.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:49.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:49.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:49.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:49.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:49.972 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:09:49.972 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:09:50.013 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:50.013 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:09:50.013 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:09:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:50.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:50.053 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:09:50.053 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:09:50.089 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:49 smithi114 ceph-mon[103029]: pgmap v766: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 687 op/s 2024-09-14T04:09:50.089 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:49 smithi114 ceph-mon[106969]: pgmap v766: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 687 op/s 2024-09-14T04:09:50.089 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:09:50.090 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:09:50.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:50.117 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:09:50.117 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:09:50.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:50.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:50.149 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:09:50.149 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:09:50.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:49 smithi160 ceph-mon[88407]: pgmap v766: 89 pgs: 89 active+clean; 6.7 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 687 op/s 2024-09-14T04:09:50.182 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:09:50.182 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:09:50.211 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:09:50.212 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:09:50.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:50.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:50.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:50.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:50.418 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:50.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:50.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:50.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:50.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:09:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:09:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:09:50.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:50.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:50.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:50.812 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:50.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:50.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:50.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:50.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:50.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:50.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:51.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:51.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:51.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:51.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:51.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:50 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/456536610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:51.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:50 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/456536610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:51.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:50 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/456536610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:09:51.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:51.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:51.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:51.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:51.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:51.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:51.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:51.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:51.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:51.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:51.651 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:51.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:51.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:51.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:51.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:51.852 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:51 smithi114 ceph-mon[103029]: pgmap v767: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 878 op/s 2024-09-14T04:09:51.852 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:09:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:09:51] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-09-14T04:09:51.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:51.904 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:51.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:52.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:52.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:52.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:52.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:52.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:51 smithi114 ceph-mon[106969]: pgmap v767: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 878 op/s 2024-09-14T04:09:52.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:51 smithi160 ceph-mon[88407]: pgmap v767: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 878 op/s 2024-09-14T04:09:52.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:52.211 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:52.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:52.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:52.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:52.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:52.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:52.472 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:52.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:52.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:52.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:52.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:52.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:52.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:52.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:52.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:52.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:52.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:52.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:53.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:53.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:53.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:53.134 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:53.160 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:53.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:53.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:53.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:53.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:53.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:53.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:53.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:53.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:53.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:53.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:53.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:09:53.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:53.717 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:53.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:53.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:53.843 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:53.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:53.868 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:53.868 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:53.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:53.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:53.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:53 smithi114 ceph-mon[106969]: pgmap v768: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 849 op/s 2024-09-14T04:09:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:53 smithi114 ceph-mon[103029]: pgmap v768: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 849 op/s 2024-09-14T04:09:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:09:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:53 smithi160 ceph-mon[88407]: pgmap v768: 89 pgs: 89 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 849 op/s 2024-09-14T04:09:53.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:53.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:54.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:54.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:54.151 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:54.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:54.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:54.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:54.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:54.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:54.432 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:54.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:54.512 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:09:54.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:54.590 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:54.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:54.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:54.692 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:54.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:54.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:54.813 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:09:54.813 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:09:54.813 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:54.813 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:09:54.814 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:09:54.814 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:54.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:54.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:54.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:09:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:09:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:09:54.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:54.996 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:54.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:55.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:55.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:55.155 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:55.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:55.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:55.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:55.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:55.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:55.432 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:55.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:55.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:55.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:55.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:55.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:55.671 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:55.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:55.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:55.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:55.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:09:55.838 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:55.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:55.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:09:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:55 smithi160 ceph-mon[88407]: pgmap v769: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-14T04:09:55.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:55.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:55.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:56.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:56.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:56.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:56.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:55 smithi114 ceph-mon[106969]: pgmap v769: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-14T04:09:56.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:55 smithi114 ceph-mon[103029]: pgmap v769: 89 pgs: 89 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-14T04:09:56.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:56.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:56.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:56.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:56.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:56.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:56.466 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:56.497 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:56.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:56.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:56.623 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:56.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:56.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:56.725 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:56.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:56.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:56.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:56.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:56.886 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:56.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:56.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:57.022 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:57.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:57.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:57.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:57.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:57.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:57.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:57.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:57.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:57.428 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:57.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:57.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:57.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:57.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:57.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:57.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:57.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:57.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:57.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:57.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:57.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:57.950 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:57.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:58.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:58.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:58.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:58.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:57 smithi114 ceph-mon[106969]: pgmap v770: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 768 op/s 2024-09-14T04:09:58.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:57 smithi114 ceph-mon[103029]: pgmap v770: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 768 op/s 2024-09-14T04:09:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:57 smithi160 ceph-mon[88407]: pgmap v770: 89 pgs: 89 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 768 op/s 2024-09-14T04:09:58.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:58.226 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:58.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:58.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:58.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:58.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:58.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:58.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:58.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:58.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:58.624 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:58.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:58.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:09:58.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:58.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:09:58.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:09:58.897 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:58.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:09:58.907 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:09:58.907 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:09:58.907 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-14T04:09:58.907 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:09:58.908 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:09:58.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-14T04:09:58.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:09:59.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:59.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:09:59.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:59.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:59.180 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:59.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:59.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:59.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:59.305 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:59.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:59.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:59.447 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:59.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:59.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:59.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:09:59.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:59.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:59.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:09:59.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:09:59.706 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:59.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:59.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:09:59.808 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:09:59.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:09:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:59.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:09:59.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:09:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:09:59.979 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:09:59.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:09:59.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:09:59 smithi114 ceph-mon[106969]: pgmap v771: 89 pgs: 89 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-14T04:09:59.991 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:09:59 smithi114 ceph-mon[103029]: pgmap v771: 89 pgs: 89 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-14T04:10:00.008 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:00.045 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:10:00.045 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:10:00.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:00.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:00.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:00.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:09:59 smithi160 ceph-mon[88407]: pgmap v771: 89 pgs: 89 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-09-14T04:10:00.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:00.276 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:00.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:00.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:00.403 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:00.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:00.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:00.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:00.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:00.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:10:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:10:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:10:00.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:00.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:00.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:00.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:00.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:00.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:00.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:00.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:00.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:01.023 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:00 smithi160 ceph-mon[88407]: overall HEALTH_OK 2024-09-14T04:10:01.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:01.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:01.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:01.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:01.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:00 smithi114 ceph-mon[103029]: overall HEALTH_OK 2024-09-14T04:10:01.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:00 smithi114 ceph-mon[106969]: overall HEALTH_OK 2024-09-14T04:10:01.182 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:01.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:01.257 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:01.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:01.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:01.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:01.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:01.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:01.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:01.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:01.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:01.659 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:01.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:01.688 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:10:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:10:01] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-14T04:10:01.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:01.805 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:01.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:01.884 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:10:01.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:01.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:01.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:01.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:02.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:01 smithi114 ceph-mon[103029]: pgmap v772: 89 pgs: 89 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 980 op/s 2024-09-14T04:10:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:01 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4122154341' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:10:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:01 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:10:02.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:01 smithi114 ceph-mon[106969]: pgmap v772: 89 pgs: 89 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 980 op/s 2024-09-14T04:10:02.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:01 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4122154341' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:10:02.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:01 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:10:02.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:02.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:01 smithi160 ceph-mon[88407]: pgmap v772: 89 pgs: 89 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 980 op/s 2024-09-14T04:10:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:01 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4122154341' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:10:02.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:01 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:10:02.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:02.235 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:02.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:02.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:02.388 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:02.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:02.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:02.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:02.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:02.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:02.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:02.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:02.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:02.799 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:02.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:02.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:02.901 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:02.926 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:02.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:02.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:03.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:03.072 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:03.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:03.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:02 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:02 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:10:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:02 smithi114 ceph-mon[103029]: osdmap e212: 8 total, 8 up, 7 in 2024-09-14T04:10:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:02 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:02 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:10:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:02 smithi114 ceph-mon[106969]: osdmap e212: 8 total, 8 up, 7 in 2024-09-14T04:10:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:02 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:02 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:10:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:02 smithi160 ceph-mon[88407]: osdmap e212: 8 total, 8 up, 7 in 2024-09-14T04:10:03.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:03.201 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:03.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:03.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:03.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:03.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:03.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:03.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:03.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:03.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:03.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:03.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-14T04:10:03.627 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:03.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:03.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:03.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:03.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:03.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:03.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:03.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:03.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:04.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:04.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:04.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:03 smithi114 ceph-mon[106969]: pgmap v774: 89 pgs: 89 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 867 op/s 2024-09-14T04:10:04.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:03 smithi114 ceph-mon[103029]: pgmap v774: 89 pgs: 89 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 867 op/s 2024-09-14T04:10:04.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:04.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:04.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:04.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:03 smithi160 ceph-mon[88407]: pgmap v774: 89 pgs: 89 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 867 op/s 2024-09-14T04:10:04.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:04.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:04.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:04.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:04.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:04.437 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:04.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:04.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:04.582 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:04.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:04.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:04.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:04.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:04.757 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:04.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:04.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:04.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:05.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:05.040 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:05.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:05.141 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:05.166 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:05.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:05.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:05.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:05.293 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:05.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:05.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:05.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:05.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:05.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:05.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:05.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:05.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:05.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:05.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:05.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:05.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:05.835 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:05.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:05.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:05.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:05.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:06.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:06.110 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:06.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:06.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:05 smithi114 ceph-mon[106969]: pgmap v775: 89 pgs: 89 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-14T04:10:06.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:05 smithi114 ceph-mon[103029]: pgmap v775: 89 pgs: 89 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-14T04:10:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:05 smithi160 ceph-mon[88407]: pgmap v775: 89 pgs: 89 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-14T04:10:06.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:06.239 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:06.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:06.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:06.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:06.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:06.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:06.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:06.507 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:06.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:06.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:06.609 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:06.633 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:06.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:06.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:06.775 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:06.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:06.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:06.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:06.908 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:10:06.908 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-14T04:10:06.908 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:10:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:06.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:06.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:07.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:07.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:07.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:07.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:07.195 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:07.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:07.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:10:07.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:07.328 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:10:07.328 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:10:07.328 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:10:07.328 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:10:07.328 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:10:07.329 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:10:07.329 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:10:07.329 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:10:07.332 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:10:07.332 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:10:07.332 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:10:07.332 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:10:07.332 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:10:07.332 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:10:07.332 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:10:07.332 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:10:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:07.358 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:07.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:07.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:07.501 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:07.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:07.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:07.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:07.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:07.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:07.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:07.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:07.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:07.885 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:07.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:07.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:07.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:07 smithi114 ceph-mon[103029]: pgmap v776: 89 pgs: 89 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-14T04:10:07.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:07 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1644923167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:07.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:10:07.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:07.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:07 smithi114 ceph-mon[106969]: pgmap v776: 89 pgs: 89 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-14T04:10:07.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:07 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1644923167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:07.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:10:07.989 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:08.016 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:08.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:08.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:08.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:08.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:08.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:08.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:07 smithi160 ceph-mon[88407]: pgmap v776: 89 pgs: 89 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-14T04:10:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:07 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1644923167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:10:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:08.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:08.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:08.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:08.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:08.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:08.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:08.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:08.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:08.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:08.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:08.670 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:08.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:08.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:08.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:08.838 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:08.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:08.940 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:08.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:08.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:09.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:09.110 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:09.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:09.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:09.237 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:09.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:09.339 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:09.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:09.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:09.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:09.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:09.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:09.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:09.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:09.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:09.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:09.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:09.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:09.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:09.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:09.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:09.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:10.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:10.045 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:10.074 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:10.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:09 smithi160 ceph-mon[88407]: pgmap v777: 89 pgs: 89 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 650 op/s 2024-09-14T04:10:10.168 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:09 smithi114 ceph-mon[106969]: pgmap v777: 89 pgs: 89 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 650 op/s 2024-09-14T04:10:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:09 smithi114 ceph-mon[103029]: pgmap v777: 89 pgs: 89 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 650 op/s 2024-09-14T04:10:10.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:10.226 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:10.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:10.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:10.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:10.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:10.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:10.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:10:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:10.499 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:10.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:10.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:10.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:10.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:10:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:10:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:10:10.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:10.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:10.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:10.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:10.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:10.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:10.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:11.008 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:10:11.008 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:10:11.008 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45572.0:195992 9.4 9:2c55a968:::benchmark_data_smithi114_153893_object66531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:06:06.801304+0000", 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 244.202066282, 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31291260399999998, 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45572", 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:10:11.009 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 195992 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.801304+0000", 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.801304+0000", 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.801305+0000", 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.626e-06 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.801310+0000", 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5229999999999997e-06 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.801319+0000", 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3409999999999992e-06 2024-09-14T04:10:11.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.801325+0000", 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4010000000000002e-06 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.801447+0000", 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012170700000000001 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.801575+0000", 2024-09-14T04:10:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012780800000000001 2024-09-14T04:10:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3,4", 2024-09-14T04:10:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.801851+0000", 2024-09-14T04:10:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00027607599999999999 2024-09-14T04:10:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.805162+0000", 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033106160000000002 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.805282+0000", 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000120594 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:06.805363+0000", 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0537000000000005e-05 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:07.113809+0000", 2024-09-14T04:10:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30844587800000001 2024-09-14T04:10:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:07.113969+0000", 2024-09-14T04:10:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016054200000000001 2024-09-14T04:10:11.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:07.114216+0000", 2024-09-14T04:10:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024695499999999998 2024-09-14T04:10:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45572.0:220863 9.4 9:2325a590:::benchmark_data_smithi114_153893_object91402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-14T04:10:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:06:39.942887+0000", 2024-09-14T04:10:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 211.060482342, 2024-09-14T04:10:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.25805054999999999, 2024-09-14T04:10:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.017 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45572", 2024-09-14T04:10:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:10:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 220863 2024-09-14T04:10:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.942887+0000", 2024-09-14T04:10:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.942887+0000", 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.942889+0000", 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8109999999999999e-06 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.942895+0000", 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8429999999999998e-06 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.942912+0000", 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7044000000000001e-05 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.942921+0000", 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0019999999999995e-06 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.020 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.942980+0000", 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9128e-05 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.943068+0000", 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8139000000000004e-05 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.943143+0000", 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4558e-05 2024-09-14T04:10:11.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.946221+0000", 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030784839999999998 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.947364+0000", 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001142987 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:40.200763+0000", 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.25339873800000001 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:40.200892+0000", 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000128551 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:40.200938+0000", 2024-09-14T04:10:11.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6264999999999998e-05 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45572.0:220878 9.4 9:262f42bf:::benchmark_data_smithi114_153893_object91417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:06:39.966590+0000", 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 211.03677954299999, 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23455796600000001, 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.024 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.025 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45572", 2024-09-14T04:10:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:10:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 220878 2024-09-14T04:10:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.966590+0000", 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.966590+0000", 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.966592+0000", 2024-09-14T04:10:11.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7719999999999999e-06 2024-09-14T04:10:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.966596+0000", 2024-09-14T04:10:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9230000000000004e-06 2024-09-14T04:10:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.966608+0000", 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2384e-05 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.966614+0000", 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9320000000000001e-06 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.966672+0000", 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7309000000000003e-05 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.966806+0000", 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013468599999999999 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.966893+0000", 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6607999999999995e-05 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.969144+0000", 2024-09-14T04:10:11.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002250939 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:39.969411+0000", 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00026669600000000002 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:40.201067+0000", 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23165602199999999 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:40.201116+0000", 2024-09-14T04:10:11.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9237999999999999e-05 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:06:40.201148+0000", 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2456999999999998e-05 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45572.0:247624 9.4 9:2dcdd696:::benchmark_data_smithi114_153893_object118163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:07:12.294118+0000", 2024-09-14T04:10:11.033 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 178.709251513, 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.22673328200000001, 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45572", 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 247624 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294118+0000", 2024-09-14T04:10:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294118+0000", 2024-09-14T04:10:11.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294119+0000", 2024-09-14T04:10:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2099999999999995e-07 2024-09-14T04:10:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294123+0000", 2024-09-14T04:10:11.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0450000000000002e-06 2024-09-14T04:10:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294134+0000", 2024-09-14T04:10:11.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0998e-05 2024-09-14T04:10:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294141+0000", 2024-09-14T04:10:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.657e-06 2024-09-14T04:10:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.297882+0000", 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037410310000000001 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.298003+0000", 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000120857 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.298120+0000", 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000116778 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.302540+0000", 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044207109999999999 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.302564+0000", 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3342000000000001e-05 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.520675+0000", 2024-09-14T04:10:11.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.218111044 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.520724+0000", 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9808999999999997e-05 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.520852+0000", 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012708899999999999 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.041 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45572.0:247625 9.4 9:20b4219a:::benchmark_data_smithi114_153893_object118164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:07:12.294565+0000", 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 178.708805111, 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23468599200000001, 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45572", 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:10:11.042 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 247625 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294565+0000", 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294565+0000", 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294566+0000", 2024-09-14T04:10:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0999999999999997e-07 2024-09-14T04:10:11.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294570+0000", 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1910000000000002e-06 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294580+0000", 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0008e-05 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.294586+0000", 2024-09-14T04:10:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4339999999999997e-06 2024-09-14T04:10:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.300558+0000", 2024-09-14T04:10:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00597184 2024-09-14T04:10:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.300650+0000", 2024-09-14T04:10:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2282999999999997e-05 2024-09-14T04:10:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.300717+0000", 2024-09-14T04:10:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6643000000000006e-05 2024-09-14T04:10:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.304293+0000", 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035758550000000002 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.304328+0000", 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5206999999999997e-05 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.529087+0000", 2024-09-14T04:10:11.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.22475943700000001 2024-09-14T04:10:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.529187+0000", 2024-09-14T04:10:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9210999999999996e-05 2024-09-14T04:10:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.049 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.529251+0000", 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3973000000000006e-05 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.45572.0:247627 9.4 9:27c156c5:::benchmark_data_smithi114_153893_object118166:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:07:12.295998+0000", 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 178.707371424, 2024-09-14T04:10:11.050 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23340366800000001, 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.45572", 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 247627 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.295998+0000", 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.295998+0000", 2024-09-14T04:10:11.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.295999+0000", 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3099999999999996e-07 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.296004+0000", 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3950000000000003e-06 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.296017+0000", 2024-09-14T04:10:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2959000000000001e-05 2024-09-14T04:10:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.296024+0000", 2024-09-14T04:10:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2849999999999997e-06 2024-09-14T04:10:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.302608+0000", 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0065841520000000002 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.302707+0000", 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8963999999999996e-05 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.302779+0000", 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2087000000000003e-05 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.304418+0000", 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00163901 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.304450+0000", 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2292999999999997e-05 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.529293+0000", 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.22484220399999999 2024-09-14T04:10:11.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.529367+0000", 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4352999999999998e-05 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:07:12.529402+0000", 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5034999999999999e-05 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.057 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.35997.0:36736 10.2 10:4345f757:::benchmark_data_smithi114_188657_object36735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:08:18.550699+0000", 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.45267104600001, 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.264692027, 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.058 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.35997", 2024-09-14T04:10:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:10:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 36736 2024-09-14T04:10:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.550699+0000", 2024-09-14T04:10:11.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.550699+0000", 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.550701+0000", 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8199999999999999e-06 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.550726+0000", 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5165e-05 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.550739+0000", 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2843999999999999e-05 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.550746+0000", 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5889999999999997e-06 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551045+0000", 2024-09-14T04:10:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000298394 2024-09-14T04:10:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551189+0000", 2024-09-14T04:10:11.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014418699999999999 2024-09-14T04:10:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551335+0000", 2024-09-14T04:10:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014642400000000001 2024-09-14T04:10:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554429+0000", 2024-09-14T04:10:11.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030941219999999999 2024-09-14T04:10:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554725+0000", 2024-09-14T04:10:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00029542799999999998 2024-09-14T04:10:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.815156+0000", 2024-09-14T04:10:11.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26043172799999997 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.815340+0000", 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000183838 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.815391+0000", 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0488000000000002e-05 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.065 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:36739 10.6 e205/199 10:615806e4:::benchmark_data_smithi114_188657_object36738:head v 205'2278, mlcod=205'2274)", 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:08:18.551551+0000", 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.45181832199999, 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26338826500000001, 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551551+0000", 2024-09-14T04:10:11.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551551+0000", 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551552+0000", 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3e-07 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551603+0000", 2024-09-14T04:10:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0859000000000002e-05 2024-09-14T04:10:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551613+0000", 2024-09-14T04:10:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6970000000000005e-06 2024-09-14T04:10:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551619+0000", 2024-09-14T04:10:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1639999999999998e-06 2024-09-14T04:10:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551684+0000", 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4752000000000006e-05 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.551703+0000", 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8834999999999998e-05 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554914+0000", 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003211443 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.814940+0000", 2024-09-14T04:10:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26002578500000001 2024-09-14T04:10:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:36745 10.4 e205/199 10:2f8781e8:::benchmark_data_smithi114_188657_object36744:head v 205'2290, mlcod=205'2288)", 2024-09-14T04:10:11.071 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:08:18.553635+0000", 2024-09-14T04:10:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.44973474699999, 2024-09-14T04:10:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.261318571, 2024-09-14T04:10:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.553635+0000", 2024-09-14T04:10:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.553635+0000", 2024-09-14T04:10:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.553636+0000", 2024-09-14T04:10:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9599999999999998e-07 2024-09-14T04:10:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.553675+0000", 2024-09-14T04:10:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9385000000000003e-05 2024-09-14T04:10:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.553686+0000", 2024-09-14T04:10:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0828e-05 2024-09-14T04:10:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.553693+0000", 2024-09-14T04:10:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5599999999999999e-06 2024-09-14T04:10:11.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.553754+0000", 2024-09-14T04:10:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1577999999999999e-05 2024-09-14T04:10:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.553773+0000", 2024-09-14T04:10:11.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8581e-05 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.814911+0000", 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26113803400000002 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.814954+0000", 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2709000000000001e-05 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.077 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:36744 10.1 e205/199 10:8a888e6e:::benchmark_data_smithi114_188657_object36743:head v 205'2265, mlcod=205'2262)", 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:08:18.554544+0000", 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.448825552, 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26462023400000001, 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554544+0000", 2024-09-14T04:10:11.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554544+0000", 2024-09-14T04:10:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554545+0000", 2024-09-14T04:10:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:11.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:11.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5300000000000003e-07 2024-09-14T04:10:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554580+0000", 2024-09-14T04:10:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4847e-05 2024-09-14T04:10:11.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554590+0000", 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8700000000000004e-06 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554596+0000", 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5479999999999998e-06 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554772+0000", 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017533800000000001 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554789+0000", 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7227e-05 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.819109+0000", 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26432048800000002 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.819164+0000", 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5163e-05 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:36749 10.0 e205/199 10:08747d5e:::benchmark_data_smithi114_188657_object36748:head v 205'2262, mlcod=205'2260)", 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:08:18.554816+0000", 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.44855411899999, 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26209028699999998, 2024-09-14T04:10:11.085 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554816+0000", 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554816+0000", 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554816+0000", 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1299999999999999e-07 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554857+0000", 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0009000000000003e-05 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.554866+0000", 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3549999999999994e-06 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.814888+0000", 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26002224699999998 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.815003+0000", 2024-09-14T04:10:11.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011461 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.815026+0000", 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2955999999999998e-05 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.816825+0000", 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001798919 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.816906+0000", 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1378000000000006e-05 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:36753 10.1 e205/199 10:8b42079f:::benchmark_data_smithi114_188657_object36752:head v 205'2266, mlcod=205'2263)", 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:08:18.556849+0000", 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.446520916, 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26398751399999998, 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.090 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.556849+0000", 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.556849+0000", 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.556850+0000", 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.95e-07 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.556882+0000", 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2292999999999997e-05 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.814897+0000", 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.25801503100000001 2024-09-14T04:10:11.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.814907+0000", 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6269999999999995e-06 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.816315+0000", 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0014085829999999999 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.816334+0000", 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.853e-05 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.820781+0000", 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044467259999999998 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.820836+0000", 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5729000000000001e-05 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.094 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.35997.0:36756 10.2 10:4036931f:::benchmark_data_smithi114_188657_object36755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:08:18.558512+0000", 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 112.444857781, 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.265469487, 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.35997", 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:10:11.095 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 36756 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.558512+0000", 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.558512+0000", 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.558514+0000", 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9829999999999999e-06 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.558540+0000", 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.597e-05 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.814902+0000", 2024-09-14T04:10:11.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.25636164900000002 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.814916+0000", 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4654000000000001e-05 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.815420+0000", 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00050363000000000003 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.815586+0000", 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016626600000000001 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.815727+0000", 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014055299999999999 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.821648+0000", 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059209579999999996 2024-09-14T04:10:11.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.821659+0000", 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1817999999999999e-05 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.823862+0000", 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022027510000000002 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.823922+0000", 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9945000000000003e-05 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:18.823982+0000", 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.931e-05 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.35997.0:68478 10.2 10:43a72527:::benchmark_data_smithi114_188657_object68477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-14T04:10:11.101 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:08:40.863400+0000", 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 90.139970117999994, 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.228695022, 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.35997", 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 68478 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.102 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.863400+0000", 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.863400+0000", 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.863402+0000", 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9860000000000001e-06 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.103 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.863427+0000", 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.497e-05 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.863442+0000", 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5087e-05 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.104 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.863450+0000", 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7329999999999992e-06 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.863501+0000", 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0167999999999998e-05 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.863601+0000", 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000100327 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.105 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.863695+0000", 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3700000000000001e-05 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.865647+0000", 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00195272 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.865675+0000", 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7946e-05 2024-09-14T04:10:11.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:41.091948+0000", 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.22627249199999999 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:41.092013+0000", 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5116999999999993e-05 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:41.092095+0000", 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1775999999999998e-05 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.35997.0:68489 10.2 10:41f5d90a:::benchmark_data_smithi114_188657_object68488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:08:40.938960+0000", 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 90.064409443000002, 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.16290025999999999, 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.108 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.35997", 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 68489 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.938960+0000", 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.938960+0000", 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.938962+0000", 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9080000000000002e-06 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.939013+0000", 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0680000000000003e-05 2024-09-14T04:10:11.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.939031+0000", 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8372999999999998e-05 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.939039+0000", 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.379e-06 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.939199+0000", 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016000500000000001 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.939388+0000", 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018884699999999999 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.939823+0000", 2024-09-14T04:10:11.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00043543999999999999 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.946460+0000", 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0066371299999999998 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:40.949605+0000", 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031449799999999999 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:41.101753+0000", 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.15214805200000001 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:41.101817+0000", 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4037999999999999e-05 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:08:41.101861+0000", 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3427999999999998e-05 2024-09-14T04:10:11.114 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.35997.0:129854 10.2 10:46f6e59a:::benchmark_data_smithi114_188657_object9658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:09:44.325358+0000", 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 26.678011457, 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.056449738999999999, 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.35997", 2024-09-14T04:10:11.115 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 129854 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.325358+0000", 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.325358+0000", 2024-09-14T04:10:11.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.325359+0000", 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7399999999999991e-07 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.325364+0000", 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3290000000000004e-06 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.325378+0000", 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3944999999999999e-05 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.325385+0000", 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.469e-06 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.339990+0000", 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014604506999999999 2024-09-14T04:10:11.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.340099+0000", 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000109867 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.340222+0000", 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012267300000000001 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.352127+0000", 2024-09-14T04:10:11.119 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011905012 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.356308+0000", 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004180408 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.381696+0000", 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.025388906999999999 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.120 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.381758+0000", 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2036e-05 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:44.381808+0000", 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9611999999999999e-05 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.35997.0:134800 10.2 10:480071e0:::benchmark_data_smithi114_188657_object14604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:09:49.915141+0000", 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 21.088228503, 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.061215051999999999, 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.35997", 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 134800 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.915141+0000", 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.915141+0000", 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.915143+0000", 2024-09-14T04:10:11.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0559999999999999e-06 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.915151+0000", 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1640000000000004e-06 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.915168+0000", 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7461000000000001e-05 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.915177+0000", 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6240000000000008e-06 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.918494+0000", 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003317146 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.918624+0000", 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012993600000000001 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.918754+0000", 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013062999999999999 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.934016+0000", 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015262161999999999 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.951669+0000", 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017652099000000001 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.975964+0000", 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024295892999999999 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.976211+0000", 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024619799999999999 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.976356+0000", 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014568299999999999 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.35997.0:134801 10.2 10:465e59ef:::benchmark_data_smithi114_188657_object14605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:09:49.920961+0000", 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 21.082408819000001, 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.055581274999999999, 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.128 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.35997", 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 134801 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.920961+0000", 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.920961+0000", 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.920963+0000", 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3810000000000002e-06 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.920971+0000", 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.593e-06 2024-09-14T04:10:11.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.920994+0000", 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3113e-05 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.921005+0000", 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0964e-05 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.926461+0000", 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054563900000000002 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.926587+0000", 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000125308 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.926729+0000", 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014187 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.951626+0000", 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024897144 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.959058+0000", 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0074327280000000004 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.976452+0000", 2024-09-14T04:10:11.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.01739373 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.976505+0000", 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2577000000000002e-05 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:09:49.976542+0000", 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7477e-05 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.35997.0:152314 10.2 10:4d16f642:::benchmark_data_smithi114_188657_object32118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:10.539984+0000", 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.46338573900000002, 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.056306226000000001, 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.35997", 2024-09-14T04:10:11.135 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 152314 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.539984+0000", 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.539984+0000", 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.539984+0000", 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8099999999999998e-07 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.539986+0000", 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9970000000000001e-06 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.539992+0000", 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0710000000000002e-06 2024-09-14T04:10:11.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.539996+0000", 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2699999999999998e-06 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.546332+0000", 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.006335841 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.546621+0000", 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000289012 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.547156+0000", 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00053492199999999996 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.551973+0000", 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0048173629999999999 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.557127+0000", 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0051544019999999998 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.596191+0000", 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.039063726999999999 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.596247+0000", 2024-09-14T04:10:11.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5841999999999998e-05 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.596290+0000", 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3398e-05 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.35997.0:152408 10.2 10:45c4e451:::benchmark_data_smithi114_188657_object32212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:10.665565+0000", 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.337805297, 2024-09-14T04:10:11.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.050584751999999997, 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.35997", 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 152408 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:10:11.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.665565+0000", 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.665565+0000", 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.665566+0000", 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.393e-06 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.665571+0000", 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4659999999999998e-06 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.665583+0000", 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1165000000000001e-05 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.665590+0000", 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0280000000000003e-06 2024-09-14T04:10:11.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.673482+0000", 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0078924939999999999 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.673747+0000", 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00026495799999999998 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.674093+0000", 2024-09-14T04:10:11.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00034594200000000002 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.687164+0000", 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013070564999999999 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.687235+0000", 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1810999999999994e-05 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.716000+0000", 2024-09-14T04:10:11.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.028764841999999999 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.716100+0000", 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9728999999999993e-05 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:10.716149+0000", 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9359e-05 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:10:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:10:11.148 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:10:11.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:11.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:11.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:11.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:11.336 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:11.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:11.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:11.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:11.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:11.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:11.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:11.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:11.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:11.758 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:11.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:11.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:11.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:10:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:10:11] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-09-14T04:10:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:11.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:11.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:12.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:12.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:12.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:12.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:11 smithi160 ceph-mon[88407]: pgmap v778: 89 pgs: 89 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 832 op/s 2024-09-14T04:10:12.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:12.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:12.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:12.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:12.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:12.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:11 smithi114 ceph-mon[103029]: pgmap v778: 89 pgs: 89 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 832 op/s 2024-09-14T04:10:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:11 smithi114 ceph-mon[106969]: pgmap v778: 89 pgs: 89 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 832 op/s 2024-09-14T04:10:12.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:12.465 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:12.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:12.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:12.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:12.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:12.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:12.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:12.739 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:12.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:12.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:12.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:12.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:12.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:12.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:13.027 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:13.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:13.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:13.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:13.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:13.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:13.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:13.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-14T04:10:13.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:13.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:13.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:13.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:13.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:13.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:13.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:13.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:13.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:13.677 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:13.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:13.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:13.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:13.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:13.854 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:13.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:13.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:13.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:14.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:14.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:14.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:14.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:13 smithi114 ceph-mon[106969]: pgmap v779: 89 pgs: 89 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:10:14.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:13 smithi114 ceph-mon[103029]: pgmap v779: 89 pgs: 89 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:10:14.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:14.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:14.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:14.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:14.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:14.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:14.396 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:10:14.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-14T04:10:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:13 smithi160 ceph-mon[88407]: pgmap v779: 89 pgs: 89 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:10:14.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:14.510 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:14.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:14.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:14.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:14.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:14.709 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:14.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:14.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:14.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:14.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:14.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:14.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:14.938 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:10:15.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:15.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:15.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:15.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:14 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3282619404' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:10:15.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:14 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3282619404' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:10:15.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:15.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:15.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:15.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:15.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:15.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:15.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:15.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:14 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3282619404' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:10:15.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:15.458 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:15.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:15.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:15.602 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:15.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:15.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:15.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:15.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:15.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:15.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:15.859 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:15.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:15.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:15.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:15.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:16.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:16.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:16.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:16.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:15 smithi114 ceph-mon[103029]: pgmap v780: 89 pgs: 89 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 978 op/s 2024-09-14T04:10:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:15 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3282619404' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:10:16.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:15 smithi114 ceph-mon[103029]: osdmap e213: 8 total, 8 up, 7 in 2024-09-14T04:10:16.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:15 smithi114 ceph-mon[106969]: pgmap v780: 89 pgs: 89 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 978 op/s 2024-09-14T04:10:16.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:15 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:16.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3282619404' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:10:16.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:15 smithi114 ceph-mon[106969]: osdmap e213: 8 total, 8 up, 7 in 2024-09-14T04:10:16.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:16.236 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:16.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:16.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:16.384 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:16.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:15 smithi160 ceph-mon[88407]: pgmap v780: 89 pgs: 89 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 978 op/s 2024-09-14T04:10:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:15 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3282619404' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:10:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:15 smithi160 ceph-mon[88407]: osdmap e213: 8 total, 8 up, 7 in 2024-09-14T04:10:16.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:16.526 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:16.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:16.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:16.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:16.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:16.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:16.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:16.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:16.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:16.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:16.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:16.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:17.099 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:17.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:17.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:17.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:17.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:17.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:17.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:17.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:17.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:17.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:17.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:17.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:17.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:17.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:17.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:17.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:17.662 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:17.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:17.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:17.805 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:17.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:17.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:17.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:17.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:17.936 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:17.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:17.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:17 smithi114 ceph-mon[106969]: pgmap v782: 89 pgs: 89 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 846 op/s 2024-09-14T04:10:17.995 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:17 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3851624421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:17.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:17 smithi114 ceph-mon[103029]: pgmap v782: 89 pgs: 89 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 846 op/s 2024-09-14T04:10:17.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:17 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3851624421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:18.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:18.063 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:18.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:18.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:18.207 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:18.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:18.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:10:18.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:10:18.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:10:18.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:10:18.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:10:18.298 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:10:18.298 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:10:18.298 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:10:18.298 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:10:18.298 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:10:18.298 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:10:18.298 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:10:18.298 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:10:18.298 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:10:18.298 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:10:18.299 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:10:18.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:18.338 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:18.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:18.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:17 smithi160 ceph-mon[88407]: pgmap v782: 89 pgs: 89 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 846 op/s 2024-09-14T04:10:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:17 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3851624421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:18.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:18.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:18.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:18.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:18.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:18.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:18.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:18.743 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:18.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:18.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:18.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:18.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:18.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:10:18.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:19.007 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:19.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:19.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:19.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:19.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:19.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:19.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:19.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:19.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:19.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:19.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:19.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:19.549 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:19.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:19.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:19.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:19.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:19.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:19.824 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:19.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:19.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:19.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:19.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:20.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:20.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:20.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:20.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:19 smithi114 ceph-mon[106969]: pgmap v783: 89 pgs: 89 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 843 op/s 2024-09-14T04:10:20.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:19 smithi114 ceph-mon[103029]: pgmap v783: 89 pgs: 89 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 843 op/s 2024-09-14T04:10:20.192 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:20.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:19 smithi160 ceph-mon[88407]: pgmap v783: 89 pgs: 89 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 843 op/s 2024-09-14T04:10:20.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:20.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:20.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:20.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:20.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:20.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:20.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:20.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:20.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:20.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:20.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:20.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:20.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:10:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:10:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:10:20.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:20.748 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:20.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:20.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:20.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:20.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:21.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:21.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:21.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:21.171 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:21.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:21.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:10:21.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:21.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:21.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:21.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:21.472 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:21.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:21.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:21.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:21.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:21.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:21.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:10:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:10:21] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-09-14T04:10:21.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:21.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:21.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:21.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:21.856 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:21.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:21.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:21.979 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:21.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:22.055 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:22.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:22.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:22.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:22.223 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:21 smithi114 ceph-mon[103029]: pgmap v784: 89 pgs: 89 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 836 op/s 2024-09-14T04:10:22.223 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:21 smithi114 ceph-mon[106969]: pgmap v784: 89 pgs: 89 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 836 op/s 2024-09-14T04:10:22.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:22.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:22.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:22.374 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:22.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:22.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:22.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:21 smithi160 ceph-mon[88407]: pgmap v784: 89 pgs: 89 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 836 op/s 2024-09-14T04:10:22.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:22.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:22.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:22.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:22.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:22.679 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:22.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:22.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:22.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:22.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:22.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:22.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:22.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:23.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:23.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:23.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:23.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:23.210 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:23.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:23.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:23.336 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:23.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:23.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:23.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:23.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:23.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:23.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:23.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:23.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:23.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:23.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:23.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:23.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:23.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:23.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:23.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:23.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:23.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:23.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:24.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:24.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:24.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:24.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:24.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:24.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:24.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:24.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:23 smithi114 ceph-mon[103029]: pgmap v785: 89 pgs: 89 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 834 op/s 2024-09-14T04:10:24.280 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:23 smithi114 ceph-mon[106969]: pgmap v785: 89 pgs: 89 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 834 op/s 2024-09-14T04:10:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:24.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:24.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:24.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:24.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:23 smithi160 ceph-mon[88407]: pgmap v785: 89 pgs: 89 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 834 op/s 2024-09-14T04:10:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:24.426 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:24.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:24.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:24.569 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:24.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:24.671 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:24.696 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:24.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:24.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:24.836 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:24.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:24.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:24.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:24.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:24.992 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:10:24.992 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:10:25.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:25.112 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:25.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:25.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:24 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2803618858' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:10:25.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:24 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2803618858' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:10:25.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:25.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:25.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:25.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:25.386 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:25.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:25.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:25.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:24 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2803618858' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:10:25.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:25.516 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:25.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:25.598 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-14T04:10:25.618 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:25.650 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:25.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:25.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:25.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:25.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:25.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:25.933 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:25.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:26.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:26.063 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:26.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:26.165 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:26.193 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:26.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:26.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:26.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:26.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:26.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:26.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:25 smithi114 ceph-mon[103029]: pgmap v786: 89 pgs: 89 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 830 op/s 2024-09-14T04:10:26.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2803618858' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:10:26.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:25 smithi114 ceph-mon[103029]: osdmap e214: 8 total, 8 up, 7 in 2024-09-14T04:10:26.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:25 smithi114 ceph-mon[106969]: pgmap v786: 89 pgs: 89 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 830 op/s 2024-09-14T04:10:26.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2803618858' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:10:26.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:25 smithi114 ceph-mon[106969]: osdmap e214: 8 total, 8 up, 7 in 2024-09-14T04:10:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:25 smithi160 ceph-mon[88407]: pgmap v786: 89 pgs: 89 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 830 op/s 2024-09-14T04:10:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2803618858' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:10:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:25 smithi160 ceph-mon[88407]: osdmap e214: 8 total, 8 up, 7 in 2024-09-14T04:10:26.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:26.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:26.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:26.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-14T04:10:26.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:26.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:26.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:26.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:26.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:26.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:26.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:26.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:26.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:27.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:27.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:27.065 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:27.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:27.166 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:27.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-14T04:10:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:27.218 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:27.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:27.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:27.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:27.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:27.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:27.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:27.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:27.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:27.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:27.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:27.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:27.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:27.782 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:27.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:27.854 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:27.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:27.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:27.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:27.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:28.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:28.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:28.111 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:27 smithi114 ceph-mon[106969]: pgmap v788: 89 pgs: 89 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-14T04:10:28.112 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:27 smithi114 ceph-mon[103029]: pgmap v788: 89 pgs: 89 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-14T04:10:28.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:28.203 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:28.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:28.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:28.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:28.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:27 smithi160 ceph-mon[88407]: pgmap v788: 89 pgs: 89 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-14T04:10:28.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:28.476 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:28.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:28.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:28.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:28.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:28.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:28.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:28.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:28.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:28.837 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:28.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:28.873 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:28.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:28.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:28.986 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:10:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:29.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:29.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:29.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:29.161 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:29.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:29.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:29.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:29.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:29.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:29.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:28 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1118464770' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:10:29.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:28 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:10:29.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:28 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1118464770' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:10:29.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:28 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:10:29.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:29.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:28 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1118464770' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:10:29.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:28 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:10:29.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:29.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:29.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:29.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:29.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:29.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:29.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:29.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:29.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:29.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:29.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:29.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:29.981 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:29.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:30.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:30.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:30.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:30.210 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:30.244 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:30.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:30.308 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:10:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:10:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:10:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:29 smithi160 ceph-mon[88407]: pgmap v789: 89 pgs: 89 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 820 op/s 2024-09-14T04:10:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:29 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:29 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:10:30.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:29 smithi160 ceph-mon[88407]: osdmap e215: 8 total, 8 up, 7 in 2024-09-14T04:10:30.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:30.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:29 smithi114 ceph-mon[103029]: pgmap v789: 89 pgs: 89 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 820 op/s 2024-09-14T04:10:30.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:29 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:30.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:29 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:10:30.375 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:29 smithi114 ceph-mon[103029]: osdmap e215: 8 total, 8 up, 7 in 2024-09-14T04:10:30.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:29 smithi114 ceph-mon[106969]: pgmap v789: 89 pgs: 89 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 820 op/s 2024-09-14T04:10:30.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:29 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:30.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:29 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:10:30.375 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:29 smithi114 ceph-mon[106969]: osdmap e215: 8 total, 8 up, 7 in 2024-09-14T04:10:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:30.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:30.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:30.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:30.511 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:30.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:30.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:30.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:30.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:30.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:30.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:30.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:30.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:30.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:30.909 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:30.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:31.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:31.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:31.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:31.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:31.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:31.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:31.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:31.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:31.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:31.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:31.435 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:10:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:10:31] "GET /metrics HTTP/1.1" 200 36486 "" "Prometheus/2.51.0" 2024-09-14T04:10:31.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:31.453 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:31.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:31.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:31.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:31.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:31.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:31.712 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:31.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:31.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:31.814 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:31.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:31.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:31.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:31.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:31.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:32.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:32.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:32.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:32.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:32.254 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:32.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:32.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:32.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:31 smithi114 ceph-mon[103029]: pgmap v791: 89 pgs: 89 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 923 op/s 2024-09-14T04:10:32.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:31 smithi114 ceph-mon[106969]: pgmap v791: 89 pgs: 89 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 923 op/s 2024-09-14T04:10:32.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:32.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:32.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:32.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:32.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:31 smithi160 ceph-mon[88407]: pgmap v791: 89 pgs: 89 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 923 op/s 2024-09-14T04:10:32.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:32.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:32.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:32.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:32.656 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:32.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:32.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:32.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:32.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:32.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:32.886 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:32.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:32.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:33.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:33.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:33.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:33.141 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:33.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:33.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:33.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:33.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:33.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:33.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:33.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:33.446 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:33.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:33.547 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:33.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:33.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:33.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:33.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:33.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:33.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:33.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:33.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:33.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:33.901 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:10:33.901 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:10:33.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:33.964 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:10:33.964 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:10:33.999 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:10:34.000 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:10:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:34.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:34.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:34.038 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:10:34.038 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:10:34.077 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:10:34.077 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:10:34.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:34.112 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:10:34.113 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:10:34.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:34.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:34.158 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:10:34.158 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:10:34.159 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:33 smithi114 ceph-mon[106969]: pgmap v792: 89 pgs: 89 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 595 op/s 2024-09-14T04:10:34.159 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:33 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/14026005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:34.159 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:33 smithi114 ceph-mon[103029]: pgmap v792: 89 pgs: 89 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 595 op/s 2024-09-14T04:10:34.159 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:33 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/14026005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:34.200 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:10:34.200 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:10:34.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:10:34.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:34.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:34.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:33 smithi160 ceph-mon[88407]: pgmap v792: 89 pgs: 89 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 595 op/s 2024-09-14T04:10:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:33 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/14026005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:34.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:34.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:34.467 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:34.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:34.568 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:34.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:34.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:34.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:34.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:34.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:34.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:34.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:34.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:34.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:34.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:34.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:34.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:35.013 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-14T04:10:35.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:35.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:35.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:35.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:35.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:35.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:35.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:35.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:34 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1412831931' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:10:35.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:34 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:10:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:34 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1412831931' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:10:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:34 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:10:35.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:35.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:35.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:35.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:34 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1412831931' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:10:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:34 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:10:35.498 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:35.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:35.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:35.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:35.656 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:35.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:35.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:35.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:35.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:35.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:35.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:35.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:35.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:35.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:36.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:36.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:36.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:36.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:36.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:36.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:36.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:36.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:36.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:36.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:36 smithi114 ceph-mon[106969]: pgmap v793: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-14T04:10:36.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:36 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:10:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:36 smithi114 ceph-mon[106969]: osdmap e216: 8 total, 8 up, 7 in 2024-09-14T04:10:36.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:36 smithi114 ceph-mon[103029]: pgmap v793: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-14T04:10:36.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:36 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:10:36.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:36 smithi114 ceph-mon[103029]: osdmap e216: 8 total, 8 up, 7 in 2024-09-14T04:10:36.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:36.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:36 smithi160 ceph-mon[88407]: pgmap v793: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-14T04:10:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:36 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:10:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:36 smithi160 ceph-mon[88407]: osdmap e216: 8 total, 8 up, 7 in 2024-09-14T04:10:36.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:36.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:36.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:36.557 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:36.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:36.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:36.690 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:36.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:36.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:36.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:36.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:36.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:36.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:36.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:37.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:37.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:37.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:37.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:37.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:37.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:37.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:37.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:37.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:37.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:37.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:37.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:37.502 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:37.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:37.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:37.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:37.638 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:37.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:37.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:37.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:37.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:37.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:37.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:37.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:38.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:38.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:38.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-14T04:10:38.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:38.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:38.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:38.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:38 smithi114 ceph-mon[103029]: pgmap v795: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 915 op/s 2024-09-14T04:10:38.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:38.323 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:38 smithi114 ceph-mon[106969]: pgmap v795: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 915 op/s 2024-09-14T04:10:38.323 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:38.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:38.390 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:10:38.390 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:10:38.390 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:10:38.390 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:10:38.390 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:10:38.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:38.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:38.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:38 smithi160 ceph-mon[88407]: pgmap v795: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 915 op/s 2024-09-14T04:10:38.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:38.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:38.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:38.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:38.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:38.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:38.672 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:38.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:38.773 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:38.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:38.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:38.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:38.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:38.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:39.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:39.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:39.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:39.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:10:39.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:39.262 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:39.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:39.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:39.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:39.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:39.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:39.522 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:39.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:39.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-14T04:10:39.622 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:39.648 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:39.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:39.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:39.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:39.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:39.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:39.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:39.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:40.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:40.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:40.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:40.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:40.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:40 smithi114 ceph-mon[103029]: pgmap v796: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 633 op/s 2024-09-14T04:10:40.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:40 smithi114 ceph-mon[106969]: pgmap v796: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 633 op/s 2024-09-14T04:10:40.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:40.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:40.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:40.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:40.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:40.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:40 smithi160 ceph-mon[88407]: pgmap v796: 89 pgs: 89 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 633 op/s 2024-09-14T04:10:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:10:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:10:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:10:40.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:40.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:40.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:40.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:40.686 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:40.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:40.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:40.814 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:40.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:40.822 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:40.827 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:40.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:40.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:40.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:41.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:41.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:41.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:41.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:41.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:41.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:41.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:41.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:41.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:41.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:41.496 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:41.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:41.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:41.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:41.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:10:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:10:41] "GET /metrics HTTP/1.1" 200 36484 "" "Prometheus/2.51.0" 2024-09-14T04:10:41.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:41.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:41.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:41.820 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:41.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:41.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:41.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:41.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:42.035 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:10:42.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:42.092 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:42.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:42.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:42.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:42.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:42.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:42.360 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:42.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:42.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:42 smithi114 ceph-mon[106969]: pgmap v797: 89 pgs: 89 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 782 op/s 2024-09-14T04:10:42.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3112666633' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:10:42.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:42 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:10:42.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:42 smithi114 ceph-mon[103029]: pgmap v797: 89 pgs: 89 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 782 op/s 2024-09-14T04:10:42.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3112666633' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:10:42.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:42 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:10:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:42 smithi160 ceph-mon[88407]: pgmap v797: 89 pgs: 89 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 782 op/s 2024-09-14T04:10:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3112666633' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:10:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:42 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:10:42.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:42.486 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:42.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:42.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:42.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:42.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:42.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:42.751 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:42.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:42.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:42.895 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:42.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:42.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:43.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:43.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:43.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:43.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:43.196 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:43.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:43.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:43.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:43.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:43.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:43.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:43.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:43 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:10:43.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:43 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:10:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:43 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:10:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:43 smithi114 ceph-mon[103029]: osdmap e217: 8 total, 8 up, 7 in 2024-09-14T04:10:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:43 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:10:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:43 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:10:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:43 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:10:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:43 smithi114 ceph-mon[106969]: osdmap e217: 8 total, 8 up, 7 in 2024-09-14T04:10:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:43 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:10:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:43 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:10:43.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:43 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:10:43.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:43 smithi160 ceph-mon[88407]: osdmap e217: 8 total, 8 up, 7 in 2024-09-14T04:10:43.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:43.456 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:43.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:43.558 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:43.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:43.585 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:43.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:43.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:43.726 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:43.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:43.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:43.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:43.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:43.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:43.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:43.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:43.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:43.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:44.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:44.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:44.139 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:44.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:44.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:44.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:44.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:44 smithi114 ceph-mon[103029]: pgmap v799: 89 pgs: 89 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 565 op/s 2024-09-14T04:10:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:44 smithi114 ceph-mon[106969]: pgmap v799: 89 pgs: 89 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 565 op/s 2024-09-14T04:10:44.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:44 smithi160 ceph-mon[88407]: pgmap v799: 89 pgs: 89 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 565 op/s 2024-09-14T04:10:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:44.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:44.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:44.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:44.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:44.554 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:44.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:44.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:44.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:44.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:44.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:44.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:44.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:44.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:44.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:44.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:44.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:44.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:45.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:45.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:45.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:45.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:45.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:45.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:45.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:45.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:45.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:45.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:45.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:45.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:45.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:45.625 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:45.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:45.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:45.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:45.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:45.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:45.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:45.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:45.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:46.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:46.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:46.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:46.076 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-14T04:10:46.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:46.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:46.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:46.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:46.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:46.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:46.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:46.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:46 smithi114 ceph-mon[106969]: pgmap v800: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 792 op/s 2024-09-14T04:10:46.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3090734866' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:10:46.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:46 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:10:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:46 smithi114 ceph-mon[103029]: pgmap v800: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 792 op/s 2024-09-14T04:10:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3090734866' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:10:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:46 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:10:46.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:46 smithi160 ceph-mon[88407]: pgmap v800: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 792 op/s 2024-09-14T04:10:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3090734866' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:10:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:46 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:10:46.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:46.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:46.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:46.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:46.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:46.536 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:46.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:46.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:10:46.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:46.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:46.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:46.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:46.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:46.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:46.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:46.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:46.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:46.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:47.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:47.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:47.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:47.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:47.226 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:47.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:47.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:47.355 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:47 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:10:47.355 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:47 smithi114 ceph-mon[106969]: osdmap e218: 8 total, 8 up, 7 in 2024-09-14T04:10:47.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:47 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:10:47.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:47 smithi114 ceph-mon[103029]: osdmap e218: 8 total, 8 up, 7 in 2024-09-14T04:10:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:47.365 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:47.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:47 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:10:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:47 smithi160 ceph-mon[88407]: osdmap e218: 8 total, 8 up, 7 in 2024-09-14T04:10:47.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:47.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:47.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:47.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:47.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:47.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:47.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:47.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:47.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:47.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:47.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:48.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-14T04:10:48.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:48.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:48.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:48.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:48.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:48.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:48.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:48.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:48.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:48 smithi114 ceph-mon[103029]: pgmap v802: 89 pgs: 89 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 884 op/s 2024-09-14T04:10:48.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:48 smithi114 ceph-mon[106969]: pgmap v802: 89 pgs: 89 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 884 op/s 2024-09-14T04:10:48.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:48 smithi160 ceph-mon[88407]: pgmap v802: 89 pgs: 89 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 884 op/s 2024-09-14T04:10:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:48.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:48.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:48.573 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:48.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:48.676 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:48.714 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:10:48.714 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:10:48.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:48.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:48.749 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:10:48.749 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:10:48.788 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:10:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:10:48.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:48.825 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:10:48.825 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:10:48.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:48.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:48.858 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:10:48.858 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:10:48.892 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:10:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:10:48.919 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:10:48.920 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:10:48.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:48.955 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:10:48.955 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:10:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:48.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:48.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:49.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:49.117 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:49.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:49.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:49.265 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:49.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:49.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:49.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:49 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2452128452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:49.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:49 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2452128452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:49.401 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:49.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:49.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:49 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2452128452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:10:49.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:49.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:49.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:49.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:49.680 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:49.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:49.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:49.806 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:49.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:49.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:49.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:49.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:49.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:50.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:50.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:50.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:50.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:50.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:50.234 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:50.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:50.258 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:50 smithi114 ceph-mon[106969]: pgmap v803: 89 pgs: 89 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 660 op/s 2024-09-14T04:10:50.259 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:50 smithi114 ceph-mon[103029]: pgmap v803: 89 pgs: 89 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 660 op/s 2024-09-14T04:10:50.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:50.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:50.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:50 smithi160 ceph-mon[88407]: pgmap v803: 89 pgs: 89 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 660 op/s 2024-09-14T04:10:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:10:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:10:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:10:50.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:50.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:50.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:50.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:50.614 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:10:50.614 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:10:50.615 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:50.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:50.651 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:50.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:50.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:50.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:50.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:50.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:50.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:50.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:50.917 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:50.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:51.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:51.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:51.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:51.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:51.194 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:51.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:51.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:51.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:51.325 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:51.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:51.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:51.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:51.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:51.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:51.576 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:51.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:51.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:10:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:10:51] "GET /metrics HTTP/1.1" 200 36484 "" "Prometheus/2.51.0" 2024-09-14T04:10:51.678 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:51.703 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:51.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:51.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:51.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-14T04:10:51.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:51.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:51.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:51.979 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:51.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:52.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:10:52.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:52.171 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:52.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:52.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:52.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:52.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:52.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:52 smithi114 ceph-mon[103029]: pgmap v804: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-14T04:10:52.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:52 smithi114 ceph-mon[106969]: pgmap v804: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-14T04:10:52.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:52 smithi160 ceph-mon[88407]: pgmap v804: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-14T04:10:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:52.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:52.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:52.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:52.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:52.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:52.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:52.734 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:52.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:52.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:52.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:52.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:52.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:52.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:52.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:52.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:53.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:53.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:53.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:53.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:53.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:53.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:53.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:53.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:53.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:53.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:53.565 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:53.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:53.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:53.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:53.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:53.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:53.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:53.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:10:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:53 smithi160 ceph-mon[88407]: pgmap v805: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-14T04:10:53.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:53.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:53.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:54.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:10:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:54.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:53 smithi114 ceph-mon[103029]: pgmap v805: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-14T04:10:54.056 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:10:54.056 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:10:54.056 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:53 smithi114 ceph-mon[106969]: pgmap v805: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-14T04:10:54.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:54.138 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:54.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:54.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:54.288 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:54.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:54.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:54.397 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:54.451 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:54.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:54.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:54.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:54.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:54.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:54.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:54.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:54.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:54.867 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:54.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:54.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:10:54.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:10:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:10:54.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:54.992 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:54.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:54.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:55.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:55.121 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:55.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:55.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:55.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:55.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:55.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:55.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:55.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:55.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:55.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:55.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:55.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:55.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:55.658 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:55.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:55.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:55.799 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:55.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:55.818 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:55 smithi160 ceph-mon[88407]: pgmap v806: 89 pgs: 89 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 823 op/s 2024-09-14T04:10:55.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:55 smithi114 ceph-mon[106969]: pgmap v806: 89 pgs: 89 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 823 op/s 2024-09-14T04:10:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:55 smithi114 ceph-mon[103029]: pgmap v806: 89 pgs: 89 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 823 op/s 2024-09-14T04:10:55.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:55.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:55.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:56.009 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:10:56.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:56.084 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:56.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:56.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:56.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:56.220 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:56.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:56.321 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:56.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:56.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:56.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:56.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:56.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:56.593 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:56.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:56.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:56.657 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:10:56.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:56.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:56.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:56.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:56 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1926324021' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:10:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:56.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:56.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:57.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:56 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1926324021' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:10:57.002 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:56 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1926324021' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:10:57.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:10:57.042 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:57.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:10:57.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:57.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:57.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:57.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:57.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:57.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:57.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:57.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:57.470 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:57.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:57.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:10:57.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:57.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:10:57.687 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-14T04:10:57.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:57.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:57.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:57.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:57.894 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:57.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:57.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:57 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:57.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:57 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1926324021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:10:57.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:57 smithi160 ceph-mon[88407]: osdmap e219: 8 total, 8 up, 7 in 2024-09-14T04:10:57.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:57 smithi160 ceph-mon[88407]: pgmap v808: 89 pgs: 89 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-14T04:10:57.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:57 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1501996876' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:10:57.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:57 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:10:57.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:58.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:58.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:58.043 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:58.043 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1926324021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:10:58.043 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[106969]: osdmap e219: 8 total, 8 up, 7 in 2024-09-14T04:10:58.043 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[106969]: pgmap v808: 89 pgs: 89 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-14T04:10:58.043 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1501996876' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:10:58.043 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:10:58.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:10:58.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1926324021' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:10:58.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[103029]: osdmap e219: 8 total, 8 up, 7 in 2024-09-14T04:10:58.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[103029]: pgmap v808: 89 pgs: 89 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-14T04:10:58.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1501996876' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:10:58.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:57 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:10:58.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:10:58.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:58.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:58.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:10:58.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:58.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:58.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:58.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:58.437 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:58.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:58.540 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:58.563 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:58.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:58.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:58.690 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:58.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:58.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:58.825 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:58.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:58.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:58.937 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:58 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:10:58.937 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:58 smithi114 ceph-mon[106969]: osdmap e220: 8 total, 8 up, 7 in 2024-09-14T04:10:58.937 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:58 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:10:58.938 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:58 smithi114 ceph-mon[103029]: osdmap e220: 8 total, 8 up, 7 in 2024-09-14T04:10:58.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:58.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:59.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:10:59.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:59.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:10:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:58 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:10:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:58 smithi160 ceph-mon[88407]: osdmap e220: 8 total, 8 up, 7 in 2024-09-14T04:10:59.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:59.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:59.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:59.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:10:59.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:10:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:59.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:10:59.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:10:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:59.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:59.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:10:59.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:10:59.631 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:59.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:10:59.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:59.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:59.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:10:59.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:10:59.910 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:10:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:10:59.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:00.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:00.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:00.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:00.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:10:59 smithi160 ceph-mon[88407]: pgmap v810: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 651 op/s 2024-09-14T04:11:00.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:11:00.117 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:10:59 smithi114 ceph-mon[106969]: pgmap v810: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 651 op/s 2024-09-14T04:11:00.117 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:10:59 smithi114 ceph-mon[103029]: pgmap v810: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 651 op/s 2024-09-14T04:11:00.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:00.228 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:00.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:00.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:00.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:00.347 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:11:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:11:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:11:00.359 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:00.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:00.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:00.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:00.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:00.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:00.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:00.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:00.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:00.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:00.743 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:00.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:00.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:00.870 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:00.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:00.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:01.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:01.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:01.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:01.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:01.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:01.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:01.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:01.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:01.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:01.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:01.439 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:01.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:01.541 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:01.566 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:01.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:01.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:01.690 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:01.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:01.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:01.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:01.837 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:01.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:01.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:01.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:11:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:11:01] "GET /metrics HTTP/1.1" 200 36487 "" "Prometheus/2.51.0" 2024-09-14T04:11:01.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:01.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:01.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:02.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:02.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:02.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:02.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:02.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-14T04:11:02.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:01 smithi160 ceph-mon[88407]: pgmap v811: 89 pgs: 89 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-09-14T04:11:02.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:02.220 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:02.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:02.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:02.226 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:01 smithi114 ceph-mon[103029]: pgmap v811: 89 pgs: 89 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-09-14T04:11:02.226 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:01 smithi114 ceph-mon[106969]: pgmap v811: 89 pgs: 89 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 968 op/s 2024-09-14T04:11:02.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:02.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:02.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:02.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:02.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:02.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:02.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:02.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:02.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:02.633 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:02.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:02.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:02.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:02.770 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:02.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:02.869 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:11:02.869 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:11:02.872 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:02.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:02.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:03.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:03.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:03.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:03.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:03.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:03.176 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:03.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:03.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:03.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:03.308 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:03.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:03.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:03.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:03.453 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:03.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:03.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:03.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:03.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:03.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:03.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:03.572 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:03.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-14T04:11:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:03.611 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:03.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:03.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:03.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:03.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:03.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:03.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:04.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:04.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:04.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:04.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:04.242 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:04.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:04.344 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:04.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:04.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:04 smithi114 ceph-mon[103029]: pgmap v812: 89 pgs: 89 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 586 op/s 2024-09-14T04:11:04.419 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:04 smithi114 ceph-mon[106969]: pgmap v812: 89 pgs: 89 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 586 op/s 2024-09-14T04:11:04.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:04.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:04.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:04.644 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:04.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:04.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:04.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:04 smithi160 ceph-mon[88407]: pgmap v812: 89 pgs: 89 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 586 op/s 2024-09-14T04:11:04.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:04.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:04.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:04.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:04.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:04.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:04.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:05.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:05.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:05.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:05.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:05.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:05.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:05.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:05.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:05.434 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:11:05.435 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:11:05.435 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:11:05.435 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:11:05.435 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:11:05.435 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:11:05.435 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:11:05.435 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:11:05.435 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:11:05.435 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:11:05.436 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:11:05.436 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:11:05.436 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:11:05.436 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:11:05.436 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:11:05.436 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:11:05.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:05.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:05.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:05.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:05.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:05 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1878702402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:05.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:05 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1878702402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:05.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:05.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:05 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1878702402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:05.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:05.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:05.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:05.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:05.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:05.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:06.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:06.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:06.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:06.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:06.179 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:06.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:06.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:06.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:06.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:06.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:06.465 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:06.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:06.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:06.596 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:06.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:06.651 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:06 smithi114 ceph-mon[103029]: pgmap v813: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 913 op/s 2024-09-14T04:11:06.651 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:06 smithi114 ceph-mon[106969]: pgmap v813: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 913 op/s 2024-09-14T04:11:06.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:06 smithi160 ceph-mon[88407]: pgmap v813: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 913 op/s 2024-09-14T04:11:06.698 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:06.742 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:06.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:06.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:06.867 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:06.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:06.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:06.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:06.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:06.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:07.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:07.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:07.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:07.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:07.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:07.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:07.336 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:07.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:07.385 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:07.397 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:11:07.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:07.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:07.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:07.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:07.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:07.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:07.649 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:07.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:07.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:07 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1170960876' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:07.661 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:07 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1170960876' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:07.661 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:07 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1170960876' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:07.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:07.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:07.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:07.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:07.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:07.949 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:07.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:08.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:08.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:08.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:08.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:11:08.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:08.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:08.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:08.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:08.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:08.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:08.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:08.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:08.504 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:08.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:08.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:08.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[106969]: pgmap v814: 89 pgs: 89 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-14T04:11:08.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:11:08.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1170960876' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:11:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[106969]: osdmap e221: 8 total, 8 up, 7 in 2024-09-14T04:11:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[103029]: pgmap v814: 89 pgs: 89 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-14T04:11:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:11:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1170960876' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:11:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[103029]: osdmap e221: 8 total, 8 up, 7 in 2024-09-14T04:11:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:08.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:08.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:08.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:08 smithi160 ceph-mon[88407]: pgmap v814: 89 pgs: 89 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-14T04:11:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:08 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:11:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1170960876' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:11:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:08 smithi160 ceph-mon[88407]: osdmap e221: 8 total, 8 up, 7 in 2024-09-14T04:11:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:08.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:08.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:08.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:08.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:08.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:08.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:09.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:09.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:09.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:09.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:09.207 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:09.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:09.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:09.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:09.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:09.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:09.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:09.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:09.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:09.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:09.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:09.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:09.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:09.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:09.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:09.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:09.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:09.886 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:09.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:09.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:10.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:10.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:10.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:10.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:10.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:10.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:10.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:10.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:10.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:10.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:10.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:10.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:10.454 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:10 smithi114 ceph-mon[103029]: pgmap v816: 89 pgs: 89 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-14T04:11:10.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:10 smithi114 ceph-mon[106969]: pgmap v816: 89 pgs: 89 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-14T04:11:10.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:10.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:10.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:10.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:10 smithi160 ceph-mon[88407]: pgmap v816: 89 pgs: 89 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-14T04:11:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:11:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:11:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:11:10.689 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:10.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:10.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:10.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:10.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:10.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:10.881 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:10.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:10.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:10.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:11.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:11.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:11:11.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:11.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:11.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:11.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:11.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:11.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:11.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:11.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:11.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:11.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:11.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:11.651 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:11.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:11.753 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:11.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:11.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:11.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:11.787 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:11:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:11:11] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-09-14T04:11:11.877 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:11.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:11.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:12.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:12.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:12.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:12.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:12.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:12.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:12.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:12.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:12.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:12.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:12.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:12.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:12.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:12.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:12.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:12.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:12 smithi114 ceph-mon[103029]: pgmap v817: 89 pgs: 89 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:11:12.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:12 smithi114 ceph-mon[106969]: pgmap v817: 89 pgs: 89 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:11:12.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:12.727 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:12.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:12.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:12.857 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:12.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:12.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:12 smithi160 ceph-mon[88407]: pgmap v817: 89 pgs: 89 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:11:12.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:12.997 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:11:12.997 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:11:12.997 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:11:12.997 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:11:12.998 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:11:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:13.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:13.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:13.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:13.131 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:13.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:13.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:13.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:13.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:13.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:13.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:13.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:13.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:13.516 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:13.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:13.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:13 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/283243247' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:11:13.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:13 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:11:13.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:13 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/283243247' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:11:13.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:13 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:11:13.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:13.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:13.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:13.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:13.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:13.830 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-14T04:11:13.854 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:13 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/283243247' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:11:13.854 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:13 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:11:13.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:13.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:13.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:14.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:14.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:14.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:14.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-14T04:11:14.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:14.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:14.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:14.303 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:14.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:14.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:14.431 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:14.488 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:14.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:14.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:14.612 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:14.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:14.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:14 smithi114 ceph-mon[106969]: pgmap v818: 89 pgs: 89 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 762 op/s 2024-09-14T04:11:14.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:14 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:11:14.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:14 smithi114 ceph-mon[106969]: osdmap e222: 8 total, 8 up, 7 in 2024-09-14T04:11:14.689 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:14 smithi114 ceph-mon[103029]: pgmap v818: 89 pgs: 89 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 762 op/s 2024-09-14T04:11:14.690 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:14 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:11:14.690 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:14 smithi114 ceph-mon[103029]: osdmap e222: 8 total, 8 up, 7 in 2024-09-14T04:11:14.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:14.758 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:14.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:14.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:14.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:14.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:14 smithi160 ceph-mon[88407]: pgmap v818: 89 pgs: 89 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 762 op/s 2024-09-14T04:11:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:14 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:11:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:14 smithi160 ceph-mon[88407]: osdmap e222: 8 total, 8 up, 7 in 2024-09-14T04:11:14.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-14T04:11:14.986 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:15.012 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:15.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:15.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:15.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:15.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:15.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:15.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:15.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:15.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:15.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:15.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:15.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:15.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:15.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:15.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:15.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:15.677 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:15.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:15.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:15.803 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:15.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:15.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:15.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:15.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:15.948 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:11:15.948 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:11:15.948 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:11:15.948 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:11:15.949 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:11:15.949 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:11:15.982 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:11:15.983 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:11:16.013 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:11:16.013 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:11:16.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:16.053 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:11:16.053 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:11:16.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:16.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:16.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:16.083 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:11:16.084 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:11:16.118 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:11:16.118 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:11:16.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:16.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:16.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:16.349 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:16.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:16.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:16.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:16.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:16.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:16.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:16 smithi114 ceph-mon[103029]: pgmap v820: 89 pgs: 89 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 950 op/s 2024-09-14T04:11:16.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:16.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:16.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:16.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:16.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:16.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:16.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:16.865 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:16 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/906750218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:16.866 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:16 smithi114 ceph-mon[106969]: pgmap v820: 89 pgs: 89 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 950 op/s 2024-09-14T04:11:16.866 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:16 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/906750218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:16.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:16.902 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:16.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:16 smithi160 ceph-mon[88407]: pgmap v820: 89 pgs: 89 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 950 op/s 2024-09-14T04:11:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:16 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/906750218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:16.975 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:17.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:17.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:17.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:17.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:17.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:17.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:17.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:17.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:17.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:17.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:17.431 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:17.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:17.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:17.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:17.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:17.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:17.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:17.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:17.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:17.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:17.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:17.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:17.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:17.949 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:17.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:18.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:18.077 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:18.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:18.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:18.208 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:18.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:18.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:18.401 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:18.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:18.503 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:18.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:18.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:18.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:18.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:18.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:18.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:18.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:18.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:18.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:18.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:18 smithi114 ceph-mon[106969]: pgmap v821: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 863 op/s 2024-09-14T04:11:18.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:18 smithi114 ceph-mon[103029]: pgmap v821: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 863 op/s 2024-09-14T04:11:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:18 smithi160 ceph-mon[88407]: pgmap v821: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 863 op/s 2024-09-14T04:11:18.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:11:18.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:18.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:18.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:19.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:19.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:19.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:19.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:19.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:19.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:19.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:19.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:19.520 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:19.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:19.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:19.642 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:19.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:19.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:19.771 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:19.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:19.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:19.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:19.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:20.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:20.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:20.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:20.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:20.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:20.169 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:20.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:20.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:20.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:20.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:20.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:20.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:20.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:20.501 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:11:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:11:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:11:20.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:20.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:20.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:20.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:20.703 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:20.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:20.803 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:20 smithi114 ceph-mon[106969]: pgmap v822: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 812 op/s 2024-09-14T04:11:20.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:20 smithi114 ceph-mon[103029]: pgmap v822: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 812 op/s 2024-09-14T04:11:20.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:20.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:20.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:20.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:20 smithi160 ceph-mon[88407]: pgmap v822: 89 pgs: 89 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 812 op/s 2024-09-14T04:11:20.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:20.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:21.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:21.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:21.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:21.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:21.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:21.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:21.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:21.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:21.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:21.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:21.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:21.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:21.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:21.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:21.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:21.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:21.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:21.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:21.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:21.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:21.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:21 smithi114 ceph-mon[106969]: pgmap v823: 89 pgs: 89 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-14T04:11:21.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:21 smithi114 ceph-mon[103029]: pgmap v823: 89 pgs: 89 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-14T04:11:21.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:11:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:11:21] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-09-14T04:11:21.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:21.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:21 smithi160 ceph-mon[88407]: pgmap v823: 89 pgs: 89 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-14T04:11:21.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:21.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:22.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:22.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:22.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:22.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:22.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:22.185 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:22.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:22.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:22.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:22.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:22.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:22.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:22.439 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:22.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:22.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:22.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:22.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:22.670 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:22.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:22.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:22.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:22.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:22.854 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:22.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:22.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:22.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:22.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:22.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:23.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:23.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:23.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:23.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:23.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:23.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:23.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:23.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:23.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:23.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:23.431 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:23.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:23.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:23.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:23.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:23.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:23.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:23.700 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:23.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:23.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:23.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:23.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:23.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:23.952 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:23.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:24.054 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:24.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:24.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:24.090 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:23 smithi114 ceph-mon[106969]: pgmap v824: 89 pgs: 89 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-14T04:11:24.090 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:23 smithi114 ceph-mon[103029]: pgmap v824: 89 pgs: 89 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-14T04:11:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:23 smithi160 ceph-mon[88407]: pgmap v824: 89 pgs: 89 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 816 op/s 2024-09-14T04:11:24.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:24.208 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:24.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:24.245 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:11:24.245 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:11:24.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:24.353 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:24.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:24.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:24.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:24.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:24.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:24.626 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:24.627 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:24.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:24.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:24.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:24.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:24.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:24.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:24.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:24.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:25.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:25.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:25.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:25.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:25.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:25.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:25.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:25.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:25.322 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:25.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:25.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:25.424 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:25.454 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:25.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:25.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:25.556 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:25.584 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:25.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:25.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:25.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:25.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:25.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:25.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:25.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:25.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:25.985 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:25.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:26.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-14T04:11:26.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:26.137 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:26.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:25 smithi160 ceph-mon[88407]: pgmap v825: 89 pgs: 89 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-09-14T04:11:26.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:25 smithi114 ceph-mon[106969]: pgmap v825: 89 pgs: 89 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-09-14T04:11:26.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:25 smithi114 ceph-mon[103029]: pgmap v825: 89 pgs: 89 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-09-14T04:11:26.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:26.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:26.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:26.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:26.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:26.377 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:11:26.409 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:26.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:26.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:26.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:26.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:26.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:26.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:26.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:26.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:26.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:26.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:26.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:26.930 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:11:26.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:26.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:26.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:27.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:27.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:27.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:27.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:26 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/653333905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:11:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:26 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:11:27.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:27.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:27.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:27.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:27.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:27.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:26 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/653333905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:11:27.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:26 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:11:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:27.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:27.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:26 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/653333905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:11:27.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:26 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:11:27.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:27.513 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:27.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:27.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:27.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:27.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:27.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:27.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:27.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:27.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:27.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:27.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:27.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:28.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:27 smithi114 ceph-mon[103029]: pgmap v826: 89 pgs: 89 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 721 op/s 2024-09-14T04:11:28.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:27 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:11:28.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:27 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:11:28.025 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:27 smithi114 ceph-mon[103029]: osdmap e223: 8 total, 8 up, 7 in 2024-09-14T04:11:28.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:27 smithi114 ceph-mon[106969]: pgmap v826: 89 pgs: 89 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 721 op/s 2024-09-14T04:11:28.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:27 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:11:28.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:27 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:11:28.026 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:27 smithi114 ceph-mon[106969]: osdmap e223: 8 total, 8 up, 7 in 2024-09-14T04:11:28.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:28.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:28.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:27 smithi160 ceph-mon[88407]: pgmap v826: 89 pgs: 89 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 721 op/s 2024-09-14T04:11:28.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:27 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:11:28.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:27 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:11:28.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:27 smithi160 ceph-mon[88407]: osdmap e223: 8 total, 8 up, 7 in 2024-09-14T04:11:28.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:28.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:28.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:28.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:28.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:28.335 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:28.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:28.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:28.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:28.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:28.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:28.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:28.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:28.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:28.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:28.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:28.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:28.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:28.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:29.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:29.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:29.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:29.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:29.190 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:29.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:29.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:29.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:29.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:29.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:29.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:29.445 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:29.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:29.546 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:29.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:29.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:29.671 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:29.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:29.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:29.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:29.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:29.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-14T04:11:29.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:29.986 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:29.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:30.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:30.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[103029]: pgmap v828: 89 pgs: 89 active+clean; 1.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-14T04:11:30.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:30.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:30.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1600978811' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:11:30.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:11:30.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:11:30.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:11:30.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:30.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[106969]: pgmap v828: 89 pgs: 89 active+clean; 1.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-14T04:11:30.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:30.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:30.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1600978811' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:11:30.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:11:30.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:11:30.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:11:30.124 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:30.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:30.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:30.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:29 smithi160 ceph-mon[88407]: pgmap v828: 89 pgs: 89 active+clean; 1.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-14T04:11:30.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:30.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:30.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:29 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1600978811' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:11:30.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:29 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:11:30.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:11:30.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:11:30.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:11:30.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:30.273 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:30.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:30.285 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:11:30.285 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:11:30.374 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:30.399 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:30.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:30.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:30.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:30.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:30.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:30.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:30.630 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:30.650 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:11:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:11:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:11:30.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:30.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:30.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:30.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:30.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:30.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:30.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:30.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:30 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:11:30.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:30 smithi160 ceph-mon[88407]: osdmap e224: 8 total, 8 up, 7 in 2024-09-14T04:11:30.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:30.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:30.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:31.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:31.085 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:31.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:31.186 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:31.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:31.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:31.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:31.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:30 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:11:31.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:30 smithi114 ceph-mon[106969]: osdmap e224: 8 total, 8 up, 7 in 2024-09-14T04:11:31.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:30 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:11:31.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:30 smithi114 ceph-mon[103029]: osdmap e224: 8 total, 8 up, 7 in 2024-09-14T04:11:31.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:31.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:31.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-14T04:11:31.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:31.513 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:31.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:31.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-14T04:11:31.610 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:11:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:11:31] "GET /metrics HTTP/1.1" 200 36483 "" "Prometheus/2.51.0" 2024-09-14T04:11:31.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:31.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:31.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:31.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:31.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:31.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:31.871 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:11:31.915 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:31.939 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:31.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:31.975 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:11:31.976 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:11:31.983 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:32.009 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:11:32.010 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:11:32.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:32.048 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:11:32.048 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:11:32.082 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:11:32.082 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:11:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:32.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:32.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:32.115 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:11:32.115 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:11:32.143 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:11:32.143 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:11:32.144 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:31 smithi114 ceph-mon[103029]: pgmap v830: 89 pgs: 89 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-14T04:11:32.144 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:31 smithi114 ceph-mon[106969]: pgmap v830: 89 pgs: 89 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-14T04:11:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:31 smithi160 ceph-mon[88407]: pgmap v830: 89 pgs: 89 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-14T04:11:32.170 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:11:32.170 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:11:32.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:32.198 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:11:32.199 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:11:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:32.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:32.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:32.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:32.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:32.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:32.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:32.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:32.505 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:32.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:32.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:11:32.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:32.658 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:32.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:32.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:32.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:32.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:32.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:32.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:32.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:32.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:33.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:33.072 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:33.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:33.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:33.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:33.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:33.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:33.342 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:33.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:33.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:32 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/970162996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:33.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:32 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/970162996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:33.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:32 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/970162996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:33.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:33.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:33.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:33.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:33.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:33.607 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:33.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:33.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:33.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:33.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:33.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:33.887 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:33.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:33.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:33 smithi114 ceph-mon[106969]: pgmap v831: 89 pgs: 89 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 566 op/s 2024-09-14T04:11:33.965 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:33 smithi114 ceph-mon[103029]: pgmap v831: 89 pgs: 89 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 566 op/s 2024-09-14T04:11:33.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:34.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:34.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:34.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:34.163 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:34.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:34.238 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:33 smithi160 ceph-mon[88407]: pgmap v831: 89 pgs: 89 active+clean; 1.5 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 566 op/s 2024-09-14T04:11:34.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:34.310 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:34.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:34.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:34.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:34.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:34.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:34.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:34.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:34.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:34.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:34.702 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:11:34.702 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:11:34.702 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:11:34.702 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:11:34.702 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:11:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:34.712 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:34.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:34.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:34.815 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:34.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:34.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:34.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:34.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:34.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:34.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:35.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:35.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:35.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:35.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:35.229 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:35.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:35.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:35.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:35.359 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:35.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:35.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:35.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:35.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-14T04:11:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:35.492 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:35.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:35.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:35.636 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:35.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:35.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:11:35.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:35.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:35.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:35.807 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:11:35.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:35.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:35.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:36.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:36.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:36.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:36.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:36.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:36.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:36.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:36.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:36.283 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:35 smithi160 ceph-mon[88407]: pgmap v832: 89 pgs: 89 active+clean; 1.3 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 932 op/s 2024-09-14T04:11:36.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:36.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:36.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:36.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:35 smithi114 ceph-mon[106969]: pgmap v832: 89 pgs: 89 active+clean; 1.3 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 932 op/s 2024-09-14T04:11:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:35 smithi114 ceph-mon[103029]: pgmap v832: 89 pgs: 89 active+clean; 1.3 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 932 op/s 2024-09-14T04:11:36.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:36.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:36.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:36.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:36.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:36.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:36.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:36.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:36.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:36.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:36.927 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:36.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:36.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:36.951 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-14T04:11:37.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:37.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:37.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:37.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:37.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:37.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:37.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:37.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:36 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1800734446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:11:37.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:36 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:11:37.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:36 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1800734446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:11:37.299 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:36 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:11:37.334 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:37.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:37.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:36 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1800734446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:11:37.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:36 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:11:37.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:37.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:37.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:37.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:37.591 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:37.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:37.663 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-14T04:11:37.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:11:37.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:37.747 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:37.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:37.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:37.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:37.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:37.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:37.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:38.020 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:38.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:38.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:38.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:38.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:37 smithi114 ceph-mon[103029]: pgmap v833: 89 pgs: 89 active+clean; 1.3 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 800 op/s 2024-09-14T04:11:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:37 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:11:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:37 smithi114 ceph-mon[103029]: osdmap e225: 8 total, 8 up, 7 in 2024-09-14T04:11:38.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:37 smithi114 ceph-mon[106969]: pgmap v833: 89 pgs: 89 active+clean; 1.3 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 800 op/s 2024-09-14T04:11:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:37 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:11:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:37 smithi114 ceph-mon[106969]: osdmap e225: 8 total, 8 up, 7 in 2024-09-14T04:11:38.240 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:38.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:38.292 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:37 smithi160 ceph-mon[88407]: pgmap v833: 89 pgs: 89 active+clean; 1.3 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 800 op/s 2024-09-14T04:11:38.292 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:37 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:11:38.293 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:37 smithi160 ceph-mon[88407]: osdmap e225: 8 total, 8 up, 7 in 2024-09-14T04:11:38.293 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:38.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:38.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:38.371 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:38.383 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:38.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:38.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:38.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:38.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:38.567 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:11:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:11:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:11:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:11:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220880 9.f 9:f96ede54:::benchmark_data_smithi114_153893_object91419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-14T04:11:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.196403+0000", 2024-09-14T04:11:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.35268587399997, 2024-09-14T04:11:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86833886999999998, 2024-09-14T04:11:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220880 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196403+0000", 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196403+0000", 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196401+0000", 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196409+0000", 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.131e-06 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196425+0000", 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6512e-05 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.196431+0000", 2024-09-14T04:11:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2910000000000004e-06 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.197080+0000", 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00064894099999999999 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.197087+0000", 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9630000000000001e-06 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.044872+0000", 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84778513200000005 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.044981+0000", 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108917 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.045069+0000", 2024-09-14T04:11:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7620000000000005e-05 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.055218+0000", 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010148634 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058484+0000", 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032667180000000001 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064649+0000", 2024-09-14T04:11:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061650619999999998 2024-09-14T04:11:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064698+0000", 2024-09-14T04:11:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8887000000000001e-05 2024-09-14T04:11:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064742+0000", 2024-09-14T04:11:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3894999999999999e-05 2024-09-14T04:11:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220882 9.f 9:f2f60c4f:::benchmark_data_smithi114_153893_object91421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.201420+0000", 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.34766913599998, 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86631311799999999, 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220882 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201420+0000", 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201420+0000", 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201418+0000", 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201427+0000", 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.721e-06 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201440+0000", 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2972e-05 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201446+0000", 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5430000000000001e-06 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201484+0000", 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8325000000000003e-05 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.201492+0000", 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6620000000000005e-06 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.048696+0000", 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84720400699999998 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.048765+0000", 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9080999999999998e-05 2024-09-14T04:11:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.048846+0000", 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1446999999999995e-05 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058553+0000", 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0097070750000000008 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058613+0000", 2024-09-14T04:11:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0253000000000001e-05 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067623+0000", 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090100319999999994 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067687+0000", 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3064999999999993e-05 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067733+0000", 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6666999999999999e-05 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220900 9.f 9:ff716adf:::benchmark_data_smithi114_153893_object91439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.216323+0000", 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.33276641399999, 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88042596299999998, 2024-09-14T04:11:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220900 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216323+0000", 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216323+0000", 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216322+0000", 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216328+0000", 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9449999999999998e-06 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216337+0000", 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6059999999999992e-06 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216343+0000", 2024-09-14T04:11:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5729999999999998e-06 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216588+0000", 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024550400000000001 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.216594+0000", 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1920000000000002e-06 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.051961+0000", 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83536656399999998 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.052033+0000", 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1588999999999995e-05 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.052105+0000", 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2805999999999994e-05 2024-09-14T04:11:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.061647+0000", 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0095416719999999993 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067789+0000", 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061421050000000001 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096653+0000", 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.028863903999999999 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096700+0000", 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6773000000000001e-05 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096749+0000", 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8934999999999998e-05 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220903 9.f 9:f4af0834:::benchmark_data_smithi114_153893_object91442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.217818+0000", 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.33127071500002, 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87909124900000002, 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220903 2024-09-14T04:11:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217818+0000", 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217818+0000", 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217818+0000", 2024-09-14T04:11:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217823+0000", 2024-09-14T04:11:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3249999999999998e-06 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217833+0000", 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0246e-05 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217839+0000", 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6069999999999998e-06 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217875+0000", 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5756999999999997e-05 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.217881+0000", 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5629999999999997e-06 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.055244+0000", 2024-09-14T04:11:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83736241899999997 2024-09-14T04:11:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.055357+0000", 2024-09-14T04:11:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011300500000000001 2024-09-14T04:11:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.055477+0000", 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119938 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064752+0000", 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0092754980000000001 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.087734+0000", 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.022982473 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096831+0000", 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090970070000000007 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096881+0000", 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9044e-05 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096910+0000", 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9207e-05 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220908 9.f 9:f2004091:::benchmark_data_smithi114_153893_object91447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.222577+0000", 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.32651248899998, 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87448498699999999, 2024-09-14T04:11:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220908 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222577+0000", 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222577+0000", 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222575+0000", 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222581+0000", 2024-09-14T04:11:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9739999999999999e-06 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222595+0000", 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3331e-05 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222601+0000", 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4980000000000003e-06 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222652+0000", 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0711000000000003e-05 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.222661+0000", 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8449999999999992e-06 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058645+0000", 2024-09-14T04:11:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83598437800000003 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058734+0000", 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9221000000000005e-05 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.058834+0000", 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9858000000000005e-05 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067744+0000", 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0089099409999999993 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.087771+0000", 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020027117000000001 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096969+0000", 2024-09-14T04:11:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0091981110000000001 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097015+0000", 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5170999999999998e-05 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097062+0000", 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7061000000000002e-05 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220919 9.f 9:f2584fce:::benchmark_data_smithi114_153893_object91458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.231868+0000", 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.31722111300002, 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.873098869, 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220919 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231868+0000", 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231868+0000", 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231867+0000", 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231871+0000", 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3699999999999997e-06 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231879+0000", 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9780000000000005e-06 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.231883+0000", 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0609999999999997e-06 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.234088+0000", 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002204559 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.234095+0000", 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0679999999999999e-06 2024-09-14T04:11:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.061684+0000", 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.827589456 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.061769+0000", 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4770000000000003e-05 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.061840+0000", 2024-09-14T04:11:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.046e-05 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067748+0000", 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059083110000000003 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097092+0000", 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029343899999999999 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.104874+0000", 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0077821729999999999 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.104924+0000", 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0223999999999999e-05 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.104967+0000", 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2629000000000001e-05 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220936 9.f 9:f88a3439:::benchmark_data_smithi114_153893_object91475:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.252191+0000", 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.29689810999997, 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85289873000000005, 2024-09-14T04:11:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220936 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252191+0000", 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252191+0000", 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252189+0000", 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252196+0000", 2024-09-14T04:11:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4999999999999996e-06 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252209+0000", 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3634e-05 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252215+0000", 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9569999999999999e-06 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252764+0000", 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00054909399999999997 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.252773+0000", 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1589999999999993e-06 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064770+0000", 2024-09-14T04:11:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81199766200000001 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064844+0000", 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3677000000000006e-05 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.064910+0000", 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6523e-05 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.075180+0000", 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010269967 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097135+0000", 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021954497 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.105017+0000", 2024-09-14T04:11:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0078818959999999993 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.105058+0000", 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0750000000000001e-05 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.105090+0000", 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2270000000000001e-05 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220943 9.f 9:f08a27fb:::benchmark_data_smithi114_153893_object91482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.260673+0000", 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.28841613200001, 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85680367899999998, 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220943 2024-09-14T04:11:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260673+0000", 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260673+0000", 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260671+0000", 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260678+0000", 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9519999999999997e-06 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260694+0000", 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6223e-05 2024-09-14T04:11:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260700+0000", 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9699999999999996e-06 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260765+0000", 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4989999999999999e-05 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.260773+0000", 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7840000000000004e-06 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067803+0000", 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.807030256 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067880+0000", 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6699999999999994e-05 2024-09-14T04:11:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.067970+0000", 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9563999999999998e-05 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.083713+0000", 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015743183000000001 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097183+0000", 2024-09-14T04:11:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013470355999999999 2024-09-14T04:11:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117401+0000", 2024-09-14T04:11:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020217386 2024-09-14T04:11:38.628 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117442+0000", 2024-09-14T04:11:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1891999999999998e-05 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117477+0000", 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4264999999999998e-05 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220947 9.f 9:fe0d007e:::benchmark_data_smithi114_153893_object91486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.263647+0000", 2024-09-14T04:11:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.28544181299998, 2024-09-14T04:11:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85404936600000003, 2024-09-14T04:11:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220947 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263647+0000", 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263647+0000", 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263646+0000", 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263652+0000", 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4670000000000003e-06 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263662+0000", 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6139999999999998e-06 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263667+0000", 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4859999999999997e-06 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263714+0000", 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7447000000000002e-05 2024-09-14T04:11:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.263721+0000", 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.387e-06 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.071744+0000", 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80802356200000003 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.071828+0000", 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3418999999999995e-05 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.071899+0000", 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0732e-05 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096601+0000", 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.024702867 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097216+0000", 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000614085 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117634+0000", 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020418546999999999 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117668+0000", 2024-09-14T04:11:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3683999999999998e-05 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117697+0000", 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8969e-05 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220953 9.f 9:fc60803b:::benchmark_data_smithi114_153893_object91492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.270585+0000", 2024-09-14T04:11:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.27850459799998, 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84728676199999997, 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220953 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270585+0000", 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270585+0000", 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270584+0000", 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270589+0000", 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7080000000000002e-06 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270605+0000", 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5421e-05 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270611+0000", 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8029999999999999e-06 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270701+0000", 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9697999999999994e-05 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.270708+0000", 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1269999999999997e-06 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.075221+0000", 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80451222499999997 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.075322+0000", 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101952 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.075437+0000", 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011485900000000001 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096610+0000", 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021172291999999999 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097251+0000", 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00064100699999999997 2024-09-14T04:11:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117821+0000", 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020570082999999999 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117844+0000", 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2796e-05 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117871+0000", 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7840000000000001e-05 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220961 9.f 9:f1118824:::benchmark_data_smithi114_153893_object91500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.280922+0000", 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.26816672299998, 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83707822399999998, 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220961 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280922+0000", 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280922+0000", 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280921+0000", 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280927+0000", 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7919999999999998e-06 2024-09-14T04:11:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280938+0000", 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1375e-05 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.280945+0000", 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.652e-06 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.281000+0000", 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5179999999999997e-05 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.281009+0000", 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8030000000000003e-06 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.079375+0000", 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.79836644999999995 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.079493+0000", 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117242 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.079610+0000", 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117735 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096761+0000", 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017151079999999999 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097273+0000", 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00051192499999999999 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117917+0000", 2024-09-14T04:11:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020643812000000001 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.117962+0000", 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4338999999999997e-05 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.118001+0000", 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9165999999999997e-05 2024-09-14T04:11:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220963 9.f 9:fa967625:::benchmark_data_smithi114_153893_object91502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.283564+0000", 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.26552558200001, 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83455748200000002, 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220963 2024-09-14T04:11:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283564+0000", 2024-09-14T04:11:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283564+0000", 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283563+0000", 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283572+0000", 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3070000000000008e-06 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283582+0000", 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4790000000000002e-06 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283586+0000", 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1899999999999997e-06 2024-09-14T04:11:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283615+0000", 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8847e-05 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.283621+0000", 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2149999999999998e-06 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.083752+0000", 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80013120199999999 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.083870+0000", 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117824 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.083994+0000", 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012407999999999999 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096768+0000", 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012773746000000001 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097306+0000", 2024-09-14T04:11:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00053861299999999996 2024-09-14T04:11:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.118040+0000", 2024-09-14T04:11:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020733445999999999 2024-09-14T04:11:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.118083+0000", 2024-09-14T04:11:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3464999999999998e-05 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.118121+0000", 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7966000000000002e-05 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220968 9.f 9:f83ac14a:::benchmark_data_smithi114_153893_object91507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.290423+0000", 2024-09-14T04:11:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.25866625200001, 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.83063764699999998, 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220968 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290423+0000", 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290423+0000", 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290422+0000", 2024-09-14T04:11:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290426+0000", 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9559999999999999e-06 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290435+0000", 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9630000000000004e-06 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.290440+0000", 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7489999999999998e-06 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.291044+0000", 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00060378600000000004 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.291051+0000", 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1350000000000003e-06 2024-09-14T04:11:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.087791+0000", 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.79674028699999999 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.087905+0000", 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011346200000000001 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.088019+0000", 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011462499999999999 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.096775+0000", 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0087554710000000008 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097356+0000", 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00058162199999999996 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.120960+0000", 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.023603678999999999 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.121005+0000", 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4607000000000002e-05 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.121061+0000", 2024-09-14T04:11:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6013000000000001e-05 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:220980 9.f 9:f3c12ea5:::benchmark_data_smithi114_153893_object91519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:40.311552+0000", 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 298.23753670899998, 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80967589200000001, 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 220980 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311552+0000", 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311552+0000", 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311551+0000", 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311557+0000", 2024-09-14T04:11:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3519999999999997e-06 2024-09-14T04:11:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311565+0000", 2024-09-14T04:11:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6689999999999995e-06 2024-09-14T04:11:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311569+0000", 2024-09-14T04:11:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1860000000000002e-06 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311601+0000", 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1862000000000001e-05 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:40.311608+0000", 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9149999999999999e-06 2024-09-14T04:11:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.091990+0000", 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78038170600000001 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.092117+0000", 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012667200000000001 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.092253+0000", 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013666899999999999 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.097073+0000", 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048195069999999998 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.104825+0000", 2024-09-14T04:11:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0077518580000000004 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.121153+0000", 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016328667000000002 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.121192+0000", 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9079999999999999e-05 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:41.121228+0000", 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6010000000000003e-05 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:74348 10.9 10:9382f1a2:::benchmark_data_smithi114_188657_object74347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:08:45.915364+0000", 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 172.633725204, 2024-09-14T04:11:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17263257400000001, 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 74348 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.915364+0000", 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.915364+0000", 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.915362+0000", 2024-09-14T04:11:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.915403+0000", 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0763999999999998e-05 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.915414+0000", 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1567e-05 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.915420+0000", 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3770000000000004e-06 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.915464+0000", 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4085999999999998e-05 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.915573+0000", 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010931599999999999 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.915690+0000", 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116591 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.920681+0000", 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049910040000000003 2024-09-14T04:11:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.920724+0000", 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2701000000000003e-05 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.087912+0000", 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.167188058 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.087963+0000", 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1010999999999997e-05 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.087997+0000", 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3949999999999999e-05 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:74355 10.9 10:9ebebcd9:::benchmark_data_smithi114_188657_object74354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-14T04:11:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:08:45.918121+0000", 2024-09-14T04:11:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 172.630968212, 2024-09-14T04:11:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17234955599999999, 2024-09-14T04:11:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 74355 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.918121+0000", 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.918121+0000", 2024-09-14T04:11:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.918119+0000", 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.918161+0000", 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1989000000000001e-05 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.918174+0000", 2024-09-14T04:11:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.258e-05 2024-09-14T04:11:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.918180+0000", 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1619999999999996e-06 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.918219+0000", 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8872999999999999e-05 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.918327+0000", 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010751600000000001 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.918478+0000", 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015122000000000001 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.920910+0000", 2024-09-14T04:11:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024323600000000002 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.922952+0000", 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002041504 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090390+0000", 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16743875599999999 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090436+0000", 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5859999999999998e-05 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090471+0000", 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4325000000000001e-05 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:74379 10.9 10:9ec773f2:::benchmark_data_smithi114_188657_object74378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-14T04:11:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:08:45.936659+0000", 2024-09-14T04:11:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 172.61243040599999, 2024-09-14T04:11:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15393995999999999, 2024-09-14T04:11:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 74379 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.936659+0000", 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.936659+0000", 2024-09-14T04:11:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.936657+0000", 2024-09-14T04:11:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.936707+0000", 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9424e-05 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.936721+0000", 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4166e-05 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.936728+0000", 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3799999999999996e-06 2024-09-14T04:11:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.936822+0000", 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3332999999999996e-05 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.936942+0000", 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012021800000000001 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:11:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.937084+0000", 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014243799999999999 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.941992+0000", 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049072660000000004 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.942415+0000", 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00042346000000000001 2024-09-14T04:11:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090523+0000", 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.14810795600000001 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090566+0000", 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3167000000000002e-05 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090599+0000", 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2622e-05 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:74380 10.9 10:95512621:::benchmark_data_smithi114_188657_object74379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:08:45.937293+0000", 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 172.611796325, 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.153584322, 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 74380 2024-09-14T04:11:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.937293+0000", 2024-09-14T04:11:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.937293+0000", 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.937292+0000", 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.937352+0000", 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0059000000000002e-05 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.937363+0000", 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1069e-05 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.937369+0000", 2024-09-14T04:11:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4419999999999997e-06 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.939400+0000", 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020311209999999999 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.939550+0000", 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0001504 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.939711+0000", 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016125800000000001 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.942213+0000", 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025016959999999999 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.942567+0000", 2024-09-14T04:11:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035349099999999999 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090778+0000", 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.14821152000000001 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090838+0000", 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0130999999999997e-05 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090877+0000", 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8992999999999997e-05 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:74395 10.9 10:920762c3:::benchmark_data_smithi114_188657_object74394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:08:45.954501+0000", 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 172.59458787899999, 2024-09-14T04:11:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.136493051, 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 74395 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.954501+0000", 2024-09-14T04:11:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.954501+0000", 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.954501+0000", 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.954554+0000", 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3575999999999997e-05 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.954562+0000", 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2619999999999998e-06 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.954566+0000", 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2480000000000001e-06 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.954616+0000", 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0549e-05 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.954751+0000", 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013481100000000001 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:11:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.954910+0000", 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015942800000000001 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.957010+0000", 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002100092 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:45.957119+0000", 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010821 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090923+0000", 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13380398700000001 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090965+0000", 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2327999999999999e-05 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:46.090994+0000", 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9371000000000001e-05 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:177488 10.6 10:6d64f37c:::benchmark_data_smithi114_188657_object57292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:10:41.726099+0000", 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 56.822989776, 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.124412039, 2024-09-14T04:11:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 177488 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.726099+0000", 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.726099+0000", 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.726098+0000", 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.726104+0000", 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5720000000000002e-06 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.726116+0000", 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1999e-05 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.726121+0000", 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6420000000000004e-06 2024-09-14T04:11:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.726165+0000", 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3662000000000003e-05 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.726278+0000", 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000112883 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.726385+0000", 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010705900000000001 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.731413+0000", 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0050284809999999996 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.732839+0000", 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014251380000000001 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.850420+0000", 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11758099499999999 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.850472+0000", 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2262999999999998e-05 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:41.850511+0000", 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9678999999999997e-05 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:38.724 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:11:38.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:38.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:38.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:38.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:38.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:38.964 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:11:38.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:39.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:39.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:39.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2084254544' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:39.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2084254544' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:39.141 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:39.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:39.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:39.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:39.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:39.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:39.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2084254544' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:39.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:39.439 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:39.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:39.540 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:39.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:39.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:39.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:39.713 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:39.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:39.721 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-14T04:11:39.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:39.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:39.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:39.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:39.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:39.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:40.072 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:11:40.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:40.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:40.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:39 smithi114 ceph-mon[103029]: pgmap v835: 89 pgs: 89 active+clean; 1.2 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 574 op/s 2024-09-14T04:11:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:39 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:11:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:39 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:11:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2084254544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:11:40.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:39 smithi114 ceph-mon[103029]: osdmap e226: 8 total, 8 up, 7 in 2024-09-14T04:11:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:39 smithi114 ceph-mon[106969]: pgmap v835: 89 pgs: 89 active+clean; 1.2 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 574 op/s 2024-09-14T04:11:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:39 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:11:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:39 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:11:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2084254544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:11:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:39 smithi114 ceph-mon[106969]: osdmap e226: 8 total, 8 up, 7 in 2024-09-14T04:11:40.207 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:39 smithi160 ceph-mon[88407]: pgmap v835: 89 pgs: 89 active+clean; 1.2 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 574 op/s 2024-09-14T04:11:40.207 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:39 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:11:40.207 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:39 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:11:40.207 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2084254544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:11:40.207 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:39 smithi160 ceph-mon[88407]: osdmap e226: 8 total, 8 up, 7 in 2024-09-14T04:11:40.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:40.276 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:40.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:40.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:40.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:40.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:40.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:40.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:40.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:40.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:40.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:40.567 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:11:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:11:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:11:40.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:40.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:40.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:40.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:40.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:40.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:40.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:40.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:40.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:41.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:41.108 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:41.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:41.210 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:41.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:41.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:41.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:41.390 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:41.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:41.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:41.522 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:41.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:41.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:41.649 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:41.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:41.751 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:41.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:41.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:41.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:11:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:11:41] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-09-14T04:11:41.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:41.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:41.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:42.006 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:41 smithi160 ceph-mon[88407]: pgmap v837: 89 pgs: 89 active+clean; 1.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-09-14T04:11:42.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:42.026 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:11:42.026 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:226083 10.d 10:b5828a46:::benchmark_data_smithi114_188657_object105887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:11:41.994596+0000", 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.010698483, 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010720036, 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 226083 2024-09-14T04:11:42.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994596+0000", 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994596+0000", 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994596+0000", 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994600+0000", 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3769999999999998e-06 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994608+0000", 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2209999999999997e-06 2024-09-14T04:11:42.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994613+0000", 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4220000000000002e-06 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994646+0000", 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3197000000000001e-05 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:42.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994727+0000", 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0609e-05 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994804+0000", 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7688999999999994e-05 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.999093+0000", 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0042882470000000002 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.031 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:226075 10.d 10:b3c3e480:::benchmark_data_smithi114_188657_object105879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:11:41.985479+0000", 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.019815966000000001, 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.019888626, 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:42.032 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 226075 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985479+0000", 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985479+0000", 2024-09-14T04:11:42.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:42.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:42.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985478+0000", 2024-09-14T04:11:42.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:11:42.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985482+0000", 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2090000000000001e-06 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985487+0000", 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2399999999999998e-06 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985490+0000", 2024-09-14T04:11:42.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9119999999999998e-06 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985514+0000", 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4207e-05 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985590+0000", 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5913000000000003e-05 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:11:42.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985705+0000", 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115207 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.991625+0000", 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059196090000000002 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:226076 10.7 10:e1495e37:::benchmark_data_smithi114_188657_object105880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-14T04:11:42.037 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:11:41.985571+0000", 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.019724179000000001, 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.019853504000000001, 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 226076 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:42.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985571+0000", 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985571+0000", 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985570+0000", 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985574+0000", 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6109999999999998e-06 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985581+0000", 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4989999999999997e-06 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985586+0000", 2024-09-14T04:11:42.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6140000000000002e-06 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985628+0000", 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2558000000000001e-05 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985724+0000", 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5329999999999997e-05 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,5", 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.985830+0000", 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106759 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994860+0000", 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090291769999999993 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994866+0000", 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8299999999999998e-06 2024-09-14T04:11:42.042 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:226078 10.7 10:e3262a75:::benchmark_data_smithi114_188657_object105882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:11:41.989745+0000", 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.015549566000000001, 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015736843, 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:42.043 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 226078 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.989745+0000", 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.989745+0000", 2024-09-14T04:11:42.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.989744+0000", 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.989749+0000", 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0509999999999996e-06 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.989761+0000", 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2126e-05 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.989767+0000", 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3550000000000004e-06 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.989835+0000", 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8167e-05 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.990503+0000", 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066846099999999997 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,5", 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.990622+0000", 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118476 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.999148+0000", 2024-09-14T04:11:42.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0085259800000000007 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:42.000908+0000", 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017599169999999999 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:226082 10.7 10:e2f74747:::benchmark_data_smithi114_188657_object105886:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-14T04:11:42.048 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:11:41.994267+0000", 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.011027657, 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011258633000000001, 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 226082 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:42.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994267+0000", 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994267+0000", 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994266+0000", 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994271+0000", 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0359999999999997e-06 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994282+0000", 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0312999999999999e-05 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994287+0000", 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-14T04:11:42.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.994914+0000", 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062768600000000002 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.995027+0000", 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011233700000000001 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,5", 2024-09-14T04:11:42.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.995160+0000", 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013317 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:41.999249+0000", 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040893149999999996 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:11:42.001029+0000", 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017795910000000001 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.053 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:42.054 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.054 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:42.054 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:11:42.054 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 5 2024-09-14T04:11:42.054 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:11:42.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:42.055 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:42.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:42.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:42.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:11:42.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:42.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:42.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:42.237 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:41 smithi114 ceph-mon[106969]: pgmap v837: 89 pgs: 89 active+clean; 1.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-09-14T04:11:42.237 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:41 smithi114 ceph-mon[103029]: pgmap v837: 89 pgs: 89 active+clean; 1.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 970 op/s 2024-09-14T04:11:42.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:42.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:42.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:42.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:42.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:42.484 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:42.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:42.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:42.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:42.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:42.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:42.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:42.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:42.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:42.872 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:42.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:42.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:43.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:43.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:43.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:43.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:43.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3531421888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:43.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3531421888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:43.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:43.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:43.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:43.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:43.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:43.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3531421888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:43.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:43.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:43.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:43.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:43.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:43.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:43.611 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:11:43.612 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:11:43.612 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:11:43.612 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:11:43.612 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:11:43.612 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:11:43.612 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:11:43.612 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:11:43.613 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:11:43.613 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:11:43.613 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:11:43.613 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:11:43.613 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:11:43.613 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:11:43.613 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:11:43.614 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:11:43.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:43.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:43.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:43.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:43.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:43.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:43.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:43.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:43.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:43.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:44.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:44.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-14T04:11:44.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:44.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:44.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:44.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:44.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:44.292 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:43 smithi114 ceph-mon[106969]: pgmap v838: 89 pgs: 89 active+clean; 1.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 543 op/s 2024-09-14T04:11:44.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:43 smithi114 ceph-mon[103029]: pgmap v838: 89 pgs: 89 active+clean; 1.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 543 op/s 2024-09-14T04:11:44.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:44.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:44.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:44.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:43 smithi160 ceph-mon[88407]: pgmap v838: 89 pgs: 89 active+clean; 1.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 543 op/s 2024-09-14T04:11:44.441 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:11:44.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:44.506 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:44.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:44.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:44.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:44.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:44.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:44.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:44.804 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:44.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:44.841 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:44.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:44.939 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:44.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:45.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:45.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:45.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:45.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:45.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:45.226 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:45.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:45.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:45.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:45.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:45.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:45.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:45.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:45.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:45.628 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:45.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:45.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:45.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:45.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:45.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:45.897 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:45.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:45.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:46.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:45 smithi114 ceph-mon[106969]: pgmap v839: 89 pgs: 89 active+clean; 830 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 923 op/s 2024-09-14T04:11:46.023 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:11:46.023 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:45 smithi114 ceph-mon[103029]: pgmap v839: 89 pgs: 89 active+clean; 830 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 923 op/s 2024-09-14T04:11:46.027 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:11:46.037 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:46.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:46.134 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:45 smithi160 ceph-mon[88407]: pgmap v839: 89 pgs: 89 active+clean; 830 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 923 op/s 2024-09-14T04:11:46.139 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:46.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:46.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:46.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:46.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:46.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:46.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:46.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:46.425 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:11:46.425 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:11:46.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:46.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:46.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:46.562 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:46.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:46.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:46.695 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:46.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:46.796 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:46.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:11:46.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:46.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:46.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:46.981 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":839,"stamp":"2024-09-14T04:11:44.830394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":870127589,"num_objects":13295,"num_object_clones":0,"num_object_copies":39885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13295,"num_whiteouts":0,"num_read":307,"num_read_kb":760,"num_write":227331,"num_write_kb":7694012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":595258,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130177,"ondisk_log_size":130177,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7586352,"kb_used_data":2598900,"kb_used_omap":27,"kb_used_meta":4987236,"kb_avail":648629712,"statfs":{"total":671965249536,"available":664196825088,"internally_reserved":0,"allocated":2661273600,"data_stored":2653944534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28483,"internal_metadata":5106929853},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,9,4,10,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-484573184,"num_objects":-7394,"num_object_clones":0,"num_object_copies":-22182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7394,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003005"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":260,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985283+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:11:38.985283+0000","last_peered":"2024-09-14T04:11:38.985283+0000","last_clean":"2024-09-14T04:11:38.985283+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:11:38.985283+0000","last_undegraded":"2024-09-14T04:11:38.985283+0000","last_fullsized":"2024-09-14T04:11:38.985283+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":209,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.947482+0000","last_change":"2024-09-14T04:05:55.701296+0000","last_active":"2024-09-14T04:11:36.947482+0000","last_peered":"2024-09-14T04:11:36.947482+0000","last_clean":"2024-09-14T04:11:36.947482+0000","last_became_active":"2024-09-14T04:05:55.676634+0000","last_became_peered":"2024-09-14T04:05:55.676634+0000","last_unstale":"2024-09-14T04:11:36.947482+0000","last_undegraded":"2024-09-14T04:11:36.947482+0000","last_fullsized":"2024-09-14T04:11:36.947482+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:44.187804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":191,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.941102+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:11:36.941102+0000","last_peered":"2024-09-14T04:11:36.941102+0000","last_clean":"2024-09-14T04:11:36.941102+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:11:36.941102+0000","last_undegraded":"2024-09-14T04:11:36.941102+0000","last_fullsized":"2024-09-14T04:11:36.941102+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":345,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985302+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:11:38.985302+0000","last_peered":"2024-09-14T04:11:38.985302+0000","last_clean":"2024-09-14T04:11:38.985302+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:11:38.985302+0000","last_undegraded":"2024-09-14T04:11:38.985302+0000","last_fullsized":"2024-09-14T04:11:38.985302+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":330,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995634+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:11:38.995634+0000","last_peered":"2024-09-14T04:11:38.995634+0000","last_clean":"2024-09-14T04:11:38.995634+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:11:38.995634+0000","last_undegraded":"2024-09-14T04:11:38.995634+0000","last_fullsized":"2024-09-14T04:11:38.995634+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":184,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.951977+0000","last_change":"2024-09-14T04:05:55.676850+0000","last_active":"2024-09-14T04:11:36.951977+0000","last_peered":"2024-09-14T04:11:36.951977+0000","last_clean":"2024-09-14T04:11:36.951977+0000","last_became_active":"2024-09-14T04:05:55.676337+0000","last_became_peered":"2024-09-14T04:05:55.676337+0000","last_unstale":"2024-09-14T04:11:36.951977+0000","last_undegraded":"2024-09-14T04:11:36.951977+0000","last_fullsized":"2024-09-14T04:11:36.951977+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:12.377665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":73,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.989151+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:11:38.989151+0000","last_peered":"2024-09-14T04:11:38.989151+0000","last_clean":"2024-09-14T04:11:38.989151+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:11:38.989151+0000","last_undegraded":"2024-09-14T04:11:38.989151+0000","last_fullsized":"2024-09-14T04:11:38.989151+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":200,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945969+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:11:36.945969+0000","last_peered":"2024-09-14T04:11:36.945969+0000","last_clean":"2024-09-14T04:11:36.945969+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:11:36.945969+0000","last_undegraded":"2024-09-14T04:11:36.945969+0000","last_fullsized":"2024-09-14T04:11:36.945969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"226'14174","reported_seq":14209,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:44.295733+0000","last_change":"2024-09-14T04:07:55.368334+0000","last_active":"2024-09-14T04:11:44.295733+0000","last_peered":"2024-09-14T04:11:44.295733+0000","last_clean":"2024-09-14T04:11:44.295733+0000","last_became_active":"2024-09-14T04:07:55.368016+0000","last_became_peered":"2024-09-14T04:07:55.368016+0000","last_unstale":"2024-09-14T04:11:44.295733+0000","last_undegraded":"2024-09-14T04:11:44.295733+0000","last_fullsized":"2024-09-14T04:11:44.295733+0000","mapping_epoch":199,"log_start":"206'4100","ondisk_log_start":"206'4100","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:13.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51511296,"num_objects":786,"num_object_clones":0,"num_object_copies":2358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14174,"num_write_kb":478720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":206,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.948162+0000","last_change":"2024-09-14T04:05:55.704897+0000","last_active":"2024-09-14T04:11:36.948162+0000","last_peered":"2024-09-14T04:11:36.948162+0000","last_clean":"2024-09-14T04:11:36.948162+0000","last_became_active":"2024-09-14T04:05:55.681093+0000","last_became_peered":"2024-09-14T04:05:55.681093+0000","last_unstale":"2024-09-14T04:11:36.948162+0000","last_undegraded":"2024-09-14T04:11:36.948162+0000","last_fullsized":"2024-09-14T04:11:36.948162+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:49.647801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":174,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.940925+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:11:36.940925+0000","last_peered":"2024-09-14T04:11:36.940925+0000","last_clean":"2024-09-14T04:11:36.940925+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:11:36.940925+0000","last_undegraded":"2024-09-14T04:11:36.940925+0000","last_fullsized":"2024-09-14T04:11:36.940925+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"226'14398","reported_seq":14433,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:43.820260+0000","last_change":"2024-09-14T04:07:55.607517+0000","last_active":"2024-09-14T04:11:43.820260+0000","last_peered":"2024-09-14T04:11:43.820260+0000","last_clean":"2024-09-14T04:11:43.820260+0000","last_became_active":"2024-09-14T04:07:55.607208+0000","last_became_peered":"2024-09-14T04:07:55.607208+0000","last_unstale":"2024-09-14T04:11:43.820260+0000","last_undegraded":"2024-09-14T04:11:43.820260+0000","last_fullsized":"2024-09-14T04:11:43.820260+0000","mapping_epoch":199,"log_start":"209'6800","ondisk_log_start":"209'6800","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7598,"log_dups_size":0,"ondisk_log_size":7598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:32:20.123209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53346304,"num_objects":814,"num_object_clones":0,"num_object_copies":2442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14398,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":207,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.948056+0000","last_change":"2024-09-14T04:05:55.702981+0000","last_active":"2024-09-14T04:11:36.948056+0000","last_peered":"2024-09-14T04:11:36.948056+0000","last_clean":"2024-09-14T04:11:36.948056+0000","last_became_active":"2024-09-14T04:05:55.679789+0000","last_became_peered":"2024-09-14T04:05:55.679789+0000","last_unstale":"2024-09-14T04:11:36.948056+0000","last_undegraded":"2024-09-14T04:11:36.948056+0000","last_fullsized":"2024-09-14T04:11:36.948056+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:34:24.358644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":250,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.991747+0000","last_change":"2024-09-14T04:07:44.340689+0000","last_active":"2024-09-14T04:11:38.991747+0000","last_peered":"2024-09-14T04:11:38.991747+0000","last_clean":"2024-09-14T04:11:38.991747+0000","last_became_active":"2024-09-14T04:07:44.340112+0000","last_became_peered":"2024-09-14T04:07:44.340112+0000","last_unstale":"2024-09-14T04:11:38.991747+0000","last_undegraded":"2024-09-14T04:11:38.991747+0000","last_fullsized":"2024-09-14T04:11:38.991747+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:04.263233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"226'14109","reported_seq":14175,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:43.668213+0000","last_change":"2024-09-14T04:07:55.608111+0000","last_active":"2024-09-14T04:11:43.668213+0000","last_peered":"2024-09-14T04:11:43.668213+0000","last_clean":"2024-09-14T04:11:43.668213+0000","last_became_active":"2024-09-14T04:07:55.607646+0000","last_became_peered":"2024-09-14T04:07:55.607646+0000","last_unstale":"2024-09-14T04:11:43.668213+0000","last_undegraded":"2024-09-14T04:11:43.668213+0000","last_fullsized":"2024-09-14T04:11:43.668213+0000","mapping_epoch":199,"log_start":"210'7100","ondisk_log_start":"210'7100","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7009,"log_dups_size":0,"ondisk_log_size":7009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:02:33.076539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54984704,"num_objects":839,"num_object_clones":0,"num_object_copies":2517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14109,"num_write_kb":478336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":175,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.948128+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:11:36.948128+0000","last_peered":"2024-09-14T04:11:36.948128+0000","last_clean":"2024-09-14T04:11:36.948128+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:11:36.948128+0000","last_undegraded":"2024-09-14T04:11:36.948128+0000","last_fullsized":"2024-09-14T04:11:36.948128+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":201,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.944606+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:11:36.944606+0000","last_peered":"2024-09-14T04:11:36.944606+0000","last_clean":"2024-09-14T04:11:36.944606+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:11:36.944606+0000","last_undegraded":"2024-09-14T04:11:36.944606+0000","last_fullsized":"2024-09-14T04:11:36.944606+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"226'14114","reported_seq":14180,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:40.744218+0000","last_change":"2024-09-14T04:07:55.607545+0000","last_active":"2024-09-14T04:11:40.744218+0000","last_peered":"2024-09-14T04:11:40.744218+0000","last_clean":"2024-09-14T04:11:40.744218+0000","last_became_active":"2024-09-14T04:07:55.607263+0000","last_became_peered":"2024-09-14T04:07:55.607263+0000","last_unstale":"2024-09-14T04:11:40.744218+0000","last_undegraded":"2024-09-14T04:11:40.744218+0000","last_fullsized":"2024-09-14T04:11:40.744218+0000","mapping_epoch":199,"log_start":"206'4100","ondisk_log_start":"206'4100","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:38:52.910168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64487424,"num_objects":984,"num_object_clones":0,"num_object_copies":2952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14114,"num_write_kb":483136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":212,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.941041+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:11:36.941041+0000","last_peered":"2024-09-14T04:11:36.941041+0000","last_clean":"2024-09-14T04:11:36.941041+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:11:36.941041+0000","last_undegraded":"2024-09-14T04:11:36.941041+0000","last_fullsized":"2024-09-14T04:11:36.941041+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":359,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.984979+0000","last_change":"2024-09-14T04:07:54.363033+0000","last_active":"2024-09-14T04:11:38.984979+0000","last_peered":"2024-09-14T04:11:38.984979+0000","last_clean":"2024-09-14T04:11:38.984979+0000","last_became_active":"2024-09-14T04:07:54.362706+0000","last_became_peered":"2024-09-14T04:07:54.362706+0000","last_unstale":"2024-09-14T04:11:38.984979+0000","last_undegraded":"2024-09-14T04:11:38.984979+0000","last_fullsized":"2024-09-14T04:11:38.984979+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:31.193597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"226'14226","reported_seq":14292,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:42.824486+0000","last_change":"2024-09-14T04:07:55.607327+0000","last_active":"2024-09-14T04:11:42.824486+0000","last_peered":"2024-09-14T04:11:42.824486+0000","last_clean":"2024-09-14T04:11:42.824486+0000","last_became_active":"2024-09-14T04:07:55.606967+0000","last_became_peered":"2024-09-14T04:07:55.606967+0000","last_unstale":"2024-09-14T04:11:42.824486+0000","last_undegraded":"2024-09-14T04:11:42.824486+0000","last_fullsized":"2024-09-14T04:11:42.824486+0000","mapping_epoch":199,"log_start":"210'7700","ondisk_log_start":"210'7700","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":6526,"log_dups_size":0,"ondisk_log_size":6526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:36.618823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58327040,"num_objects":890,"num_object_clones":0,"num_object_copies":2670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14226,"num_write_kb":483712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":299,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.991962+0000","last_change":"2024-09-14T04:05:55.695825+0000","last_active":"2024-09-14T04:11:38.991962+0000","last_peered":"2024-09-14T04:11:38.991962+0000","last_clean":"2024-09-14T04:11:38.991962+0000","last_became_active":"2024-09-14T04:05:55.680668+0000","last_became_peered":"2024-09-14T04:05:55.680668+0000","last_unstale":"2024-09-14T04:11:38.991962+0000","last_undegraded":"2024-09-14T04:11:38.991962+0000","last_fullsized":"2024-09-14T04:11:38.991962+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:30.262947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":219,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985367+0000","last_change":"2024-09-14T04:05:55.677723+0000","last_active":"2024-09-14T04:11:38.985367+0000","last_peered":"2024-09-14T04:11:38.985367+0000","last_clean":"2024-09-14T04:11:38.985367+0000","last_became_active":"2024-09-14T04:05:55.677254+0000","last_became_peered":"2024-09-14T04:05:55.677254+0000","last_unstale":"2024-09-14T04:11:38.985367+0000","last_undegraded":"2024-09-14T04:11:38.985367+0000","last_fullsized":"2024-09-14T04:11:38.985367+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:53.163120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"226'13981","reported_seq":14047,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:40.741546+0000","last_change":"2024-09-14T04:07:55.607260+0000","last_active":"2024-09-14T04:11:40.741546+0000","last_peered":"2024-09-14T04:11:40.741546+0000","last_clean":"2024-09-14T04:11:40.741546+0000","last_became_active":"2024-09-14T04:07:55.606973+0000","last_became_peered":"2024-09-14T04:07:55.606973+0000","last_unstale":"2024-09-14T04:11:40.741546+0000","last_undegraded":"2024-09-14T04:11:40.741546+0000","last_fullsized":"2024-09-14T04:11:40.741546+0000","mapping_epoch":199,"log_start":"206'3900","ondisk_log_start":"206'3900","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:59:15.715573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64290816,"num_objects":981,"num_object_clones":0,"num_object_copies":2943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13981,"num_write_kb":478784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":202,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.944744+0000","last_change":"2024-09-14T04:05:55.679201+0000","last_active":"2024-09-14T04:11:36.944744+0000","last_peered":"2024-09-14T04:11:36.944744+0000","last_clean":"2024-09-14T04:11:36.944744+0000","last_became_active":"2024-09-14T04:05:55.678843+0000","last_became_peered":"2024-09-14T04:05:55.678843+0000","last_unstale":"2024-09-14T04:11:36.944744+0000","last_undegraded":"2024-09-14T04:11:36.944744+0000","last_fullsized":"2024-09-14T04:11:36.944744+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:12:57.418514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":210,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.979242+0000","last_change":"2024-09-14T04:05:55.673650+0000","last_active":"2024-09-14T04:11:38.979242+0000","last_peered":"2024-09-14T04:11:38.979242+0000","last_clean":"2024-09-14T04:11:38.979242+0000","last_became_active":"2024-09-14T04:05:55.673258+0000","last_became_peered":"2024-09-14T04:05:55.673258+0000","last_unstale":"2024-09-14T04:11:38.979242+0000","last_undegraded":"2024-09-14T04:11:38.979242+0000","last_fullsized":"2024-09-14T04:11:38.979242+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:21.089797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"226'14167","reported_seq":14202,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:43.826043+0000","last_change":"2024-09-14T04:07:55.365272+0000","last_active":"2024-09-14T04:11:43.826043+0000","last_peered":"2024-09-14T04:11:43.826043+0000","last_clean":"2024-09-14T04:11:43.826043+0000","last_became_active":"2024-09-14T04:07:55.364722+0000","last_became_peered":"2024-09-14T04:07:55.364722+0000","last_unstale":"2024-09-14T04:11:43.826043+0000","last_undegraded":"2024-09-14T04:11:43.826043+0000","last_fullsized":"2024-09-14T04:11:43.826043+0000","mapping_epoch":199,"log_start":"209'6600","ondisk_log_start":"209'6600","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:40:55.470085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51838976,"num_objects":791,"num_object_clones":0,"num_object_copies":2373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":478656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":194,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.978627+0000","last_change":"2024-09-14T04:05:55.671329+0000","last_active":"2024-09-14T04:11:38.978627+0000","last_peered":"2024-09-14T04:11:38.978627+0000","last_clean":"2024-09-14T04:11:38.978627+0000","last_became_active":"2024-09-14T04:05:55.671032+0000","last_became_peered":"2024-09-14T04:05:55.671032+0000","last_unstale":"2024-09-14T04:11:38.978627+0000","last_undegraded":"2024-09-14T04:11:38.978627+0000","last_fullsized":"2024-09-14T04:11:38.978627+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:58:10.779452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":123,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945102+0000","last_change":"2024-09-14T04:05:55.679870+0000","last_active":"2024-09-14T04:11:36.945102+0000","last_peered":"2024-09-14T04:11:36.945102+0000","last_clean":"2024-09-14T04:11:36.945102+0000","last_became_active":"2024-09-14T04:05:55.679574+0000","last_became_peered":"2024-09-14T04:05:55.679574+0000","last_unstale":"2024-09-14T04:11:36.945102+0000","last_undegraded":"2024-09-14T04:11:36.945102+0000","last_fullsized":"2024-09-14T04:11:36.945102+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:51:09.931574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"226'14188","reported_seq":14223,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:44.180696+0000","last_change":"2024-09-14T04:07:55.367126+0000","last_active":"2024-09-14T04:11:44.180696+0000","last_peered":"2024-09-14T04:11:44.180696+0000","last_clean":"2024-09-14T04:11:44.180696+0000","last_became_active":"2024-09-14T04:07:55.366798+0000","last_became_peered":"2024-09-14T04:07:55.366798+0000","last_unstale":"2024-09-14T04:11:44.180696+0000","last_undegraded":"2024-09-14T04:11:44.180696+0000","last_fullsized":"2024-09-14T04:11:44.180696+0000","mapping_epoch":199,"log_start":"208'6000","ondisk_log_start":"208'6000","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":8188,"log_dups_size":0,"ondisk_log_size":8188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:24:51.791784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52297728,"num_objects":798,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14188,"num_write_kb":479552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":73,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.979336+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:11:38.979336+0000","last_peered":"2024-09-14T04:11:38.979336+0000","last_clean":"2024-09-14T04:11:38.979336+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:11:38.979336+0000","last_undegraded":"2024-09-14T04:11:38.979336+0000","last_fullsized":"2024-09-14T04:11:38.979336+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":303,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995725+0000","last_change":"2024-09-14T04:05:55.678520+0000","last_active":"2024-09-14T04:11:38.995725+0000","last_peered":"2024-09-14T04:11:38.995725+0000","last_clean":"2024-09-14T04:11:38.995725+0000","last_became_active":"2024-09-14T04:05:55.678008+0000","last_became_peered":"2024-09-14T04:05:55.678008+0000","last_unstale":"2024-09-14T04:11:38.995725+0000","last_undegraded":"2024-09-14T04:11:38.995725+0000","last_fullsized":"2024-09-14T04:11:38.995725+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:40:26.778212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"226'13981","reported_seq":14016,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:43.793086+0000","last_change":"2024-09-14T04:07:55.367659+0000","last_active":"2024-09-14T04:11:43.793086+0000","last_peered":"2024-09-14T04:11:43.793086+0000","last_clean":"2024-09-14T04:11:43.793086+0000","last_became_active":"2024-09-14T04:07:55.367247+0000","last_became_peered":"2024-09-14T04:07:55.367247+0000","last_unstale":"2024-09-14T04:11:43.793086+0000","last_undegraded":"2024-09-14T04:11:43.793086+0000","last_fullsized":"2024-09-14T04:11:43.793086+0000","mapping_epoch":199,"log_start":"209'6400","ondisk_log_start":"209'6400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7581,"log_dups_size":0,"ondisk_log_size":7581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:58:08.638370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51970048,"num_objects":793,"num_object_clones":0,"num_object_copies":2379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13981,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":308,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.947455+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:11:36.947455+0000","last_peered":"2024-09-14T04:11:36.947455+0000","last_clean":"2024-09-14T04:11:36.947455+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:11:36.947455+0000","last_undegraded":"2024-09-14T04:11:36.947455+0000","last_fullsized":"2024-09-14T04:11:36.947455+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":205,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.941387+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:11:36.941387+0000","last_peered":"2024-09-14T04:11:36.941387+0000","last_clean":"2024-09-14T04:11:36.941387+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:11:36.941387+0000","last_undegraded":"2024-09-14T04:11:36.941387+0000","last_fullsized":"2024-09-14T04:11:36.941387+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":73,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.980712+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:11:38.980712+0000","last_peered":"2024-09-14T04:11:38.980712+0000","last_clean":"2024-09-14T04:11:38.980712+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:11:38.980712+0000","last_undegraded":"2024-09-14T04:11:38.980712+0000","last_fullsized":"2024-09-14T04:11:38.980712+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":184,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945111+0000","last_change":"2024-09-14T04:05:55.676445+0000","last_active":"2024-09-14T04:11:36.945111+0000","last_peered":"2024-09-14T04:11:36.945111+0000","last_clean":"2024-09-14T04:11:36.945111+0000","last_became_active":"2024-09-14T04:05:55.674141+0000","last_became_peered":"2024-09-14T04:05:55.674141+0000","last_unstale":"2024-09-14T04:11:36.945111+0000","last_undegraded":"2024-09-14T04:11:36.945111+0000","last_fullsized":"2024-09-14T04:11:36.945111+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:28:50.348309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"226'14302","reported_seq":14366,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:43.664685+0000","last_change":"2024-09-14T04:07:55.366975+0000","last_active":"2024-09-14T04:11:43.664685+0000","last_peered":"2024-09-14T04:11:43.664685+0000","last_clean":"2024-09-14T04:11:43.664685+0000","last_became_active":"2024-09-14T04:07:55.366645+0000","last_became_peered":"2024-09-14T04:07:55.366645+0000","last_unstale":"2024-09-14T04:11:43.664685+0000","last_undegraded":"2024-09-14T04:11:43.664685+0000","last_fullsized":"2024-09-14T04:11:43.664685+0000","mapping_epoch":199,"log_start":"210'7300","ondisk_log_start":"210'7300","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7002,"log_dups_size":0,"ondisk_log_size":7002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:39:56.637518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51904512,"num_objects":792,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14302,"num_write_kb":483008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":220,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.947381+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:11:36.947381+0000","last_peered":"2024-09-14T04:11:36.947381+0000","last_clean":"2024-09-14T04:11:36.947381+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:11:36.947381+0000","last_undegraded":"2024-09-14T04:11:36.947381+0000","last_fullsized":"2024-09-14T04:11:36.947381+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":353,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985721+0000","last_change":"2024-09-14T04:07:44.337314+0000","last_active":"2024-09-14T04:11:38.985721+0000","last_peered":"2024-09-14T04:11:38.985721+0000","last_clean":"2024-09-14T04:11:38.985721+0000","last_became_active":"2024-09-14T04:07:44.336916+0000","last_became_peered":"2024-09-14T04:07:44.336916+0000","last_unstale":"2024-09-14T04:11:38.985721+0000","last_undegraded":"2024-09-14T04:11:38.985721+0000","last_fullsized":"2024-09-14T04:11:38.985721+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:23.153578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":342,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995338+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:11:38.995338+0000","last_peered":"2024-09-14T04:11:38.995338+0000","last_clean":"2024-09-14T04:11:38.995338+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:11:38.995338+0000","last_undegraded":"2024-09-14T04:11:38.995338+0000","last_fullsized":"2024-09-14T04:11:38.995338+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"226'14270","reported_seq":14304,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:44.349492+0000","last_change":"2024-09-14T04:07:55.365448+0000","last_active":"2024-09-14T04:11:44.349492+0000","last_peered":"2024-09-14T04:11:44.349492+0000","last_clean":"2024-09-14T04:11:44.349492+0000","last_became_active":"2024-09-14T04:07:55.365168+0000","last_became_peered":"2024-09-14T04:07:55.365168+0000","last_unstale":"2024-09-14T04:11:44.349492+0000","last_undegraded":"2024-09-14T04:11:44.349492+0000","last_fullsized":"2024-09-14T04:11:44.349492+0000","mapping_epoch":199,"log_start":"206'4200","ondisk_log_start":"206'4200","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:25.997258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50987008,"num_objects":778,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14270,"num_write_kb":481536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":336,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.978831+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:11:38.978831+0000","last_peered":"2024-09-14T04:11:38.978831+0000","last_clean":"2024-09-14T04:11:38.978831+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:11:38.978831+0000","last_undegraded":"2024-09-14T04:11:38.978831+0000","last_fullsized":"2024-09-14T04:11:38.978831+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":297,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.990304+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:11:38.990304+0000","last_peered":"2024-09-14T04:11:38.990304+0000","last_clean":"2024-09-14T04:11:38.990304+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:11:38.990304+0000","last_undegraded":"2024-09-14T04:11:38.990304+0000","last_fullsized":"2024-09-14T04:11:38.990304+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":67,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.977048+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:11:38.977048+0000","last_peered":"2024-09-14T04:11:38.977048+0000","last_clean":"2024-09-14T04:11:38.977048+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:11:38.977048+0000","last_undegraded":"2024-09-14T04:11:38.977048+0000","last_fullsized":"2024-09-14T04:11:38.977048+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"226'14434","reported_seq":14469,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:43.826447+0000","last_change":"2024-09-14T04:07:55.367851+0000","last_active":"2024-09-14T04:11:43.826447+0000","last_peered":"2024-09-14T04:11:43.826447+0000","last_clean":"2024-09-14T04:11:43.826447+0000","last_became_active":"2024-09-14T04:07:55.367146+0000","last_became_peered":"2024-09-14T04:07:55.367146+0000","last_unstale":"2024-09-14T04:11:43.826447+0000","last_undegraded":"2024-09-14T04:11:43.826447+0000","last_fullsized":"2024-09-14T04:11:43.826447+0000","mapping_epoch":199,"log_start":"209'6900","ondisk_log_start":"209'6900","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7534,"log_dups_size":0,"ondisk_log_size":7534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:26:46.495508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53739520,"num_objects":820,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14434,"num_write_kb":488128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":206,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.941243+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:11:36.941243+0000","last_peered":"2024-09-14T04:11:36.941243+0000","last_clean":"2024-09-14T04:11:36.941243+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:11:36.941243+0000","last_undegraded":"2024-09-14T04:11:36.941243+0000","last_fullsized":"2024-09-14T04:11:36.941243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":165,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985284+0000","last_change":"2024-09-14T04:05:55.676693+0000","last_active":"2024-09-14T04:11:38.985284+0000","last_peered":"2024-09-14T04:11:38.985284+0000","last_clean":"2024-09-14T04:11:38.985284+0000","last_became_active":"2024-09-14T04:05:55.676335+0000","last_became_peered":"2024-09-14T04:05:55.676335+0000","last_unstale":"2024-09-14T04:11:38.985284+0000","last_undegraded":"2024-09-14T04:11:38.985284+0000","last_fullsized":"2024-09-14T04:11:38.985284+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:19:43.626016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":322,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.994435+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:11:38.994435+0000","last_peered":"2024-09-14T04:11:38.994435+0000","last_clean":"2024-09-14T04:11:38.994435+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:11:38.994435+0000","last_undegraded":"2024-09-14T04:11:38.994435+0000","last_fullsized":"2024-09-14T04:11:38.994435+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"226'14141","reported_seq":14176,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:44.195151+0000","last_change":"2024-09-14T04:07:55.607796+0000","last_active":"2024-09-14T04:11:44.195151+0000","last_peered":"2024-09-14T04:11:44.195151+0000","last_clean":"2024-09-14T04:11:44.195151+0000","last_became_active":"2024-09-14T04:07:55.607447+0000","last_became_peered":"2024-09-14T04:07:55.607447+0000","last_unstale":"2024-09-14T04:11:44.195151+0000","last_undegraded":"2024-09-14T04:11:44.195151+0000","last_fullsized":"2024-09-14T04:11:44.195151+0000","mapping_epoch":199,"log_start":"208'6000","ondisk_log_start":"208'6000","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:09:44.307466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53673984,"num_objects":819,"num_object_clones":0,"num_object_copies":2457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14141,"num_write_kb":478720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":229,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.941273+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:11:36.941273+0000","last_peered":"2024-09-14T04:11:36.941273+0000","last_clean":"2024-09-14T04:11:36.941273+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:11:36.941273+0000","last_undegraded":"2024-09-14T04:11:36.941273+0000","last_fullsized":"2024-09-14T04:11:36.941273+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":336,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.994522+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:11:38.994522+0000","last_peered":"2024-09-14T04:11:38.994522+0000","last_clean":"2024-09-14T04:11:38.994522+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:11:38.994522+0000","last_undegraded":"2024-09-14T04:11:38.994522+0000","last_fullsized":"2024-09-14T04:11:38.994522+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":293,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.993123+0000","last_change":"2024-09-14T04:05:55.674511+0000","last_active":"2024-09-14T04:11:38.993123+0000","last_peered":"2024-09-14T04:11:38.993123+0000","last_clean":"2024-09-14T04:11:38.993123+0000","last_became_active":"2024-09-14T04:05:55.674182+0000","last_became_peered":"2024-09-14T04:05:55.674182+0000","last_unstale":"2024-09-14T04:11:38.993123+0000","last_undegraded":"2024-09-14T04:11:38.993123+0000","last_fullsized":"2024-09-14T04:11:38.993123+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:12:34.453647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"226'14277","reported_seq":14313,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:43.774418+0000","last_change":"2024-09-14T04:07:55.607679+0000","last_active":"2024-09-14T04:11:43.774418+0000","last_peered":"2024-09-14T04:11:43.774418+0000","last_clean":"2024-09-14T04:11:43.774418+0000","last_became_active":"2024-09-14T04:07:55.607338+0000","last_became_peered":"2024-09-14T04:07:55.607338+0000","last_unstale":"2024-09-14T04:11:43.774418+0000","last_undegraded":"2024-09-14T04:11:43.774418+0000","last_fullsized":"2024-09-14T04:11:43.774418+0000","mapping_epoch":199,"log_start":"209'6700","ondisk_log_start":"209'6700","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7577,"log_dups_size":0,"ondisk_log_size":7577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:02:20.945987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51773464,"num_objects":791,"num_object_clones":0,"num_object_copies":2373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14277,"num_write_kb":482113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":297,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.994656+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:11:38.994656+0000","last_peered":"2024-09-14T04:11:38.994656+0000","last_clean":"2024-09-14T04:11:38.994656+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:11:38.994656+0000","last_undegraded":"2024-09-14T04:11:38.994656+0000","last_fullsized":"2024-09-14T04:11:38.994656+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":213,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.947916+0000","last_change":"2024-09-14T04:05:55.681238+0000","last_active":"2024-09-14T04:11:36.947916+0000","last_peered":"2024-09-14T04:11:36.947916+0000","last_clean":"2024-09-14T04:11:36.947916+0000","last_became_active":"2024-09-14T04:05:55.680946+0000","last_became_peered":"2024-09-14T04:05:55.680946+0000","last_unstale":"2024-09-14T04:11:36.947916+0000","last_undegraded":"2024-09-14T04:11:36.947916+0000","last_fullsized":"2024-09-14T04:11:36.947916+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:13:39.006545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":321,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985058+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:11:38.985058+0000","last_peered":"2024-09-14T04:11:38.985058+0000","last_clean":"2024-09-14T04:11:38.985058+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:11:38.985058+0000","last_undegraded":"2024-09-14T04:11:38.985058+0000","last_fullsized":"2024-09-14T04:11:38.985058+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"226'14243","reported_seq":14278,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:43.801562+0000","last_change":"2024-09-14T04:07:55.366077+0000","last_active":"2024-09-14T04:11:43.801562+0000","last_peered":"2024-09-14T04:11:43.801562+0000","last_clean":"2024-09-14T04:11:43.801562+0000","last_became_active":"2024-09-14T04:07:55.365753+0000","last_became_peered":"2024-09-14T04:07:55.365753+0000","last_unstale":"2024-09-14T04:11:43.801562+0000","last_undegraded":"2024-09-14T04:11:43.801562+0000","last_fullsized":"2024-09-14T04:11:43.801562+0000","mapping_epoch":199,"log_start":"209'6700","ondisk_log_start":"209'6700","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:04:18.784725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51445760,"num_objects":785,"num_object_clones":0,"num_object_copies":2355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14243,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":289,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.989324+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:11:38.989324+0000","last_peered":"2024-09-14T04:11:38.989324+0000","last_clean":"2024-09-14T04:11:38.989324+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:11:38.989324+0000","last_undegraded":"2024-09-14T04:11:38.989324+0000","last_fullsized":"2024-09-14T04:11:38.989324+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":304,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995891+0000","last_change":"2024-09-14T04:05:55.676617+0000","last_active":"2024-09-14T04:11:38.995891+0000","last_peered":"2024-09-14T04:11:38.995891+0000","last_clean":"2024-09-14T04:11:38.995891+0000","last_became_active":"2024-09-14T04:05:55.676318+0000","last_became_peered":"2024-09-14T04:05:55.676318+0000","last_unstale":"2024-09-14T04:11:38.995891+0000","last_undegraded":"2024-09-14T04:11:38.995891+0000","last_fullsized":"2024-09-14T04:11:38.995891+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:38.529611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.941254+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:11:36.941254+0000","last_peered":"2024-09-14T04:11:36.941254+0000","last_clean":"2024-09-14T04:11:36.941254+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:11:36.941254+0000","last_undegraded":"2024-09-14T04:11:36.941254+0000","last_fullsized":"2024-09-14T04:11:36.941254+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.f","version":"226'14114","reported_seq":14149,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:43.801490+0000","last_change":"2024-09-14T04:07:55.607496+0000","last_active":"2024-09-14T04:11:43.801490+0000","last_peered":"2024-09-14T04:11:43.801490+0000","last_clean":"2024-09-14T04:11:43.801490+0000","last_became_active":"2024-09-14T04:07:55.607124+0000","last_became_peered":"2024-09-14T04:07:55.607124+0000","last_unstale":"2024-09-14T04:11:43.801490+0000","last_undegraded":"2024-09-14T04:11:43.801490+0000","last_fullsized":"2024-09-14T04:11:43.801490+0000","mapping_epoch":199,"log_start":"209'6600","ondisk_log_start":"209'6600","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7514,"log_dups_size":0,"ondisk_log_size":7514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:15:10.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52953088,"num_objects":808,"num_object_clones":0,"num_object_copies":2424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14114,"num_write_kb":477504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":345,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995718+0000","last_change":"2024-09-14T04:05:55.690010+0000","last_active":"2024-09-14T04:11:38.995718+0000","last_peered":"2024-09-14T04:11:38.995718+0000","last_clean":"2024-09-14T04:11:38.995718+0000","last_became_active":"2024-09-14T04:05:55.670892+0000","last_became_peered":"2024-09-14T04:05:55.670892+0000","last_unstale":"2024-09-14T04:11:38.995718+0000","last_undegraded":"2024-09-14T04:11:38.995718+0000","last_fullsized":"2024-09-14T04:11:38.995718+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:40:33.024617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945129+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:11:36.945129+0000","last_peered":"2024-09-14T04:11:36.945129+0000","last_clean":"2024-09-14T04:11:36.945129+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:11:36.945129+0000","last_undegraded":"2024-09-14T04:11:36.945129+0000","last_fullsized":"2024-09-14T04:11:36.945129+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":69,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.940997+0000","last_change":"2024-09-14T04:05:55.674629+0000","last_active":"2024-09-14T04:11:36.940997+0000","last_peered":"2024-09-14T04:11:36.940997+0000","last_clean":"2024-09-14T04:11:36.940997+0000","last_became_active":"2024-09-14T04:05:55.674242+0000","last_became_peered":"2024-09-14T04:05:55.674242+0000","last_unstale":"2024-09-14T04:11:36.940997+0000","last_undegraded":"2024-09-14T04:11:36.940997+0000","last_fullsized":"2024-09-14T04:11:36.940997+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:06:16.827114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":82,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.944941+0000","last_change":"2024-09-14T04:05:55.677636+0000","last_active":"2024-09-14T04:11:36.944941+0000","last_peered":"2024-09-14T04:11:36.944941+0000","last_clean":"2024-09-14T04:11:36.944941+0000","last_became_active":"2024-09-14T04:05:55.677313+0000","last_became_peered":"2024-09-14T04:05:55.677313+0000","last_unstale":"2024-09-14T04:11:36.944941+0000","last_undegraded":"2024-09-14T04:11:36.944941+0000","last_fullsized":"2024-09-14T04:11:36.944941+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:00:13.489641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":122,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.947688+0000","last_change":"2024-09-14T04:05:55.686083+0000","last_active":"2024-09-14T04:11:36.947688+0000","last_peered":"2024-09-14T04:11:36.947688+0000","last_clean":"2024-09-14T04:11:36.947688+0000","last_became_active":"2024-09-14T04:05:55.673710+0000","last_became_peered":"2024-09-14T04:05:55.673710+0000","last_unstale":"2024-09-14T04:11:36.947688+0000","last_undegraded":"2024-09-14T04:11:36.947688+0000","last_fullsized":"2024-09-14T04:11:36.947688+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:25.169218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":296,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.992977+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:11:38.992977+0000","last_peered":"2024-09-14T04:11:38.992977+0000","last_clean":"2024-09-14T04:11:38.992977+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:11:38.992977+0000","last_undegraded":"2024-09-14T04:11:38.992977+0000","last_fullsized":"2024-09-14T04:11:38.992977+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":210,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.947842+0000","last_change":"2024-09-14T04:05:55.698766+0000","last_active":"2024-09-14T04:11:36.947842+0000","last_peered":"2024-09-14T04:11:36.947842+0000","last_clean":"2024-09-14T04:11:36.947842+0000","last_became_active":"2024-09-14T04:05:55.674923+0000","last_became_peered":"2024-09-14T04:05:55.674923+0000","last_unstale":"2024-09-14T04:11:36.947842+0000","last_undegraded":"2024-09-14T04:11:36.947842+0000","last_fullsized":"2024-09-14T04:11:36.947842+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:52:37.679728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":192,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.941165+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:11:36.941165+0000","last_peered":"2024-09-14T04:11:36.941165+0000","last_clean":"2024-09-14T04:11:36.941165+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:11:36.941165+0000","last_undegraded":"2024-09-14T04:11:36.941165+0000","last_fullsized":"2024-09-14T04:11:36.941165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":248,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.979667+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:11:38.979667+0000","last_peered":"2024-09-14T04:11:38.979667+0000","last_clean":"2024-09-14T04:11:38.979667+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:11:38.979667+0000","last_undegraded":"2024-09-14T04:11:38.979667+0000","last_fullsized":"2024-09-14T04:11:38.979667+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":344,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.998959+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:11:38.998959+0000","last_peered":"2024-09-14T04:11:38.998959+0000","last_clean":"2024-09-14T04:11:38.998959+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:11:38.998959+0000","last_undegraded":"2024-09-14T04:11:38.998959+0000","last_fullsized":"2024-09-14T04:11:38.998959+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":260,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.979410+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:11:38.979410+0000","last_peered":"2024-09-14T04:11:38.979410+0000","last_clean":"2024-09-14T04:11:38.979410+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:11:38.979410+0000","last_undegraded":"2024-09-14T04:11:38.979410+0000","last_fullsized":"2024-09-14T04:11:38.979410+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":201,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945065+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:11:36.945065+0000","last_peered":"2024-09-14T04:11:36.945065+0000","last_clean":"2024-09-14T04:11:36.945065+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:11:36.945065+0000","last_undegraded":"2024-09-14T04:11:36.945065+0000","last_fullsized":"2024-09-14T04:11:36.945065+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":73,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.979248+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:11:38.979248+0000","last_peered":"2024-09-14T04:11:38.979248+0000","last_clean":"2024-09-14T04:11:38.979248+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:11:38.979248+0000","last_undegraded":"2024-09-14T04:11:38.979248+0000","last_fullsized":"2024-09-14T04:11:38.979248+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":346,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995529+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:11:38.995529+0000","last_peered":"2024-09-14T04:11:38.995529+0000","last_clean":"2024-09-14T04:11:38.995529+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:11:38.995529+0000","last_undegraded":"2024-09-14T04:11:38.995529+0000","last_fullsized":"2024-09-14T04:11:38.995529+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":303,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.991707+0000","last_change":"2024-09-14T04:05:55.677962+0000","last_active":"2024-09-14T04:11:38.991707+0000","last_peered":"2024-09-14T04:11:38.991707+0000","last_clean":"2024-09-14T04:11:38.991707+0000","last_became_active":"2024-09-14T04:05:55.677277+0000","last_became_peered":"2024-09-14T04:05:55.677277+0000","last_unstale":"2024-09-14T04:11:38.991707+0000","last_undegraded":"2024-09-14T04:11:38.991707+0000","last_fullsized":"2024-09-14T04:11:38.991707+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:24.908568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":73,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.982921+0000","last_change":"2024-09-14T04:07:44.338546+0000","last_active":"2024-09-14T04:11:38.982921+0000","last_peered":"2024-09-14T04:11:38.982921+0000","last_clean":"2024-09-14T04:11:38.982921+0000","last_became_active":"2024-09-14T04:07:44.338210+0000","last_became_peered":"2024-09-14T04:07:44.338210+0000","last_unstale":"2024-09-14T04:11:38.982921+0000","last_undegraded":"2024-09-14T04:11:38.982921+0000","last_fullsized":"2024-09-14T04:11:38.982921+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:56.100013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":127,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.993200+0000","last_change":"2024-09-14T04:05:55.671091+0000","last_active":"2024-09-14T04:11:38.993200+0000","last_peered":"2024-09-14T04:11:38.993200+0000","last_clean":"2024-09-14T04:11:38.993200+0000","last_became_active":"2024-09-14T04:05:55.670180+0000","last_became_peered":"2024-09-14T04:05:55.670180+0000","last_unstale":"2024-09-14T04:11:38.993200+0000","last_undegraded":"2024-09-14T04:11:38.993200+0000","last_fullsized":"2024-09-14T04:11:38.993200+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:33.489272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":339,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.977321+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:11:38.977321+0000","last_peered":"2024-09-14T04:11:38.977321+0000","last_clean":"2024-09-14T04:11:38.977321+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:11:38.977321+0000","last_undegraded":"2024-09-14T04:11:38.977321+0000","last_fullsized":"2024-09-14T04:11:38.977321+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":329,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985133+0000","last_change":"2024-09-14T04:05:55.671536+0000","last_active":"2024-09-14T04:11:38.985133+0000","last_peered":"2024-09-14T04:11:38.985133+0000","last_clean":"2024-09-14T04:11:38.985133+0000","last_became_active":"2024-09-14T04:05:55.671029+0000","last_became_peered":"2024-09-14T04:05:55.671029+0000","last_unstale":"2024-09-14T04:11:38.985133+0000","last_undegraded":"2024-09-14T04:11:38.985133+0000","last_fullsized":"2024-09-14T04:11:38.985133+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:35.745580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":236,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.991910+0000","last_change":"2024-09-14T04:05:55.681242+0000","last_active":"2024-09-14T04:11:38.991910+0000","last_peered":"2024-09-14T04:11:38.991910+0000","last_clean":"2024-09-14T04:11:38.991910+0000","last_became_active":"2024-09-14T04:05:55.680502+0000","last_became_peered":"2024-09-14T04:05:55.680502+0000","last_unstale":"2024-09-14T04:11:38.991910+0000","last_undegraded":"2024-09-14T04:11:38.991910+0000","last_fullsized":"2024-09-14T04:11:38.991910+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:23.760294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":208,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.940962+0000","last_change":"2024-09-14T04:05:55.687662+0000","last_active":"2024-09-14T04:11:36.940962+0000","last_peered":"2024-09-14T04:11:36.940962+0000","last_clean":"2024-09-14T04:11:36.940962+0000","last_became_active":"2024-09-14T04:05:55.673561+0000","last_became_peered":"2024-09-14T04:05:55.673561+0000","last_unstale":"2024-09-14T04:11:36.940962+0000","last_undegraded":"2024-09-14T04:11:36.940962+0000","last_fullsized":"2024-09-14T04:11:36.940962+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:14:51.708356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":236,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981659+0000","last_change":"2024-09-14T04:05:55.679885+0000","last_active":"2024-09-14T04:11:38.981659+0000","last_peered":"2024-09-14T04:11:38.981659+0000","last_clean":"2024-09-14T04:11:38.981659+0000","last_became_active":"2024-09-14T04:05:55.678589+0000","last_became_peered":"2024-09-14T04:05:55.678589+0000","last_unstale":"2024-09-14T04:11:38.981659+0000","last_undegraded":"2024-09-14T04:11:38.981659+0000","last_fullsized":"2024-09-14T04:11:38.981659+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:51:32.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":191,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.941193+0000","last_change":"2024-09-14T04:05:55.697832+0000","last_active":"2024-09-14T04:11:36.941193+0000","last_peered":"2024-09-14T04:11:36.941193+0000","last_clean":"2024-09-14T04:11:36.941193+0000","last_became_active":"2024-09-14T04:05:55.673923+0000","last_became_peered":"2024-09-14T04:05:55.673923+0000","last_unstale":"2024-09-14T04:11:36.941193+0000","last_undegraded":"2024-09-14T04:11:36.941193+0000","last_fullsized":"2024-09-14T04:11:36.941193+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:44.697302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":73,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.979600+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:11:38.979600+0000","last_peered":"2024-09-14T04:11:38.979600+0000","last_clean":"2024-09-14T04:11:38.979600+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:11:38.979600+0000","last_undegraded":"2024-09-14T04:11:38.979600+0000","last_fullsized":"2024-09-14T04:11:38.979600+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":74,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.980055+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:11:38.980055+0000","last_peered":"2024-09-14T04:11:38.980055+0000","last_clean":"2024-09-14T04:11:38.980055+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:11:38.980055+0000","last_undegraded":"2024-09-14T04:11:38.980055+0000","last_fullsized":"2024-09-14T04:11:38.980055+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":143,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945886+0000","last_change":"2024-09-14T04:07:44.338957+0000","last_active":"2024-09-14T04:11:36.945886+0000","last_peered":"2024-09-14T04:11:36.945886+0000","last_clean":"2024-09-14T04:11:36.945886+0000","last_became_active":"2024-09-14T04:07:44.338190+0000","last_became_peered":"2024-09-14T04:07:44.338190+0000","last_unstale":"2024-09-14T04:11:36.945886+0000","last_undegraded":"2024-09-14T04:11:36.945886+0000","last_fullsized":"2024-09-14T04:11:36.945886+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:36:27.488115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":335,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.998644+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:11:38.998644+0000","last_peered":"2024-09-14T04:11:38.998644+0000","last_clean":"2024-09-14T04:11:38.998644+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:11:38.998644+0000","last_undegraded":"2024-09-14T04:11:38.998644+0000","last_fullsized":"2024-09-14T04:11:38.998644+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":869531672,"num_objects":13269,"num_object_clones":0,"num_object_copies":39807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13269,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227119,"num_write_kb":7692353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2595303424,"data_stored":2595291208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130019,"ondisk_log_size":130019,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4890,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430576,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1157664,"kb_used_data":258712,"kb_used_omap":1,"kb_used_meta":898878,"kb_avail":92587488,"statfs":{"total":95995035648,"available":94809587712,"internally_reserved":0,"allocated":264921088,"data_stored":263860367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920451416},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626805,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1272412,"kb_used_data":370900,"kb_used_omap":1,"kb_used_meta":901502,"kb_avail":92472740,"statfs":{"total":95995035648,"available":94692085760,"internally_reserved":0,"allocated":379801600,"data_stored":378744591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923138392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823033,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1225220,"kb_used_data":315772,"kb_used_omap":1,"kb_used_meta":909438,"kb_avail":92519932,"statfs":{"total":95995035648,"available":94740410368,"internally_reserved":0,"allocated":323350528,"data_stored":322313004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931264856},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":798863917115,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552414,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":909256,"kb_used_data":406848,"kb_used_omap":4,"kb_used_meta":502395,"kb_avail":92835896,"statfs":{"total":95995035648,"available":95063957504,"internally_reserved":0,"allocated":416612352,"data_stored":415570837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":514452925},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435455,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":809680,"kb_used_data":373828,"kb_used_omap":6,"kb_used_meta":435833,"kb_avail":92935472,"statfs":{"total":95995035648,"available":95165923328,"internally_reserved":0,"allocated":382799872,"data_stored":381758033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":446293453},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827925,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957576,"kb_used_data":334464,"kb_used_omap":5,"kb_used_meta":623098,"kb_avail":92787576,"statfs":{"total":95995035648,"available":95014477824,"internally_reserved":0,"allocated":342491136,"data_stored":341449548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":638052591},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383913,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1254544,"kb_used_data":538376,"kb_used_omap":5,"kb_used_meta":716090,"kb_avail":92490608,"statfs":{"total":95995035648,"available":94710382592,"internally_reserved":0,"allocated":551297024,"data_stored":550248154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":733276220},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":542576640,"data_stored":542572568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":337903616,"data_stored":337903616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375324672,"data_stored":375324672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407900160,"data_stored":407896088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":307167232,"data_stored":307167232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":366743552,"data_stored":366739480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":257687552,"data_stored":257687552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:11:46.986 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:11:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:46.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:46.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:47.080 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:46 smithi160 ceph-mon[88407]: from='client.46136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:11:47.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:46 smithi114 ceph-mon[103029]: from='client.46136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:11:47.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:46 smithi114 ceph-mon[106969]: from='client.46136 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:11:47.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:47.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:47.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-14T04:11:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:47.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:47.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:47.264 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:47.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:47.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:47.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:47.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:47.488 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:11:47.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:47.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:47.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:47.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:47.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:47.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:47.705 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:47.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:47.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:47.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:47.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:47.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:47.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:48.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:48.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:48.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:48.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:48 smithi114 ceph-mon[106969]: pgmap v840: 89 pgs: 89 active+clean; 799 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 797 op/s 2024-09-14T04:11:48.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:48 smithi114 ceph-mon[103029]: pgmap v840: 89 pgs: 89 active+clean; 799 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 797 op/s 2024-09-14T04:11:48.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:48.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:48.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:48.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:48.307 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:48.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:48 smithi160 ceph-mon[88407]: pgmap v840: 89 pgs: 89 active+clean; 799 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 797 op/s 2024-09-14T04:11:48.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:48.447 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:48.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:48.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:48.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:48.609 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:48.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:48.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:48.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:48.751 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:48.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:48.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:48.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:48.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:48.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:49.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:11:49.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:49 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3141219396' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:11:49.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:49 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:11:49.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:49 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3141219396' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:11:49.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:49 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:11:49.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:49 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3141219396' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:11:49.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:49 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:11:49.160 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:49.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:49.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:49.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:49.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:49.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:49.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:49.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:49.449 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:11:49.449 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:11:49.520 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:49.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:49.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:49.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:49.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:49.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:49.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:49.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:49.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:49.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:49.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:49.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:50.061 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:50.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:50.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:50.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:50.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:50 smithi114 ceph-mon[103029]: pgmap v841: 89 pgs: 89 active+clean; 753 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:11:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:50 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:11:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:50 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:11:50.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:50 smithi114 ceph-mon[103029]: osdmap e227: 8 total, 8 up, 7 in 2024-09-14T04:11:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:50 smithi114 ceph-mon[106969]: pgmap v841: 89 pgs: 89 active+clean; 753 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:11:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:50 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:11:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:50 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:11:50.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:50 smithi114 ceph-mon[106969]: osdmap e227: 8 total, 8 up, 7 in 2024-09-14T04:11:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:50 smithi160 ceph-mon[88407]: pgmap v841: 89 pgs: 89 active+clean; 753 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-09-14T04:11:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:50 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:11:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:50 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:11:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:50 smithi160 ceph-mon[88407]: osdmap e227: 8 total, 8 up, 7 in 2024-09-14T04:11:50.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:50.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-14T04:11:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:50.236 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:50.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:50.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:50.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:50.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:50.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:50.478 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:50.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:50.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:50.531 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:11:50.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:11:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:11:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:11:50.667 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:50.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:50.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:50.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:50.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:50.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:50.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:50.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:50.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:51.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:51.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:51.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:51.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:51.210 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:51.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:51.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:51.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:51.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:51.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:51.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:51.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:51.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:51.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:51.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:51.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:51.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:51.759 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:51.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:51.791 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:11:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:11:51] "GET /metrics HTTP/1.1" 200 36482 "" "Prometheus/2.51.0" 2024-09-14T04:11:51.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:51.906 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:51.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:52.008 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:52.042 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:52.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:52.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:52 smithi114 ceph-mon[103029]: pgmap v843: 89 pgs: 89 active+clean; 552 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-14T04:11:52.123 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:52 smithi114 ceph-mon[106969]: pgmap v843: 89 pgs: 89 active+clean; 552 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-14T04:11:52.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:52.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:52.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:52.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:52.290 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:52 smithi160 ceph-mon[88407]: pgmap v843: 89 pgs: 89 active+clean; 552 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-14T04:11:52.296 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:52.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:52.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:52.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:52.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:52.524 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:11:52.525 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:11:52.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:52.564 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:11:52.564 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:11:52.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:52.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:52.575 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:11:52.575 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:11:52.611 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:11:52.611 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:11:52.651 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:11:52.651 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:11:52.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:52.690 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:11:52.691 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:11:52.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:52.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:52.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:52.721 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:11:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:11:52.753 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:11:52.753 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:11:52.781 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:11:52.781 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:11:52.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:52.814 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:11:52.814 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:11:52.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:52.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:52.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:52.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:52.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:53.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:53.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:53.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:53.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:53.214 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:53.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:53.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:53.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:53.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/409493055' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:53.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/409493055' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:53.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/409493055' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:11:53.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:11:53.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:53.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:53.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:53.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:11:53.520 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-14T04:11:53.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:53.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:53.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:53.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:53.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:53.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:53.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:53.896 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:11:53.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:53.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:54.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:54.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:54.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:54.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:54.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:54.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:54.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:54.308 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:54.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:54.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:54.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:54 smithi114 ceph-mon[103029]: from='client.46145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:11:54.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:54 smithi114 ceph-mon[103029]: pgmap v844: 89 pgs: 89 active+clean; 552 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-14T04:11:54.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:54 smithi114 ceph-mon[106969]: from='client.46145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:11:54.356 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:54 smithi114 ceph-mon[106969]: pgmap v844: 89 pgs: 89 active+clean; 552 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-14T04:11:54.367 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:54 smithi160 ceph-mon[88407]: from='client.46145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:11:54.367 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:54 smithi160 ceph-mon[88407]: pgmap v844: 89 pgs: 89 active+clean; 552 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-14T04:11:54.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:54.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:54.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:54.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:54.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:54.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:54.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:54.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:54.706 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:54.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:54.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:54.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:54.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:54.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:54.848 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":843,"stamp":"2024-09-14T04:11:50.833768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":578426853,"num_objects":8844,"num_object_clones":0,"num_object_copies":26532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8844,"num_whiteouts":0,"num_read":307,"num_read_kb":760,"num_write":231782,"num_write_kb":7694012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":595258,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130228,"ondisk_log_size":130228,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6737612,"kb_used_data":1745932,"kb_used_omap":27,"kb_used_meta":4991396,"kb_avail":649478452,"statfs":{"total":671965249536,"available":665065934848,"internally_reserved":0,"allocated":1787834368,"data_stored":1780497332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28483,"internal_metadata":5111189693},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-523698176,"num_objects":-7991,"num_object_clones":0,"num_object_copies":-23973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7991,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004972"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":262,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.059207+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:11:49.059207+0000","last_peered":"2024-09-14T04:11:49.059207+0000","last_clean":"2024-09-14T04:11:49.059207+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:11:49.059207+0000","last_undegraded":"2024-09-14T04:11:49.059207+0000","last_fullsized":"2024-09-14T04:11:49.059207+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":210,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981181+0000","last_change":"2024-09-14T04:05:55.701296+0000","last_active":"2024-09-14T04:11:38.981181+0000","last_peered":"2024-09-14T04:11:38.981181+0000","last_clean":"2024-09-14T04:11:38.981181+0000","last_became_active":"2024-09-14T04:05:55.676634+0000","last_became_peered":"2024-09-14T04:05:55.676634+0000","last_unstale":"2024-09-14T04:11:38.981181+0000","last_undegraded":"2024-09-14T04:11:38.981181+0000","last_fullsized":"2024-09-14T04:11:38.981181+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:44.187804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":192,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.970004+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:11:38.970004+0000","last_peered":"2024-09-14T04:11:38.970004+0000","last_clean":"2024-09-14T04:11:38.970004+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:11:38.970004+0000","last_undegraded":"2024-09-14T04:11:38.970004+0000","last_fullsized":"2024-09-14T04:11:38.970004+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":345,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985302+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:11:38.985302+0000","last_peered":"2024-09-14T04:11:38.985302+0000","last_clean":"2024-09-14T04:11:38.985302+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:11:38.985302+0000","last_undegraded":"2024-09-14T04:11:38.985302+0000","last_fullsized":"2024-09-14T04:11:38.985302+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":331,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995669+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:11:38.995669+0000","last_peered":"2024-09-14T04:11:38.995669+0000","last_clean":"2024-09-14T04:11:38.995669+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:11:38.995669+0000","last_undegraded":"2024-09-14T04:11:38.995669+0000","last_fullsized":"2024-09-14T04:11:38.995669+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":185,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981049+0000","last_change":"2024-09-14T04:05:55.676850+0000","last_active":"2024-09-14T04:11:38.981049+0000","last_peered":"2024-09-14T04:11:38.981049+0000","last_clean":"2024-09-14T04:11:38.981049+0000","last_became_active":"2024-09-14T04:05:55.676337+0000","last_became_peered":"2024-09-14T04:05:55.676337+0000","last_unstale":"2024-09-14T04:11:38.981049+0000","last_undegraded":"2024-09-14T04:11:38.981049+0000","last_fullsized":"2024-09-14T04:11:38.981049+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:12.377665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.058042+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:11:49.058042+0000","last_peered":"2024-09-14T04:11:49.058042+0000","last_clean":"2024-09-14T04:11:49.058042+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:11:49.058042+0000","last_undegraded":"2024-09-14T04:11:49.058042+0000","last_fullsized":"2024-09-14T04:11:49.058042+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":200,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945969+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:11:36.945969+0000","last_peered":"2024-09-14T04:11:36.945969+0000","last_clean":"2024-09-14T04:11:36.945969+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:11:36.945969+0000","last_undegraded":"2024-09-14T04:11:36.945969+0000","last_fullsized":"2024-09-14T04:11:36.945969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"227'14432","reported_seq":14468,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.310901+0000","last_change":"2024-09-14T04:07:55.368334+0000","last_active":"2024-09-14T04:11:49.310901+0000","last_peered":"2024-09-14T04:11:49.310901+0000","last_clean":"2024-09-14T04:11:49.310901+0000","last_became_active":"2024-09-14T04:07:55.368016+0000","last_became_peered":"2024-09-14T04:07:55.368016+0000","last_unstale":"2024-09-14T04:11:49.310901+0000","last_undegraded":"2024-09-14T04:11:49.310901+0000","last_fullsized":"2024-09-14T04:11:49.310901+0000","mapping_epoch":199,"log_start":"207'4400","ondisk_log_start":"207'4400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:13.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34603008,"num_objects":528,"num_object_clones":0,"num_object_copies":1584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14432,"num_write_kb":478720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":207,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981931+0000","last_change":"2024-09-14T04:05:55.704897+0000","last_active":"2024-09-14T04:11:38.981931+0000","last_peered":"2024-09-14T04:11:38.981931+0000","last_clean":"2024-09-14T04:11:38.981931+0000","last_became_active":"2024-09-14T04:05:55.681093+0000","last_became_peered":"2024-09-14T04:05:55.681093+0000","last_unstale":"2024-09-14T04:11:38.981931+0000","last_undegraded":"2024-09-14T04:11:38.981931+0000","last_fullsized":"2024-09-14T04:11:38.981931+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:49.647801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":175,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969564+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:11:38.969564+0000","last_peered":"2024-09-14T04:11:38.969564+0000","last_clean":"2024-09-14T04:11:38.969564+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:11:38.969564+0000","last_undegraded":"2024-09-14T04:11:38.969564+0000","last_fullsized":"2024-09-14T04:11:38.969564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"226'14642","reported_seq":14677,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:48.816097+0000","last_change":"2024-09-14T04:07:55.607517+0000","last_active":"2024-09-14T04:11:48.816097+0000","last_peered":"2024-09-14T04:11:48.816097+0000","last_clean":"2024-09-14T04:11:48.816097+0000","last_became_active":"2024-09-14T04:07:55.607208+0000","last_became_peered":"2024-09-14T04:07:55.607208+0000","last_unstale":"2024-09-14T04:11:48.816097+0000","last_undegraded":"2024-09-14T04:11:48.816097+0000","last_fullsized":"2024-09-14T04:11:48.816097+0000","mapping_epoch":199,"log_start":"210'7100","ondisk_log_start":"210'7100","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:32:20.123209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14642,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":208,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981739+0000","last_change":"2024-09-14T04:05:55.702981+0000","last_active":"2024-09-14T04:11:38.981739+0000","last_peered":"2024-09-14T04:11:38.981739+0000","last_clean":"2024-09-14T04:11:38.981739+0000","last_became_active":"2024-09-14T04:05:55.679789+0000","last_became_peered":"2024-09-14T04:05:55.679789+0000","last_unstale":"2024-09-14T04:11:38.981739+0000","last_undegraded":"2024-09-14T04:11:38.981739+0000","last_fullsized":"2024-09-14T04:11:38.981739+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:34:24.358644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":250,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.991747+0000","last_change":"2024-09-14T04:07:44.340689+0000","last_active":"2024-09-14T04:11:38.991747+0000","last_peered":"2024-09-14T04:11:38.991747+0000","last_clean":"2024-09-14T04:11:38.991747+0000","last_became_active":"2024-09-14T04:07:44.340112+0000","last_became_peered":"2024-09-14T04:07:44.340112+0000","last_unstale":"2024-09-14T04:11:38.991747+0000","last_undegraded":"2024-09-14T04:11:38.991747+0000","last_fullsized":"2024-09-14T04:11:38.991747+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:04.263233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"226'14372","reported_seq":14438,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:48.642891+0000","last_change":"2024-09-14T04:07:55.608111+0000","last_active":"2024-09-14T04:11:48.642891+0000","last_peered":"2024-09-14T04:11:48.642891+0000","last_clean":"2024-09-14T04:11:48.642891+0000","last_became_active":"2024-09-14T04:07:55.607646+0000","last_became_peered":"2024-09-14T04:07:55.607646+0000","last_unstale":"2024-09-14T04:11:48.642891+0000","last_undegraded":"2024-09-14T04:11:48.642891+0000","last_fullsized":"2024-09-14T04:11:48.642891+0000","mapping_epoch":199,"log_start":"210'7300","ondisk_log_start":"210'7300","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7072,"log_dups_size":0,"ondisk_log_size":7072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:02:33.076539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748736,"num_objects":576,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14372,"num_write_kb":478336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":176,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981529+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:11:38.981529+0000","last_peered":"2024-09-14T04:11:38.981529+0000","last_clean":"2024-09-14T04:11:38.981529+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:11:38.981529+0000","last_undegraded":"2024-09-14T04:11:38.981529+0000","last_fullsized":"2024-09-14T04:11:38.981529+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":201,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.944606+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:11:36.944606+0000","last_peered":"2024-09-14T04:11:36.944606+0000","last_clean":"2024-09-14T04:11:36.944606+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:11:36.944606+0000","last_undegraded":"2024-09-14T04:11:36.944606+0000","last_fullsized":"2024-09-14T04:11:36.944606+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"227'14647","reported_seq":14715,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:50.801077+0000","last_change":"2024-09-14T04:07:55.607545+0000","last_active":"2024-09-14T04:11:50.801077+0000","last_peered":"2024-09-14T04:11:50.801077+0000","last_clean":"2024-09-14T04:11:50.801077+0000","last_became_active":"2024-09-14T04:07:55.607263+0000","last_became_peered":"2024-09-14T04:07:55.607263+0000","last_unstale":"2024-09-14T04:11:50.801077+0000","last_undegraded":"2024-09-14T04:11:50.801077+0000","last_fullsized":"2024-09-14T04:11:50.801077+0000","mapping_epoch":199,"log_start":"207'4600","ondisk_log_start":"207'4600","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:38:52.910168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29556736,"num_objects":451,"num_object_clones":0,"num_object_copies":1353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14647,"num_write_kb":483136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":213,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969800+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:11:38.969800+0000","last_peered":"2024-09-14T04:11:38.969800+0000","last_clean":"2024-09-14T04:11:38.969800+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:11:38.969800+0000","last_undegraded":"2024-09-14T04:11:38.969800+0000","last_fullsized":"2024-09-14T04:11:38.969800+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":359,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.984979+0000","last_change":"2024-09-14T04:07:54.363033+0000","last_active":"2024-09-14T04:11:38.984979+0000","last_peered":"2024-09-14T04:11:38.984979+0000","last_clean":"2024-09-14T04:11:38.984979+0000","last_became_active":"2024-09-14T04:07:54.362706+0000","last_became_peered":"2024-09-14T04:07:54.362706+0000","last_unstale":"2024-09-14T04:11:38.984979+0000","last_undegraded":"2024-09-14T04:11:38.984979+0000","last_fullsized":"2024-09-14T04:11:38.984979+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:31.193597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"226'14462","reported_seq":14528,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:47.802588+0000","last_change":"2024-09-14T04:07:55.607327+0000","last_active":"2024-09-14T04:11:47.802588+0000","last_peered":"2024-09-14T04:11:47.802588+0000","last_clean":"2024-09-14T04:11:47.802588+0000","last_became_active":"2024-09-14T04:07:55.606967+0000","last_became_peered":"2024-09-14T04:07:55.606967+0000","last_unstale":"2024-09-14T04:11:47.802588+0000","last_undegraded":"2024-09-14T04:11:47.802588+0000","last_fullsized":"2024-09-14T04:11:47.802588+0000","mapping_epoch":199,"log_start":"210'7900","ondisk_log_start":"210'7900","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:36.618823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42860544,"num_objects":654,"num_object_clones":0,"num_object_copies":1962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14462,"num_write_kb":483712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":301,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.053705+0000","last_change":"2024-09-14T04:05:55.695825+0000","last_active":"2024-09-14T04:11:49.053705+0000","last_peered":"2024-09-14T04:11:49.053705+0000","last_clean":"2024-09-14T04:11:49.053705+0000","last_became_active":"2024-09-14T04:05:55.680668+0000","last_became_peered":"2024-09-14T04:05:55.680668+0000","last_unstale":"2024-09-14T04:11:49.053705+0000","last_undegraded":"2024-09-14T04:11:49.053705+0000","last_fullsized":"2024-09-14T04:11:49.053705+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:30.262947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":221,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.059609+0000","last_change":"2024-09-14T04:05:55.677723+0000","last_active":"2024-09-14T04:11:49.059609+0000","last_peered":"2024-09-14T04:11:49.059609+0000","last_clean":"2024-09-14T04:11:49.059609+0000","last_became_active":"2024-09-14T04:05:55.677254+0000","last_became_peered":"2024-09-14T04:05:55.677254+0000","last_unstale":"2024-09-14T04:11:49.059609+0000","last_undegraded":"2024-09-14T04:11:49.059609+0000","last_fullsized":"2024-09-14T04:11:49.059609+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:53.163120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"227'14471","reported_seq":14538,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:50.822185+0000","last_change":"2024-09-14T04:07:55.607260+0000","last_active":"2024-09-14T04:11:50.822185+0000","last_peered":"2024-09-14T04:11:50.822185+0000","last_clean":"2024-09-14T04:11:50.822185+0000","last_became_active":"2024-09-14T04:07:55.606973+0000","last_became_peered":"2024-09-14T04:07:55.606973+0000","last_unstale":"2024-09-14T04:11:50.822185+0000","last_undegraded":"2024-09-14T04:11:50.822185+0000","last_fullsized":"2024-09-14T04:11:50.822185+0000","mapping_epoch":199,"log_start":"207'4400","ondisk_log_start":"207'4400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:59:15.715573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":491,"num_object_clones":0,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14471,"num_write_kb":478784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":202,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.944744+0000","last_change":"2024-09-14T04:05:55.679201+0000","last_active":"2024-09-14T04:11:36.944744+0000","last_peered":"2024-09-14T04:11:36.944744+0000","last_clean":"2024-09-14T04:11:36.944744+0000","last_became_active":"2024-09-14T04:05:55.678843+0000","last_became_peered":"2024-09-14T04:05:55.678843+0000","last_unstale":"2024-09-14T04:11:36.944744+0000","last_undegraded":"2024-09-14T04:11:36.944744+0000","last_fullsized":"2024-09-14T04:11:36.944744+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:12:57.418514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":212,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.039556+0000","last_change":"2024-09-14T04:05:55.673650+0000","last_active":"2024-09-14T04:11:49.039556+0000","last_peered":"2024-09-14T04:11:49.039556+0000","last_clean":"2024-09-14T04:11:49.039556+0000","last_became_active":"2024-09-14T04:05:55.673258+0000","last_became_peered":"2024-09-14T04:05:55.673258+0000","last_unstale":"2024-09-14T04:11:49.039556+0000","last_undegraded":"2024-09-14T04:11:49.039556+0000","last_fullsized":"2024-09-14T04:11:49.039556+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:21.089797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"226'14421","reported_seq":14456,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:48.818841+0000","last_change":"2024-09-14T04:07:55.365272+0000","last_active":"2024-09-14T04:11:48.818841+0000","last_peered":"2024-09-14T04:11:48.818841+0000","last_clean":"2024-09-14T04:11:48.818841+0000","last_became_active":"2024-09-14T04:07:55.364722+0000","last_became_peered":"2024-09-14T04:07:55.364722+0000","last_unstale":"2024-09-14T04:11:48.818841+0000","last_undegraded":"2024-09-14T04:11:48.818841+0000","last_fullsized":"2024-09-14T04:11:48.818841+0000","mapping_epoch":199,"log_start":"209'6900","ondisk_log_start":"209'6900","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7521,"log_dups_size":0,"ondisk_log_size":7521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:40:55.470085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35192832,"num_objects":537,"num_object_clones":0,"num_object_copies":1611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14421,"num_write_kb":478656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":196,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.036851+0000","last_change":"2024-09-14T04:05:55.671329+0000","last_active":"2024-09-14T04:11:49.036851+0000","last_peered":"2024-09-14T04:11:49.036851+0000","last_clean":"2024-09-14T04:11:49.036851+0000","last_became_active":"2024-09-14T04:05:55.671032+0000","last_became_peered":"2024-09-14T04:05:55.671032+0000","last_unstale":"2024-09-14T04:11:49.036851+0000","last_undegraded":"2024-09-14T04:11:49.036851+0000","last_fullsized":"2024-09-14T04:11:49.036851+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:58:10.779452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":123,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945102+0000","last_change":"2024-09-14T04:05:55.679870+0000","last_active":"2024-09-14T04:11:36.945102+0000","last_peered":"2024-09-14T04:11:36.945102+0000","last_clean":"2024-09-14T04:11:36.945102+0000","last_became_active":"2024-09-14T04:05:55.679574+0000","last_became_peered":"2024-09-14T04:05:55.679574+0000","last_unstale":"2024-09-14T04:11:36.945102+0000","last_undegraded":"2024-09-14T04:11:36.945102+0000","last_fullsized":"2024-09-14T04:11:36.945102+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:51:09.931574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"227'14445","reported_seq":14481,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.174849+0000","last_change":"2024-09-14T04:07:55.367126+0000","last_active":"2024-09-14T04:11:49.174849+0000","last_peered":"2024-09-14T04:11:49.174849+0000","last_clean":"2024-09-14T04:11:49.174849+0000","last_became_active":"2024-09-14T04:07:55.366798+0000","last_became_peered":"2024-09-14T04:07:55.366798+0000","last_unstale":"2024-09-14T04:11:49.174849+0000","last_undegraded":"2024-09-14T04:11:49.174849+0000","last_fullsized":"2024-09-14T04:11:49.174849+0000","mapping_epoch":199,"log_start":"209'6300","ondisk_log_start":"209'6300","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:24:51.791784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35454976,"num_objects":541,"num_object_clones":0,"num_object_copies":1623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14445,"num_write_kb":479552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.050427+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:11:49.050427+0000","last_peered":"2024-09-14T04:11:49.050427+0000","last_clean":"2024-09-14T04:11:49.050427+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:11:49.050427+0000","last_undegraded":"2024-09-14T04:11:49.050427+0000","last_fullsized":"2024-09-14T04:11:49.050427+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":303,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995725+0000","last_change":"2024-09-14T04:05:55.678520+0000","last_active":"2024-09-14T04:11:38.995725+0000","last_peered":"2024-09-14T04:11:38.995725+0000","last_clean":"2024-09-14T04:11:38.995725+0000","last_became_active":"2024-09-14T04:05:55.678008+0000","last_became_peered":"2024-09-14T04:05:55.678008+0000","last_unstale":"2024-09-14T04:11:38.995725+0000","last_undegraded":"2024-09-14T04:11:38.995725+0000","last_fullsized":"2024-09-14T04:11:38.995725+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:40:26.778212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"226'14192","reported_seq":14227,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:48.833394+0000","last_change":"2024-09-14T04:07:55.367659+0000","last_active":"2024-09-14T04:11:48.833394+0000","last_peered":"2024-09-14T04:11:48.833394+0000","last_clean":"2024-09-14T04:11:48.833394+0000","last_became_active":"2024-09-14T04:07:55.367247+0000","last_became_peered":"2024-09-14T04:07:55.367247+0000","last_unstale":"2024-09-14T04:11:48.833394+0000","last_undegraded":"2024-09-14T04:11:48.833394+0000","last_fullsized":"2024-09-14T04:11:48.833394+0000","mapping_epoch":199,"log_start":"209'6600","ondisk_log_start":"209'6600","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7592,"log_dups_size":0,"ondisk_log_size":7592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:58:08.638370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":582,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14192,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":309,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.983830+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:11:38.983830+0000","last_peered":"2024-09-14T04:11:38.983830+0000","last_clean":"2024-09-14T04:11:38.983830+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:11:38.983830+0000","last_undegraded":"2024-09-14T04:11:38.983830+0000","last_fullsized":"2024-09-14T04:11:38.983830+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":206,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.973572+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:11:38.973572+0000","last_peered":"2024-09-14T04:11:38.973572+0000","last_clean":"2024-09-14T04:11:38.973572+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:11:38.973572+0000","last_undegraded":"2024-09-14T04:11:38.973572+0000","last_fullsized":"2024-09-14T04:11:38.973572+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.049003+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:11:49.049003+0000","last_peered":"2024-09-14T04:11:49.049003+0000","last_clean":"2024-09-14T04:11:49.049003+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:11:49.049003+0000","last_undegraded":"2024-09-14T04:11:49.049003+0000","last_fullsized":"2024-09-14T04:11:49.049003+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":184,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945111+0000","last_change":"2024-09-14T04:05:55.676445+0000","last_active":"2024-09-14T04:11:36.945111+0000","last_peered":"2024-09-14T04:11:36.945111+0000","last_clean":"2024-09-14T04:11:36.945111+0000","last_became_active":"2024-09-14T04:05:55.674141+0000","last_became_peered":"2024-09-14T04:05:55.674141+0000","last_unstale":"2024-09-14T04:11:36.945111+0000","last_undegraded":"2024-09-14T04:11:36.945111+0000","last_fullsized":"2024-09-14T04:11:36.945111+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:28:50.348309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"226'14546","reported_seq":14612,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:48.651687+0000","last_change":"2024-09-14T04:07:55.366975+0000","last_active":"2024-09-14T04:11:48.651687+0000","last_peered":"2024-09-14T04:11:48.651687+0000","last_clean":"2024-09-14T04:11:48.651687+0000","last_became_active":"2024-09-14T04:07:55.366645+0000","last_became_peered":"2024-09-14T04:07:55.366645+0000","last_unstale":"2024-09-14T04:11:48.651687+0000","last_undegraded":"2024-09-14T04:11:48.651687+0000","last_fullsized":"2024-09-14T04:11:48.651687+0000","mapping_epoch":199,"log_start":"210'7500","ondisk_log_start":"210'7500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7046,"log_dups_size":0,"ondisk_log_size":7046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:39:56.637518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":548,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14546,"num_write_kb":483008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":221,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.983831+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:11:38.983831+0000","last_peered":"2024-09-14T04:11:38.983831+0000","last_clean":"2024-09-14T04:11:38.983831+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:11:38.983831+0000","last_undegraded":"2024-09-14T04:11:38.983831+0000","last_fullsized":"2024-09-14T04:11:38.983831+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":353,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985721+0000","last_change":"2024-09-14T04:07:44.337314+0000","last_active":"2024-09-14T04:11:38.985721+0000","last_peered":"2024-09-14T04:11:38.985721+0000","last_clean":"2024-09-14T04:11:38.985721+0000","last_became_active":"2024-09-14T04:07:44.336916+0000","last_became_peered":"2024-09-14T04:07:44.336916+0000","last_unstale":"2024-09-14T04:11:38.985721+0000","last_undegraded":"2024-09-14T04:11:38.985721+0000","last_fullsized":"2024-09-14T04:11:38.985721+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:23.153578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":342,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995338+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:11:38.995338+0000","last_peered":"2024-09-14T04:11:38.995338+0000","last_clean":"2024-09-14T04:11:38.995338+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:11:38.995338+0000","last_undegraded":"2024-09-14T04:11:38.995338+0000","last_fullsized":"2024-09-14T04:11:38.995338+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"227'14511","reported_seq":14547,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.399676+0000","last_change":"2024-09-14T04:07:55.365448+0000","last_active":"2024-09-14T04:11:49.399676+0000","last_peered":"2024-09-14T04:11:49.399676+0000","last_clean":"2024-09-14T04:11:49.399676+0000","last_became_active":"2024-09-14T04:07:55.365168+0000","last_became_peered":"2024-09-14T04:07:55.365168+0000","last_unstale":"2024-09-14T04:11:49.399676+0000","last_undegraded":"2024-09-14T04:11:49.399676+0000","last_fullsized":"2024-09-14T04:11:49.399676+0000","mapping_epoch":199,"log_start":"207'4500","ondisk_log_start":"207'4500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:25.997258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35192832,"num_objects":537,"num_object_clones":0,"num_object_copies":1611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14511,"num_write_kb":481536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":338,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.036990+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:11:49.036990+0000","last_peered":"2024-09-14T04:11:49.036990+0000","last_clean":"2024-09-14T04:11:49.036990+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:11:49.036990+0000","last_undegraded":"2024-09-14T04:11:49.036990+0000","last_fullsized":"2024-09-14T04:11:49.036990+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":297,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.990304+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:11:38.990304+0000","last_peered":"2024-09-14T04:11:38.990304+0000","last_clean":"2024-09-14T04:11:38.990304+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:11:38.990304+0000","last_undegraded":"2024-09-14T04:11:38.990304+0000","last_fullsized":"2024-09-14T04:11:38.990304+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":69,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.036849+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:11:49.036849+0000","last_peered":"2024-09-14T04:11:49.036849+0000","last_clean":"2024-09-14T04:11:49.036849+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:11:49.036849+0000","last_undegraded":"2024-09-14T04:11:49.036849+0000","last_fullsized":"2024-09-14T04:11:49.036849+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"226'14668","reported_seq":14703,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:48.707485+0000","last_change":"2024-09-14T04:07:55.367851+0000","last_active":"2024-09-14T04:11:48.707485+0000","last_peered":"2024-09-14T04:11:48.707485+0000","last_clean":"2024-09-14T04:11:48.707485+0000","last_became_active":"2024-09-14T04:07:55.367146+0000","last_became_peered":"2024-09-14T04:07:55.367146+0000","last_unstale":"2024-09-14T04:11:48.707485+0000","last_undegraded":"2024-09-14T04:11:48.707485+0000","last_fullsized":"2024-09-14T04:11:48.707485+0000","mapping_epoch":199,"log_start":"210'7100","ondisk_log_start":"210'7100","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:26:46.495508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38404096,"num_objects":586,"num_object_clones":0,"num_object_copies":1758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14668,"num_write_kb":488128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":207,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969697+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:11:38.969697+0000","last_peered":"2024-09-14T04:11:38.969697+0000","last_clean":"2024-09-14T04:11:38.969697+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:11:38.969697+0000","last_undegraded":"2024-09-14T04:11:38.969697+0000","last_fullsized":"2024-09-14T04:11:38.969697+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":165,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985284+0000","last_change":"2024-09-14T04:05:55.676693+0000","last_active":"2024-09-14T04:11:38.985284+0000","last_peered":"2024-09-14T04:11:38.985284+0000","last_clean":"2024-09-14T04:11:38.985284+0000","last_became_active":"2024-09-14T04:05:55.676335+0000","last_became_peered":"2024-09-14T04:05:55.676335+0000","last_unstale":"2024-09-14T04:11:38.985284+0000","last_undegraded":"2024-09-14T04:11:38.985284+0000","last_fullsized":"2024-09-14T04:11:38.985284+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:19:43.626016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":322,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.994435+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:11:38.994435+0000","last_peered":"2024-09-14T04:11:38.994435+0000","last_clean":"2024-09-14T04:11:38.994435+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:11:38.994435+0000","last_undegraded":"2024-09-14T04:11:38.994435+0000","last_fullsized":"2024-09-14T04:11:38.994435+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"227'14401","reported_seq":14437,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.191554+0000","last_change":"2024-09-14T04:07:55.607796+0000","last_active":"2024-09-14T04:11:49.191554+0000","last_peered":"2024-09-14T04:11:49.191554+0000","last_clean":"2024-09-14T04:11:49.191554+0000","last_became_active":"2024-09-14T04:07:55.607447+0000","last_became_peered":"2024-09-14T04:07:55.607447+0000","last_unstale":"2024-09-14T04:11:49.191554+0000","last_undegraded":"2024-09-14T04:11:49.191554+0000","last_fullsized":"2024-09-14T04:11:49.191554+0000","mapping_epoch":199,"log_start":"209'6200","ondisk_log_start":"209'6200","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:09:44.307466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36634624,"num_objects":559,"num_object_clones":0,"num_object_copies":1677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14401,"num_write_kb":478720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":230,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.970183+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:11:38.970183+0000","last_peered":"2024-09-14T04:11:38.970183+0000","last_clean":"2024-09-14T04:11:38.970183+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:11:38.970183+0000","last_undegraded":"2024-09-14T04:11:38.970183+0000","last_fullsized":"2024-09-14T04:11:38.970183+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":336,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.994522+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:11:38.994522+0000","last_peered":"2024-09-14T04:11:38.994522+0000","last_clean":"2024-09-14T04:11:38.994522+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:11:38.994522+0000","last_undegraded":"2024-09-14T04:11:38.994522+0000","last_fullsized":"2024-09-14T04:11:38.994522+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":293,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.993123+0000","last_change":"2024-09-14T04:05:55.674511+0000","last_active":"2024-09-14T04:11:38.993123+0000","last_peered":"2024-09-14T04:11:38.993123+0000","last_clean":"2024-09-14T04:11:38.993123+0000","last_became_active":"2024-09-14T04:05:55.674182+0000","last_became_peered":"2024-09-14T04:05:55.674182+0000","last_unstale":"2024-09-14T04:11:38.993123+0000","last_undegraded":"2024-09-14T04:11:38.993123+0000","last_fullsized":"2024-09-14T04:11:38.993123+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:12:34.453647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"226'14508","reported_seq":14544,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:48.765675+0000","last_change":"2024-09-14T04:07:55.607679+0000","last_active":"2024-09-14T04:11:48.765675+0000","last_peered":"2024-09-14T04:11:48.765675+0000","last_clean":"2024-09-14T04:11:48.765675+0000","last_became_active":"2024-09-14T04:07:55.607338+0000","last_became_peered":"2024-09-14T04:07:55.607338+0000","last_unstale":"2024-09-14T04:11:48.765675+0000","last_undegraded":"2024-09-14T04:11:48.765675+0000","last_fullsized":"2024-09-14T04:11:48.765675+0000","mapping_epoch":199,"log_start":"210'7000","ondisk_log_start":"210'7000","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7508,"log_dups_size":0,"ondisk_log_size":7508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:02:20.945987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36634648,"num_objects":560,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":560,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14508,"num_write_kb":482113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":297,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.994656+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:11:38.994656+0000","last_peered":"2024-09-14T04:11:38.994656+0000","last_clean":"2024-09-14T04:11:38.994656+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:11:38.994656+0000","last_undegraded":"2024-09-14T04:11:38.994656+0000","last_fullsized":"2024-09-14T04:11:38.994656+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":214,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981522+0000","last_change":"2024-09-14T04:05:55.681238+0000","last_active":"2024-09-14T04:11:38.981522+0000","last_peered":"2024-09-14T04:11:38.981522+0000","last_clean":"2024-09-14T04:11:38.981522+0000","last_became_active":"2024-09-14T04:05:55.680946+0000","last_became_peered":"2024-09-14T04:05:55.680946+0000","last_unstale":"2024-09-14T04:11:38.981522+0000","last_undegraded":"2024-09-14T04:11:38.981522+0000","last_fullsized":"2024-09-14T04:11:38.981522+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:13:39.006545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":321,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985058+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:11:38.985058+0000","last_peered":"2024-09-14T04:11:38.985058+0000","last_clean":"2024-09-14T04:11:38.985058+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:11:38.985058+0000","last_undegraded":"2024-09-14T04:11:38.985058+0000","last_fullsized":"2024-09-14T04:11:38.985058+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"226'14489","reported_seq":14524,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:48.828463+0000","last_change":"2024-09-14T04:07:55.366077+0000","last_active":"2024-09-14T04:11:48.828463+0000","last_peered":"2024-09-14T04:11:48.828463+0000","last_clean":"2024-09-14T04:11:48.828463+0000","last_became_active":"2024-09-14T04:07:55.365753+0000","last_became_peered":"2024-09-14T04:07:55.365753+0000","last_unstale":"2024-09-14T04:11:48.828463+0000","last_undegraded":"2024-09-14T04:11:48.828463+0000","last_fullsized":"2024-09-14T04:11:48.828463+0000","mapping_epoch":199,"log_start":"209'6900","ondisk_log_start":"209'6900","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:04:18.784725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35323904,"num_objects":539,"num_object_clones":0,"num_object_copies":1617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14489,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":291,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.058178+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:11:49.058178+0000","last_peered":"2024-09-14T04:11:49.058178+0000","last_clean":"2024-09-14T04:11:49.058178+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:11:49.058178+0000","last_undegraded":"2024-09-14T04:11:49.058178+0000","last_fullsized":"2024-09-14T04:11:49.058178+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":304,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995891+0000","last_change":"2024-09-14T04:05:55.676617+0000","last_active":"2024-09-14T04:11:38.995891+0000","last_peered":"2024-09-14T04:11:38.995891+0000","last_clean":"2024-09-14T04:11:38.995891+0000","last_became_active":"2024-09-14T04:05:55.676318+0000","last_became_peered":"2024-09-14T04:05:55.676318+0000","last_unstale":"2024-09-14T04:11:38.995891+0000","last_undegraded":"2024-09-14T04:11:38.995891+0000","last_fullsized":"2024-09-14T04:11:38.995891+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:38.529611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":38,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969445+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:11:38.969445+0000","last_peered":"2024-09-14T04:11:38.969445+0000","last_clean":"2024-09-14T04:11:38.969445+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:11:38.969445+0000","last_undegraded":"2024-09-14T04:11:38.969445+0000","last_fullsized":"2024-09-14T04:11:38.969445+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.f","version":"226'14363","reported_seq":14398,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:48.805519+0000","last_change":"2024-09-14T04:07:55.607496+0000","last_active":"2024-09-14T04:11:48.805519+0000","last_peered":"2024-09-14T04:11:48.805519+0000","last_clean":"2024-09-14T04:11:48.805519+0000","last_became_active":"2024-09-14T04:07:55.607124+0000","last_became_peered":"2024-09-14T04:07:55.607124+0000","last_unstale":"2024-09-14T04:11:48.805519+0000","last_undegraded":"2024-09-14T04:11:48.805519+0000","last_fullsized":"2024-09-14T04:11:48.805519+0000","mapping_epoch":199,"log_start":"209'6800","ondisk_log_start":"209'6800","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:15:10.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36634624,"num_objects":559,"num_object_clones":0,"num_object_copies":1677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14363,"num_write_kb":477504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":345,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995718+0000","last_change":"2024-09-14T04:05:55.690010+0000","last_active":"2024-09-14T04:11:38.995718+0000","last_peered":"2024-09-14T04:11:38.995718+0000","last_clean":"2024-09-14T04:11:38.995718+0000","last_became_active":"2024-09-14T04:05:55.670892+0000","last_became_peered":"2024-09-14T04:05:55.670892+0000","last_unstale":"2024-09-14T04:11:38.995718+0000","last_undegraded":"2024-09-14T04:11:38.995718+0000","last_fullsized":"2024-09-14T04:11:38.995718+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:40:33.024617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945129+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:11:36.945129+0000","last_peered":"2024-09-14T04:11:36.945129+0000","last_clean":"2024-09-14T04:11:36.945129+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:11:36.945129+0000","last_undegraded":"2024-09-14T04:11:36.945129+0000","last_fullsized":"2024-09-14T04:11:36.945129+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":70,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969847+0000","last_change":"2024-09-14T04:05:55.674629+0000","last_active":"2024-09-14T04:11:38.969847+0000","last_peered":"2024-09-14T04:11:38.969847+0000","last_clean":"2024-09-14T04:11:38.969847+0000","last_became_active":"2024-09-14T04:05:55.674242+0000","last_became_peered":"2024-09-14T04:05:55.674242+0000","last_unstale":"2024-09-14T04:11:38.969847+0000","last_undegraded":"2024-09-14T04:11:38.969847+0000","last_fullsized":"2024-09-14T04:11:38.969847+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:06:16.827114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":82,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.944941+0000","last_change":"2024-09-14T04:05:55.677636+0000","last_active":"2024-09-14T04:11:36.944941+0000","last_peered":"2024-09-14T04:11:36.944941+0000","last_clean":"2024-09-14T04:11:36.944941+0000","last_became_active":"2024-09-14T04:05:55.677313+0000","last_became_peered":"2024-09-14T04:05:55.677313+0000","last_unstale":"2024-09-14T04:11:36.944941+0000","last_undegraded":"2024-09-14T04:11:36.944941+0000","last_fullsized":"2024-09-14T04:11:36.944941+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:00:13.489641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":123,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981276+0000","last_change":"2024-09-14T04:05:55.686083+0000","last_active":"2024-09-14T04:11:38.981276+0000","last_peered":"2024-09-14T04:11:38.981276+0000","last_clean":"2024-09-14T04:11:38.981276+0000","last_became_active":"2024-09-14T04:05:55.673710+0000","last_became_peered":"2024-09-14T04:05:55.673710+0000","last_unstale":"2024-09-14T04:11:38.981276+0000","last_undegraded":"2024-09-14T04:11:38.981276+0000","last_fullsized":"2024-09-14T04:11:38.981276+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:25.169218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":296,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.992977+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:11:38.992977+0000","last_peered":"2024-09-14T04:11:38.992977+0000","last_clean":"2024-09-14T04:11:38.992977+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:11:38.992977+0000","last_undegraded":"2024-09-14T04:11:38.992977+0000","last_fullsized":"2024-09-14T04:11:38.992977+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":211,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981398+0000","last_change":"2024-09-14T04:05:55.698766+0000","last_active":"2024-09-14T04:11:38.981398+0000","last_peered":"2024-09-14T04:11:38.981398+0000","last_clean":"2024-09-14T04:11:38.981398+0000","last_became_active":"2024-09-14T04:05:55.674923+0000","last_became_peered":"2024-09-14T04:05:55.674923+0000","last_unstale":"2024-09-14T04:11:38.981398+0000","last_undegraded":"2024-09-14T04:11:38.981398+0000","last_fullsized":"2024-09-14T04:11:38.981398+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:52:37.679728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":193,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.971915+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:11:38.971915+0000","last_peered":"2024-09-14T04:11:38.971915+0000","last_clean":"2024-09-14T04:11:38.971915+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:11:38.971915+0000","last_undegraded":"2024-09-14T04:11:38.971915+0000","last_fullsized":"2024-09-14T04:11:38.971915+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":250,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.049122+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:11:49.049122+0000","last_peered":"2024-09-14T04:11:49.049122+0000","last_clean":"2024-09-14T04:11:49.049122+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:11:49.049122+0000","last_undegraded":"2024-09-14T04:11:49.049122+0000","last_fullsized":"2024-09-14T04:11:49.049122+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":345,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.999013+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:11:38.999013+0000","last_peered":"2024-09-14T04:11:38.999013+0000","last_clean":"2024-09-14T04:11:38.999013+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:11:38.999013+0000","last_undegraded":"2024-09-14T04:11:38.999013+0000","last_fullsized":"2024-09-14T04:11:38.999013+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":262,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.050503+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:11:49.050503+0000","last_peered":"2024-09-14T04:11:49.050503+0000","last_clean":"2024-09-14T04:11:49.050503+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:11:49.050503+0000","last_undegraded":"2024-09-14T04:11:49.050503+0000","last_fullsized":"2024-09-14T04:11:49.050503+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":201,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945065+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:11:36.945065+0000","last_peered":"2024-09-14T04:11:36.945065+0000","last_clean":"2024-09-14T04:11:36.945065+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:11:36.945065+0000","last_undegraded":"2024-09-14T04:11:36.945065+0000","last_fullsized":"2024-09-14T04:11:36.945065+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.050337+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:11:49.050337+0000","last_peered":"2024-09-14T04:11:49.050337+0000","last_clean":"2024-09-14T04:11:49.050337+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:11:49.050337+0000","last_undegraded":"2024-09-14T04:11:49.050337+0000","last_fullsized":"2024-09-14T04:11:49.050337+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":346,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.995529+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:11:38.995529+0000","last_peered":"2024-09-14T04:11:38.995529+0000","last_clean":"2024-09-14T04:11:38.995529+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:11:38.995529+0000","last_undegraded":"2024-09-14T04:11:38.995529+0000","last_fullsized":"2024-09-14T04:11:38.995529+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":303,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.991707+0000","last_change":"2024-09-14T04:05:55.677962+0000","last_active":"2024-09-14T04:11:38.991707+0000","last_peered":"2024-09-14T04:11:38.991707+0000","last_clean":"2024-09-14T04:11:38.991707+0000","last_became_active":"2024-09-14T04:05:55.677277+0000","last_became_peered":"2024-09-14T04:05:55.677277+0000","last_unstale":"2024-09-14T04:11:38.991707+0000","last_undegraded":"2024-09-14T04:11:38.991707+0000","last_fullsized":"2024-09-14T04:11:38.991707+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:24.908568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.057620+0000","last_change":"2024-09-14T04:07:44.338546+0000","last_active":"2024-09-14T04:11:49.057620+0000","last_peered":"2024-09-14T04:11:49.057620+0000","last_clean":"2024-09-14T04:11:49.057620+0000","last_became_active":"2024-09-14T04:07:44.338210+0000","last_became_peered":"2024-09-14T04:07:44.338210+0000","last_unstale":"2024-09-14T04:11:49.057620+0000","last_undegraded":"2024-09-14T04:11:49.057620+0000","last_fullsized":"2024-09-14T04:11:49.057620+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:56.100013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":127,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.993200+0000","last_change":"2024-09-14T04:05:55.671091+0000","last_active":"2024-09-14T04:11:38.993200+0000","last_peered":"2024-09-14T04:11:38.993200+0000","last_clean":"2024-09-14T04:11:38.993200+0000","last_became_active":"2024-09-14T04:05:55.670180+0000","last_became_peered":"2024-09-14T04:05:55.670180+0000","last_unstale":"2024-09-14T04:11:38.993200+0000","last_undegraded":"2024-09-14T04:11:38.993200+0000","last_fullsized":"2024-09-14T04:11:38.993200+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:33.489272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.037032+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:11:49.037032+0000","last_peered":"2024-09-14T04:11:49.037032+0000","last_clean":"2024-09-14T04:11:49.037032+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:11:49.037032+0000","last_undegraded":"2024-09-14T04:11:49.037032+0000","last_fullsized":"2024-09-14T04:11:49.037032+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":329,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.985133+0000","last_change":"2024-09-14T04:05:55.671536+0000","last_active":"2024-09-14T04:11:38.985133+0000","last_peered":"2024-09-14T04:11:38.985133+0000","last_clean":"2024-09-14T04:11:38.985133+0000","last_became_active":"2024-09-14T04:05:55.671029+0000","last_became_peered":"2024-09-14T04:05:55.671029+0000","last_unstale":"2024-09-14T04:11:38.985133+0000","last_undegraded":"2024-09-14T04:11:38.985133+0000","last_fullsized":"2024-09-14T04:11:38.985133+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:35.745580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":238,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.053608+0000","last_change":"2024-09-14T04:05:55.681242+0000","last_active":"2024-09-14T04:11:49.053608+0000","last_peered":"2024-09-14T04:11:49.053608+0000","last_clean":"2024-09-14T04:11:49.053608+0000","last_became_active":"2024-09-14T04:05:55.680502+0000","last_became_peered":"2024-09-14T04:05:55.680502+0000","last_unstale":"2024-09-14T04:11:49.053608+0000","last_undegraded":"2024-09-14T04:11:49.053608+0000","last_fullsized":"2024-09-14T04:11:49.053608+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:23.760294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":209,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969717+0000","last_change":"2024-09-14T04:05:55.687662+0000","last_active":"2024-09-14T04:11:38.969717+0000","last_peered":"2024-09-14T04:11:38.969717+0000","last_clean":"2024-09-14T04:11:38.969717+0000","last_became_active":"2024-09-14T04:05:55.673561+0000","last_became_peered":"2024-09-14T04:05:55.673561+0000","last_unstale":"2024-09-14T04:11:38.969717+0000","last_undegraded":"2024-09-14T04:11:38.969717+0000","last_fullsized":"2024-09-14T04:11:38.969717+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:14:51.708356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":238,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.053930+0000","last_change":"2024-09-14T04:05:55.679885+0000","last_active":"2024-09-14T04:11:49.053930+0000","last_peered":"2024-09-14T04:11:49.053930+0000","last_clean":"2024-09-14T04:11:49.053930+0000","last_became_active":"2024-09-14T04:05:55.678589+0000","last_became_peered":"2024-09-14T04:05:55.678589+0000","last_unstale":"2024-09-14T04:11:49.053930+0000","last_undegraded":"2024-09-14T04:11:49.053930+0000","last_fullsized":"2024-09-14T04:11:49.053930+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:51:32.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":192,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.970087+0000","last_change":"2024-09-14T04:05:55.697832+0000","last_active":"2024-09-14T04:11:38.970087+0000","last_peered":"2024-09-14T04:11:38.970087+0000","last_clean":"2024-09-14T04:11:38.970087+0000","last_became_active":"2024-09-14T04:05:55.673923+0000","last_became_peered":"2024-09-14T04:05:55.673923+0000","last_unstale":"2024-09-14T04:11:38.970087+0000","last_undegraded":"2024-09-14T04:11:38.970087+0000","last_fullsized":"2024-09-14T04:11:38.970087+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:44.697302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.049045+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:11:49.049045+0000","last_peered":"2024-09-14T04:11:49.049045+0000","last_clean":"2024-09-14T04:11:49.049045+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:11:49.049045+0000","last_undegraded":"2024-09-14T04:11:49.049045+0000","last_fullsized":"2024-09-14T04:11:49.049045+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":76,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.057180+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:11:49.057180+0000","last_peered":"2024-09-14T04:11:49.057180+0000","last_clean":"2024-09-14T04:11:49.057180+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:11:49.057180+0000","last_undegraded":"2024-09-14T04:11:49.057180+0000","last_fullsized":"2024-09-14T04:11:49.057180+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":143,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-09-14T04:11:36.945886+0000","last_change":"2024-09-14T04:07:44.338957+0000","last_active":"2024-09-14T04:11:36.945886+0000","last_peered":"2024-09-14T04:11:36.945886+0000","last_clean":"2024-09-14T04:11:36.945886+0000","last_became_active":"2024-09-14T04:07:44.338190+0000","last_became_peered":"2024-09-14T04:07:44.338190+0000","last_unstale":"2024-09-14T04:11:36.945886+0000","last_undegraded":"2024-09-14T04:11:36.945886+0000","last_fullsized":"2024-09-14T04:11:36.945886+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:36:27.488115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":335,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.998644+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:11:38.998644+0000","last_peered":"2024-09-14T04:11:38.998644+0000","last_clean":"2024-09-14T04:11:38.998644+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:11:38.998644+0000","last_undegraded":"2024-09-14T04:11:38.998644+0000","last_fullsized":"2024-09-14T04:11:38.998644+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":577830936,"num_objects":8818,"num_object_clones":0,"num_object_copies":26454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8818,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231570,"num_write_kb":7692353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1722298368,"data_stored":1722286152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130070,"ondisk_log_size":130070,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4890,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430577,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1081568,"kb_used_data":178520,"kb_used_omap":1,"kb_used_meta":903038,"kb_avail":92663584,"statfs":{"total":95995035648,"available":94887510016,"internally_reserved":0,"allocated":182804480,"data_stored":181743759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924711256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626806,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1154460,"kb_used_data":261268,"kb_used_omap":1,"kb_used_meta":893182,"kb_avail":92590692,"statfs":{"total":95995035648,"available":94812868608,"internally_reserved":0,"allocated":267538432,"data_stored":266481423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914618712},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823034,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1124868,"kb_used_data":223740,"kb_used_omap":1,"kb_used_meta":901118,"kb_avail":92620284,"statfs":{"total":95995035648,"available":94843170816,"internally_reserved":0,"allocated":229109760,"data_stored":228072236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922745176},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":798863917116,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552415,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":792408,"kb_used_data":281612,"kb_used_omap":4,"kb_used_meta":510715,"kb_avail":92952744,"statfs":{"total":95995035648,"available":95183609856,"internally_reserved":0,"allocated":288370688,"data_stored":287325188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":522972605},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435457,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":613852,"kb_used_data":182096,"kb_used_omap":6,"kb_used_meta":431673,"kb_avail":93131300,"statfs":{"total":95995035648,"available":95366451200,"internally_reserved":0,"allocated":186466304,"data_stored":185420480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":442033613},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827926,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":868424,"kb_used_data":241152,"kb_used_omap":5,"kb_used_meta":627258,"kb_avail":92876728,"statfs":{"total":95995035648,"available":95105769472,"internally_reserved":0,"allocated":246939648,"data_stored":245898060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":642312431},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383914,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1102032,"kb_used_data":377544,"kb_used_omap":5,"kb_used_meta":724410,"kb_avail":92643120,"statfs":{"total":95995035648,"available":94866554880,"internally_reserved":0,"allocated":386605056,"data_stored":385556186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":741795900},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":377688064,"data_stored":377683992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":240517120,"data_stored":240517120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":179765248,"data_stored":179765248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":277680128,"data_stored":277676056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":212729856,"data_stored":212729856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":257363968,"data_stored":257359896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":176553984,"data_stored":176553984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:11:54.852 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-14T04:11:54.852 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:11:54.852 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_5 2024-09-14T04:11:54.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:11:54.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:54.960 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:55.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:55.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:55.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:55.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:55.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:55.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:55.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:55.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:55.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:55.438 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:55.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:55.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:55.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:55.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:55.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:55.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:55.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:55.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:55.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:55.858 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:55.859 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:55.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:55.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:56.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:56.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:56.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:56.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:56.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:56 smithi160 ceph-mon[88407]: pgmap v845: 89 pgs: 89 active+clean; 332 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 795 op/s 2024-09-14T04:11:56.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:11:56.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:56.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:11:56.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:56 smithi114 ceph-mon[106969]: pgmap v845: 89 pgs: 89 active+clean; 332 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 795 op/s 2024-09-14T04:11:56.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:56 smithi114 ceph-mon[103029]: pgmap v845: 89 pgs: 89 active+clean; 332 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 795 op/s 2024-09-14T04:11:56.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:56.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:11:56.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:56.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:56.467 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:11:56.467 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:11:56.468 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:11:56.468 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:11:56.468 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:11:56.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:56.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:56.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:56.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:56.712 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:56.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:56.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:56.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:56.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:56.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:56.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:56.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:56.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:57.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:57.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:57.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:57.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-14T04:11:57.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:57.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:57.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:57.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:57.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:57.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:57.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:11:57.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:57.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:11:57.527 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:11:57.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:57.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:57.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:57.773 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:57.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:57.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:57.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:57.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:57.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:57.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:58.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:58.079 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:58.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:58.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:58.181 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:58.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:58.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:58.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:58.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:11:58.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:58.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:11:58.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:58 smithi114 ceph-mon[103029]: pgmap v846: 89 pgs: 89 active+clean; 300 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 799 op/s 2024-09-14T04:11:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:58 smithi114 ceph-mon[106969]: pgmap v846: 89 pgs: 89 active+clean; 300 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 799 op/s 2024-09-14T04:11:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:58 smithi160 ceph-mon[88407]: pgmap v846: 89 pgs: 89 active+clean; 300 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 799 op/s 2024-09-14T04:11:58.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:11:58.490 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:58.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:58.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:11:58.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:58.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:58.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:58.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:58.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:58.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:58.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:58.927 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:58.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:59.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:11:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:59.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:59.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:11:59.078 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:11:59.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:59.193 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:59.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:59.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:11:59.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:59 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4110211187' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:59.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:11:59 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:59.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:59 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4110211187' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:59.312 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:11:59 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:59.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:11:59.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:11:59.372 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:59 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4110211187' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:59.372 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:11:59 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:11:59.403 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:11:59.403 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:11:59.403 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:11:59.403 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:79163 9.d 9:b8bb6715:::benchmark_data_smithi114_153893_object79162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:04:02.322866+0000", 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 477.07618474899999, 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32682998800000002, 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 79163 2024-09-14T04:11:59.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322866+0000", 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322866+0000", 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322865+0000", 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322920+0000", 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5031999999999998e-05 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322932+0000", 2024-09-14T04:11:59.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2268e-05 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.322941+0000", 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6689999999999995e-06 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.323046+0000", 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010484000000000001 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.323232+0000", 2024-09-14T04:11:59.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018588599999999999 2024-09-14T04:11:59.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,5", 2024-09-14T04:11:59.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.323552+0000", 2024-09-14T04:11:59.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00032002499999999999 2024-09-14T04:11:59.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.326534+0000", 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029825390000000002 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.328765+0000", 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022310860000000002 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.649306+0000", 2024-09-14T04:11:59.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32054082099999998 2024-09-14T04:11:59.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.649661+0000", 2024-09-14T04:11:59.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035479400000000002 2024-09-14T04:11:59.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:04:02.649696+0000", 2024-09-14T04:11:59.411 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5377999999999999e-05 2024-09-14T04:11:59.411 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.411 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.411 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.411 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:195981 9.e 9:741dedbf:::benchmark_data_smithi114_153893_object66520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-14T04:11:59.411 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:06.788523+0000", 2024-09-14T04:11:59.412 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 352.61052761500002, 2024-09-14T04:11:59.412 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.331278462, 2024-09-14T04:11:59.412 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.412 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.412 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.413 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.413 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.413 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195981 2024-09-14T04:11:59.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.413 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.414 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.414 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.414 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788523+0000", 2024-09-14T04:11:59.414 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.414 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.414 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.414 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788523+0000", 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788522+0000", 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788528+0000", 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3659999999999997e-06 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788541+0000", 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2755000000000001e-05 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788546+0000", 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0239999999999997e-06 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788597+0000", 2024-09-14T04:11:59.416 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1316000000000001e-05 2024-09-14T04:11:59.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.417 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.417 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788726+0000", 2024-09-14T04:11:59.417 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012843300000000001 2024-09-14T04:11:59.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.417 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,4", 2024-09-14T04:11:59.417 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788884+0000", 2024-09-14T04:11:59.417 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015847600000000001 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.792615+0000", 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003730424 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.792730+0000", 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011485900000000001 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.418 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.805843+0000", 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013113855000000001 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119607+0000", 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31376335799999999 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119669+0000", 2024-09-14T04:11:59.419 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2397000000000006e-05 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119802+0000", 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013257899999999999 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:195983 9.e 9:76113637:::benchmark_data_smithi114_153893_object66522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:06.793113+0000", 2024-09-14T04:11:59.420 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 352.60593841299999, 2024-09-14T04:11:59.421 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32693720900000001, 2024-09-14T04:11:59.421 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.421 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.421 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.421 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.421 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.421 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195983 2024-09-14T04:11:59.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793113+0000", 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793113+0000", 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.422 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793111+0000", 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793117+0000", 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4919999999999998e-06 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793130+0000", 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2347999999999999e-05 2024-09-14T04:11:59.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793136+0000", 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2310000000000001e-06 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793173+0000", 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7468000000000001e-05 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.424 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793290+0000", 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116841 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,4", 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793372+0000", 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1866000000000005e-05 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.796363+0000", 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029914579999999998 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.425 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.797301+0000", 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000937089 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.805934+0000", 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0086334329999999994 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119867+0000", 2024-09-14T04:11:59.426 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31393343200000001 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119922+0000", 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4187000000000001e-05 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.120050+0000", 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012817899999999999 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:195999 9.e 9:7a76e1c2:::benchmark_data_smithi114_153893_object66538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:06.807061+0000", 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 352.59198984400001, 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31979472599999997, 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195999 2024-09-14T04:11:59.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807061+0000", 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807061+0000", 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807060+0000", 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807064+0000", 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.912e-06 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807075+0000", 2024-09-14T04:11:59.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0807e-05 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807081+0000", 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9699999999999996e-06 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807122+0000", 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1507000000000001e-05 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807247+0000", 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012494900000000001 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,4", 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807381+0000", 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000133621 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.810839+0000", 2024-09-14T04:11:59.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034575180000000001 2024-09-14T04:11:59.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.810906+0000", 2024-09-14T04:11:59.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7020999999999994e-05 2024-09-14T04:11:59.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.434 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.811227+0000", 2024-09-14T04:11:59.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00032114300000000003 2024-09-14T04:11:59.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.434 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.434 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.126723+0000", 2024-09-14T04:11:59.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31549654399999999 2024-09-14T04:11:59.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.126759+0000", 2024-09-14T04:11:59.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5809999999999998e-05 2024-09-14T04:11:59.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.126856+0000", 2024-09-14T04:11:59.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6773000000000003e-05 2024-09-14T04:11:59.437 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.437 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.437 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.438 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221222 9.e 9:71e3d4ca:::benchmark_data_smithi114_153893_object91761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.438 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.075396+0000", 2024-09-14T04:11:59.438 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.32365525500001, 2024-09-14T04:11:59.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98772043600000003, 2024-09-14T04:11:59.438 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.438 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.438 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.439 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.439 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.439 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221222 2024-09-14T04:11:59.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.439 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:59.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075396+0000", 2024-09-14T04:11:59.441 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075396+0000", 2024-09-14T04:11:59.441 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075394+0000", 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075399+0000", 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2059999999999998e-06 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075414+0000", 2024-09-14T04:11:59.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4457999999999999e-05 2024-09-14T04:11:59.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075419+0000", 2024-09-14T04:11:59.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0590000000000002e-06 2024-09-14T04:11:59.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075463+0000", 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4357000000000002e-05 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075471+0000", 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5900000000000002e-06 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.444 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.054310+0000", 2024-09-14T04:11:59.445 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97883944700000003 2024-09-14T04:11:59.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.445 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.054405+0000", 2024-09-14T04:11:59.445 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4752000000000004e-05 2024-09-14T04:11:59.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.054480+0000", 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4501999999999999e-05 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.062905+0000", 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0084256130000000002 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.062914+0000", 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8750000000000006e-06 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063021+0000", 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106359 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063077+0000", 2024-09-14T04:11:59.447 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6490999999999998e-05 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063116+0000", 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9150999999999998e-05 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221279 9.e 9:7a0e57f8:::benchmark_data_smithi114_153893_object91818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.448 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.162273+0000", 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.23677762099999, 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90723786100000003, 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221279 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.449 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162273+0000", 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162273+0000", 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.450 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162271+0000", 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162279+0000", 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9810000000000003e-06 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162291+0000", 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1804000000000001e-05 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162297+0000", 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2790000000000004e-06 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162350+0000", 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2383999999999999e-05 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162358+0000", 2024-09-14T04:11:59.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.057426+0000", 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89506750499999999 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.057490+0000", 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4579999999999995e-05 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.453 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.057567+0000", 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6237999999999998e-05 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.062944+0000", 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0053775539999999997 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063126+0000", 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000181977 2024-09-14T04:11:59.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.455 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.455 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069407+0000", 2024-09-14T04:11:59.455 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062807540000000004 2024-09-14T04:11:59.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.455 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.455 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069473+0000", 2024-09-14T04:11:59.455 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6450000000000002e-05 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069511+0000", 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7894e-05 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221287 9.e 9:7c5f7cde:::benchmark_data_smithi114_153893_object91826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.456 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.168058+0000", 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.23099345600002, 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90162981099999995, 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221287 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.457 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168058+0000", 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168058+0000", 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168056+0000", 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168061+0000", 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8089999999999998e-06 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168075+0000", 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3091000000000001e-05 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168080+0000", 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7799999999999997e-06 2024-09-14T04:11:59.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168154+0000", 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3906999999999996e-05 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168162+0000", 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4819999999999997e-06 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.461 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.060288+0000", 2024-09-14T04:11:59.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89212670500000002 2024-09-14T04:11:59.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.461 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.461 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.060357+0000", 2024-09-14T04:11:59.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8893999999999995e-05 2024-09-14T04:11:59.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.461 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.060439+0000", 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.174e-05 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063132+0000", 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002692664 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069374+0000", 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062418459999999997 2024-09-14T04:11:59.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069617+0000", 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024384999999999999 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069658+0000", 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0562000000000002e-05 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069687+0000", 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9346999999999999e-05 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221290 9.e 9:7e953b68:::benchmark_data_smithi114_153893_object91829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.169237+0000", 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.22981410699998, 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90707892899999998, 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.464 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221290 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169237+0000", 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.465 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169237+0000", 2024-09-14T04:11:59.466 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169236+0000", 2024-09-14T04:11:59.466 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:59.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169242+0000", 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0349999999999998e-06 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169254+0000", 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2427e-05 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169260+0000", 2024-09-14T04:11:59.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9259999999999997e-06 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169302+0000", 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2283999999999997e-05 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169309+0000", 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0589999999999997e-06 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063175+0000", 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89386510500000005 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063285+0000", 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011004 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063400+0000", 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011540300000000001 2024-09-14T04:11:59.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069380+0000", 2024-09-14T04:11:59.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059798780000000001 2024-09-14T04:11:59.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069573+0000", 2024-09-14T04:11:59.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000193191 2024-09-14T04:11:59.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076249+0000", 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0066763270000000001 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076282+0000", 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2206999999999998e-05 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076316+0000", 2024-09-14T04:11:59.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4171000000000002e-05 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221308 9.e 9:7b1cb916:::benchmark_data_smithi114_153893_object91847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.194179+0000", 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.20487166499998, 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88227606199999997, 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.472 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.473 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.473 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.473 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221308 2024-09-14T04:11:59.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.473 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.473 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.473 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194179+0000", 2024-09-14T04:11:59.473 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194179+0000", 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194178+0000", 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194184+0000", 2024-09-14T04:11:59.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5339999999999999e-06 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194195+0000", 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1090000000000001e-05 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194200+0000", 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4079999999999997e-06 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194253+0000", 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3186999999999997e-05 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194260+0000", 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8920000000000003e-06 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.065682+0000", 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87142183299999998 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.065750+0000", 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7461999999999999e-05 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.065824+0000", 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4950999999999994e-05 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069695+0000", 2024-09-14T04:11:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0038707199999999998 2024-09-14T04:11:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069743+0000", 2024-09-14T04:11:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7444999999999997e-05 2024-09-14T04:11:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076385+0000", 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0066419080000000002 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076424+0000", 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9212000000000003e-05 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076455+0000", 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1605e-05 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221336 9.e 9:7fbc0f9f:::benchmark_data_smithi114_153893_object91875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.17721478800001, 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86189694800000005, 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.480 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.481 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.481 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.481 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.481 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221336 2024-09-14T04:11:59.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.481 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:11:59.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:11:59.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221839+0000", 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8779999999999998e-06 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221844+0000", 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9030000000000001e-06 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221848+0000", 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.506e-06 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221883+0000", 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5491000000000003e-05 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221888+0000", 2024-09-14T04:11:59.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7690000000000004e-06 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069784+0000", 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84789569300000001 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069903+0000", 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119528 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.070022+0000", 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118662 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076341+0000", 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0063191180000000003 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076464+0000", 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012243599999999999 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083657+0000", 2024-09-14T04:11:59.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0071937420000000004 2024-09-14T04:11:59.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083700+0000", 2024-09-14T04:11:59.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2095000000000003e-05 2024-09-14T04:11:59.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083733+0000", 2024-09-14T04:11:59.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3596000000000002e-05 2024-09-14T04:11:59.488 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.488 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.489 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.489 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221353 9.e 9:7a884025:::benchmark_data_smithi114_153893_object91892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.489 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.247072+0000", 2024-09-14T04:11:59.489 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.15197856600003, 2024-09-14T04:11:59.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85085830500000004, 2024-09-14T04:11:59.489 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.490 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.490 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.490 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221353 2024-09-14T04:11:59.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.490 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247072+0000", 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247072+0000", 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247071+0000", 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247078+0000", 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6290000000000004e-06 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247089+0000", 2024-09-14T04:11:59.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1168999999999999e-05 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247093+0000", 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.578e-06 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247140+0000", 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6113999999999999e-05 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247148+0000", 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1839999999999999e-06 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.073043+0000", 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82589522500000001 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.073170+0000", 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000127541 2024-09-14T04:11:59.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.073311+0000", 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014082 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083510+0000", 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010198239 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083573+0000", 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3004999999999997e-05 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097818+0000", 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014245821000000001 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097866+0000", 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8029999999999999e-05 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097931+0000", 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4312000000000004e-05 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221359 9.e 9:76f61d7d:::benchmark_data_smithi114_153893_object91898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.497 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.252688+0000", 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.14636263800003, 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84535683900000003, 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221359 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252688+0000", 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252688+0000", 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252687+0000", 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252696+0000", 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5210000000000008e-06 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252709+0000", 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2852e-05 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252716+0000", 2024-09-14T04:11:59.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5749999999999997e-06 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252760+0000", 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.405e-05 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252768+0000", 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0490000000000004e-06 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076490+0000", 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82372204199999999 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076560+0000", 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9900999999999994e-05 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076623+0000", 2024-09-14T04:11:59.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3226999999999996e-05 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083562+0000", 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0069392339999999999 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083593+0000", 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0880999999999998e-05 2024-09-14T04:11:59.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097986+0000", 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014392922000000001 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098017+0000", 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1362000000000002e-05 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098045+0000", 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7875999999999999e-05 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221381 9.e 9:70816107:::benchmark_data_smithi114_153893_object91920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.278485+0000", 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.120565489, 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81964557199999999, 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.505 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221381 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278485+0000", 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278485+0000", 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278484+0000", 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278490+0000", 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5910000000000003e-06 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278502+0000", 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2756e-05 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278508+0000", 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3410000000000002e-06 2024-09-14T04:11:59.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278558+0000", 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9959000000000001e-05 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278564+0000", 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3319999999999997e-06 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.080018+0000", 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80145439200000002 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.080117+0000", 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8821999999999997e-05 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.080228+0000", 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111292 2024-09-14T04:11:59.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083771+0000", 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003542842 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097698+0000", 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013926565 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098075+0000", 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037682900000000002 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098104+0000", 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9519e-05 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098131+0000", 2024-09-14T04:11:59.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6872999999999998e-05 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221383 9.e 9:78a714d0:::benchmark_data_smithi114_153893_object91922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.282793+0000", 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.11625806400002, 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82646461900000001, 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.513 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.514 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.514 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221383 2024-09-14T04:11:59.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.514 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282793+0000", 2024-09-14T04:11:59.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282793+0000", 2024-09-14T04:11:59.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282791+0000", 2024-09-14T04:11:59.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:59.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282797+0000", 2024-09-14T04:11:59.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5149999999999997e-06 2024-09-14T04:11:59.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282808+0000", 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1358e-05 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282820+0000", 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1313e-05 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282857+0000", 2024-09-14T04:11:59.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6928000000000003e-05 2024-09-14T04:11:59.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282862+0000", 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.677e-06 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083790+0000", 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80092809499999995 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083893+0000", 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103038 2024-09-14T04:11:59.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083996+0000", 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010259600000000001 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097702+0000", 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013706476 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097754+0000", 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1768000000000003e-05 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109094+0000", 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011340195000000001 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109221+0000", 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000127019 2024-09-14T04:11:59.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109258+0000", 2024-09-14T04:11:59.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6075000000000003e-05 2024-09-14T04:11:59.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.522 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221421 9.e 9:7e5137ba:::benchmark_data_smithi114_153893_object91960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.359708+0000", 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.039343084, 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.74967277499999996, 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221421 2024-09-14T04:11:59.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359708+0000", 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359708+0000", 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359706+0000", 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359713+0000", 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5989999999999998e-06 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359732+0000", 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9040000000000001e-05 2024-09-14T04:11:59.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359738+0000", 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.404e-06 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359780+0000", 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1572000000000001e-05 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359787+0000", 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4490000000000002e-06 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.087250+0000", 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72746270599999996 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.087357+0000", 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010766 2024-09-14T04:11:59.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.087467+0000", 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109231 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097707+0000", 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010240383 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097786+0000", 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9331000000000003e-05 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109308+0000", 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011522107 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109345+0000", 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6078999999999998e-05 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109381+0000", 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6149000000000002e-05 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221427 9.e 9:73b3c5ab:::benchmark_data_smithi114_153893_object91966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.371115+0000", 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.02793556400002, 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.73836333300000001, 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.531 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221427 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371115+0000", 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371115+0000", 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371114+0000", 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371120+0000", 2024-09-14T04:11:59.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3069999999999999e-06 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371134+0000", 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4318000000000001e-05 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371141+0000", 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9890000000000003e-06 2024-09-14T04:11:59.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371193+0000", 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1620999999999999e-05 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371201+0000", 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2719999999999997e-06 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.090232+0000", 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71903041000000001 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.090303+0000", 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1452000000000006e-05 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.090372+0000", 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8422000000000003e-05 2024-09-14T04:11:59.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097760+0000", 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.007388309 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097961+0000", 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000200619 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109420+0000", 2024-09-14T04:11:59.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011458983000000001 2024-09-14T04:11:59.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109451+0000", 2024-09-14T04:11:59.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1189999999999998e-05 2024-09-14T04:11:59.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109479+0000", 2024-09-14T04:11:59.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8042e-05 2024-09-14T04:11:59.539 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.539 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.540 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221428 9.e 9:7f963fb1:::benchmark_data_smithi114_153893_object91967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.540 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.372209+0000", 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.026842375, 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.73735522099999995, 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221428 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372209+0000", 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372209+0000", 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372208+0000", 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:11:59.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372213+0000", 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4999999999999999e-06 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372227+0000", 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3754000000000001e-05 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372233+0000", 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7749999999999998e-06 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372281+0000", 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8371999999999997e-05 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372288+0000", 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.306e-06 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.093512+0000", 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72122381700000004 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.093613+0000", 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100993 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.093719+0000", 2024-09-14T04:11:59.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105808 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105360+0000", 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01164106 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105400+0000", 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0420999999999998e-05 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109507+0000", 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041070669999999998 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109537+0000", 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9247999999999998e-05 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109564+0000", 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7118e-05 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.547 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221441 9.e 9:79734224:::benchmark_data_smithi114_153893_object91980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.396588+0000", 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.00246342299999, 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71322003, 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.548 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221441 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396588+0000", 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396588+0000", 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396586+0000", 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396591+0000", 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9289999999999997e-06 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396602+0000", 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0547000000000001e-05 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396607+0000", 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1000000000000003e-06 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396649+0000", 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1941999999999999e-05 2024-09-14T04:11:59.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396656+0000", 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8369999999999999e-06 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098153+0000", 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70149738800000006 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098264+0000", 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011047100000000001 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098389+0000", 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000125053 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109038+0000", 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010649614 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109276+0000", 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000237999 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109715+0000", 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043909999999999999 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109776+0000", 2024-09-14T04:11:59.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0977999999999998e-05 2024-09-14T04:11:59.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109808+0000", 2024-09-14T04:11:59.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1319000000000003e-05 2024-09-14T04:11:59.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.555 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221442 9.e 9:7a9a1506:::benchmark_data_smithi114_153893_object91981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.396654+0000", 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 317.00239731900001, 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72359706499999998, 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.556 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221442 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396654+0000", 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396654+0000", 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396653+0000", 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396656+0000", 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0419999999999998e-06 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396667+0000", 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1181e-05 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396671+0000", 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9879999999999998e-06 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396699+0000", 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7458e-05 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396704+0000", 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0039999999999999e-06 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.101911+0000", 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70520688399999998 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.102000+0000", 2024-09-14T04:11:59.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9349000000000001e-05 2024-09-14T04:11:59.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.102100+0000", 2024-09-14T04:11:59.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9623999999999993e-05 2024-09-14T04:11:59.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109387+0000", 2024-09-14T04:11:59.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0072876240000000004 2024-09-14T04:11:59.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.117933+0000", 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0085459390000000007 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120171+0000", 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022373919999999999 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120213+0000", 2024-09-14T04:11:59.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2882e-05 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120251+0000", 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7323000000000002e-05 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221473 9.e 9:74ac708e:::benchmark_data_smithi114_153893_object92012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.530644+0000", 2024-09-14T04:11:59.567 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 316.86840718000002, 2024-09-14T04:11:59.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.58970111199999997, 2024-09-14T04:11:59.568 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:11:59.568 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:11:59.568 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:11:59.568 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:11:59.568 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:11:59.568 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221473 2024-09-14T04:11:59.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.569 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:11:59.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:11:59.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530644+0000", 2024-09-14T04:11:59.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:11:59.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530644+0000", 2024-09-14T04:11:59.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:11:59.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:11:59.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530643+0000", 2024-09-14T04:11:59.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:11:59.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530649+0000", 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1480000000000003e-06 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530660+0000", 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1765e-05 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:11:59.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530666+0000", 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7529999999999998e-06 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530701+0000", 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4487999999999999e-05 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530706+0000", 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4689999999999996e-06 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105426+0000", 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.57472014800000004 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105530+0000", 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103225 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105641+0000", 2024-09-14T04:11:59.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011111399999999999 2024-09-14T04:11:59.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:11:59.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109392+0000", 2024-09-14T04:11:59.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037509470000000001 2024-09-14T04:11:59.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.117959+0000", 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.008567346 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120281+0000", 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023216310000000002 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120314+0000", 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3865000000000002e-05 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:11:59.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:11:59.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:11:59.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120345+0000", 2024-09-14T04:11:59.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0487000000000001e-05 2024-09-14T04:11:59.576 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.576 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.576 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.576 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:11:59.576 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:11:59.576 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:11:59.577 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:11:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:59.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:59.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:59.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:11:59.726 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:59.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:11:59.742 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:11:59.743 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:11:59.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:11:59.859 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:11:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:11:59.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:11:59.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:00.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:00.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:00.080 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 120193 objects 2024-09-14T04:12:00.080 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :146.812 2024-09-14T04:12:00.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:00.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:00.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:00.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:00 smithi160 ceph-mon[88407]: pgmap v847: 89 pgs: 89 active+clean; 250 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 805 op/s 2024-09-14T04:12:00.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:00 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:00.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:00 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:12:00.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:00 smithi160 ceph-mon[88407]: osdmap e228: 8 total, 8 up, 7 in 2024-09-14T04:12:00.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:00.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:00.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:00.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:00 smithi114 ceph-mon[106969]: pgmap v847: 89 pgs: 89 active+clean; 250 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 805 op/s 2024-09-14T04:12:00.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:00 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:00.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:00 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:12:00.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:00 smithi114 ceph-mon[106969]: osdmap e228: 8 total, 8 up, 7 in 2024-09-14T04:12:00.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:00 smithi114 ceph-mon[103029]: pgmap v847: 89 pgs: 89 active+clean; 250 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 805 op/s 2024-09-14T04:12:00.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:00 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:00.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:00 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:12:00.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:00 smithi114 ceph-mon[103029]: osdmap e228: 8 total, 8 up, 7 in 2024-09-14T04:12:00.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:00.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:00.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:00.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:00.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:00.493 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:00.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:00.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:00.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:00.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:12:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:12:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:12:00.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:00.739 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:00.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:00.765 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":848,"stamp":"2024-09-14T04:11:59.070339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":147658725,"num_objects":2271,"num_object_clones":0,"num_object_copies":6813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2271,"num_whiteouts":0,"num_read":307,"num_read_kb":760,"num_write":238355,"num_write_kb":7694012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":595258,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130201,"ondisk_log_size":130201,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5646208,"kb_used_data":605960,"kb_used_omap":27,"kb_used_meta":5039780,"kb_avail":650569856,"statfs":{"total":671965249536,"available":666183532544,"internally_reserved":0,"allocated":620503040,"data_stored":613146079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28483,"internal_metadata":5160734909},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,15,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-532414464,"num_objects":-8124,"num_object_clones":0,"num_object_copies":-24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.051501"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":262,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.059207+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:11:49.059207+0000","last_peered":"2024-09-14T04:11:49.059207+0000","last_clean":"2024-09-14T04:11:49.059207+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:11:49.059207+0000","last_undegraded":"2024-09-14T04:11:49.059207+0000","last_fullsized":"2024-09-14T04:11:49.059207+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":210,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981181+0000","last_change":"2024-09-14T04:05:55.701296+0000","last_active":"2024-09-14T04:11:38.981181+0000","last_peered":"2024-09-14T04:11:38.981181+0000","last_clean":"2024-09-14T04:11:38.981181+0000","last_became_active":"2024-09-14T04:05:55.676634+0000","last_became_peered":"2024-09-14T04:05:55.676634+0000","last_unstale":"2024-09-14T04:11:38.981181+0000","last_undegraded":"2024-09-14T04:11:38.981181+0000","last_fullsized":"2024-09-14T04:11:38.981181+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:44.187804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":192,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.970004+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:11:38.970004+0000","last_peered":"2024-09-14T04:11:38.970004+0000","last_clean":"2024-09-14T04:11:38.970004+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:11:38.970004+0000","last_undegraded":"2024-09-14T04:11:38.970004+0000","last_fullsized":"2024-09-14T04:11:38.970004+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.043171+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:11:49.043171+0000","last_peered":"2024-09-14T04:11:49.043171+0000","last_clean":"2024-09-14T04:11:49.043171+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:11:49.043171+0000","last_undegraded":"2024-09-14T04:11:49.043171+0000","last_fullsized":"2024-09-14T04:11:49.043171+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":333,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.046777+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:11:49.046777+0000","last_peered":"2024-09-14T04:11:49.046777+0000","last_clean":"2024-09-14T04:11:49.046777+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:11:49.046777+0000","last_undegraded":"2024-09-14T04:11:49.046777+0000","last_fullsized":"2024-09-14T04:11:49.046777+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":185,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981049+0000","last_change":"2024-09-14T04:05:55.676850+0000","last_active":"2024-09-14T04:11:38.981049+0000","last_peered":"2024-09-14T04:11:38.981049+0000","last_clean":"2024-09-14T04:11:38.981049+0000","last_became_active":"2024-09-14T04:05:55.676337+0000","last_became_peered":"2024-09-14T04:05:55.676337+0000","last_unstale":"2024-09-14T04:11:38.981049+0000","last_undegraded":"2024-09-14T04:11:38.981049+0000","last_fullsized":"2024-09-14T04:11:38.981049+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:12.377665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.058042+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:11:49.058042+0000","last_peered":"2024-09-14T04:11:49.058042+0000","last_clean":"2024-09-14T04:11:49.058042+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:11:49.058042+0000","last_undegraded":"2024-09-14T04:11:49.058042+0000","last_fullsized":"2024-09-14T04:11:49.058042+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":201,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.971269+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:11:38.971269+0000","last_peered":"2024-09-14T04:11:38.971269+0000","last_clean":"2024-09-14T04:11:38.971269+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:11:38.971269+0000","last_undegraded":"2024-09-14T04:11:38.971269+0000","last_fullsized":"2024-09-14T04:11:38.971269+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"227'14684","reported_seq":14720,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:54.366254+0000","last_change":"2024-09-14T04:07:55.368334+0000","last_active":"2024-09-14T04:11:54.366254+0000","last_peered":"2024-09-14T04:11:54.366254+0000","last_clean":"2024-09-14T04:11:54.366254+0000","last_became_active":"2024-09-14T04:07:55.368016+0000","last_became_peered":"2024-09-14T04:07:55.368016+0000","last_unstale":"2024-09-14T04:11:54.366254+0000","last_undegraded":"2024-09-14T04:11:54.366254+0000","last_fullsized":"2024-09-14T04:11:54.366254+0000","mapping_epoch":199,"log_start":"207'4600","ondisk_log_start":"207'4600","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:13.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18087936,"num_objects":276,"num_object_clones":0,"num_object_copies":828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14684,"num_write_kb":478720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":207,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981931+0000","last_change":"2024-09-14T04:05:55.704897+0000","last_active":"2024-09-14T04:11:38.981931+0000","last_peered":"2024-09-14T04:11:38.981931+0000","last_clean":"2024-09-14T04:11:38.981931+0000","last_became_active":"2024-09-14T04:05:55.681093+0000","last_became_peered":"2024-09-14T04:05:55.681093+0000","last_unstale":"2024-09-14T04:11:38.981931+0000","last_undegraded":"2024-09-14T04:11:38.981931+0000","last_fullsized":"2024-09-14T04:11:38.981931+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:49.647801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":175,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969564+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:11:38.969564+0000","last_peered":"2024-09-14T04:11:38.969564+0000","last_clean":"2024-09-14T04:11:38.969564+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:11:38.969564+0000","last_undegraded":"2024-09-14T04:11:38.969564+0000","last_fullsized":"2024-09-14T04:11:38.969564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"227'15129","reported_seq":15165,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:58.835331+0000","last_change":"2024-09-14T04:07:55.607517+0000","last_active":"2024-09-14T04:11:58.835331+0000","last_peered":"2024-09-14T04:11:58.835331+0000","last_clean":"2024-09-14T04:11:58.835331+0000","last_became_active":"2024-09-14T04:07:55.607208+0000","last_became_peered":"2024-09-14T04:07:55.607208+0000","last_unstale":"2024-09-14T04:11:58.835331+0000","last_undegraded":"2024-09-14T04:11:58.835331+0000","last_fullsized":"2024-09-14T04:11:58.835331+0000","mapping_epoch":199,"log_start":"210'7600","ondisk_log_start":"210'7600","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7529,"log_dups_size":0,"ondisk_log_size":7529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:32:20.123209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5439488,"num_objects":83,"num_object_clones":0,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15129,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":208,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981739+0000","last_change":"2024-09-14T04:05:55.702981+0000","last_active":"2024-09-14T04:11:38.981739+0000","last_peered":"2024-09-14T04:11:38.981739+0000","last_clean":"2024-09-14T04:11:38.981739+0000","last_became_active":"2024-09-14T04:05:55.679789+0000","last_became_peered":"2024-09-14T04:05:55.679789+0000","last_unstale":"2024-09-14T04:11:38.981739+0000","last_undegraded":"2024-09-14T04:11:38.981739+0000","last_fullsized":"2024-09-14T04:11:38.981739+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:34:24.358644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":252,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.032084+0000","last_change":"2024-09-14T04:07:44.340689+0000","last_active":"2024-09-14T04:11:49.032084+0000","last_peered":"2024-09-14T04:11:49.032084+0000","last_clean":"2024-09-14T04:11:49.032084+0000","last_became_active":"2024-09-14T04:07:44.340112+0000","last_became_peered":"2024-09-14T04:07:44.340112+0000","last_unstale":"2024-09-14T04:11:49.032084+0000","last_undegraded":"2024-09-14T04:11:49.032084+0000","last_fullsized":"2024-09-14T04:11:49.032084+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:04.263233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"227'14881","reported_seq":14949,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:58.651804+0000","last_change":"2024-09-14T04:07:55.608111+0000","last_active":"2024-09-14T04:11:58.651804+0000","last_peered":"2024-09-14T04:11:58.651804+0000","last_clean":"2024-09-14T04:11:58.651804+0000","last_became_active":"2024-09-14T04:07:55.607646+0000","last_became_peered":"2024-09-14T04:07:55.607646+0000","last_unstale":"2024-09-14T04:11:58.651804+0000","last_undegraded":"2024-09-14T04:11:58.651804+0000","last_fullsized":"2024-09-14T04:11:58.651804+0000","mapping_epoch":199,"log_start":"210'7900","ondisk_log_start":"210'7900","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":6981,"log_dups_size":0,"ondisk_log_size":6981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:02:33.076539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4390912,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14881,"num_write_kb":478336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":176,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981529+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:11:38.981529+0000","last_peered":"2024-09-14T04:11:38.981529+0000","last_clean":"2024-09-14T04:11:38.981529+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:11:38.981529+0000","last_undegraded":"2024-09-14T04:11:38.981529+0000","last_fullsized":"2024-09-14T04:11:38.981529+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":202,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.971503+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:11:38.971503+0000","last_peered":"2024-09-14T04:11:38.971503+0000","last_clean":"2024-09-14T04:11:38.971503+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:11:38.971503+0000","last_undegraded":"2024-09-14T04:11:38.971503+0000","last_fullsized":"2024-09-14T04:11:38.971503+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"227'14897","reported_seq":14965,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:55.746931+0000","last_change":"2024-09-14T04:07:55.607545+0000","last_active":"2024-09-14T04:11:55.746931+0000","last_peered":"2024-09-14T04:11:55.746931+0000","last_clean":"2024-09-14T04:11:55.746931+0000","last_became_active":"2024-09-14T04:07:55.607263+0000","last_became_peered":"2024-09-14T04:07:55.607263+0000","last_unstale":"2024-09-14T04:11:55.746931+0000","last_undegraded":"2024-09-14T04:11:55.746931+0000","last_fullsized":"2024-09-14T04:11:55.746931+0000","mapping_epoch":199,"log_start":"207'4800","ondisk_log_start":"207'4800","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:38:52.910168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13172736,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14897,"num_write_kb":483136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":213,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969800+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:11:38.969800+0000","last_peered":"2024-09-14T04:11:38.969800+0000","last_clean":"2024-09-14T04:11:38.969800+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:11:38.969800+0000","last_undegraded":"2024-09-14T04:11:38.969800+0000","last_fullsized":"2024-09-14T04:11:38.969800+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":361,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.042140+0000","last_change":"2024-09-14T04:07:54.363033+0000","last_active":"2024-09-14T04:11:49.042140+0000","last_peered":"2024-09-14T04:11:49.042140+0000","last_clean":"2024-09-14T04:11:49.042140+0000","last_became_active":"2024-09-14T04:07:54.362706+0000","last_became_peered":"2024-09-14T04:07:54.362706+0000","last_unstale":"2024-09-14T04:11:49.042140+0000","last_undegraded":"2024-09-14T04:11:49.042140+0000","last_fullsized":"2024-09-14T04:11:49.042140+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:31.193597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"227'14996","reported_seq":15064,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:57.834380+0000","last_change":"2024-09-14T04:07:55.607327+0000","last_active":"2024-09-14T04:11:57.834380+0000","last_peered":"2024-09-14T04:11:57.834380+0000","last_clean":"2024-09-14T04:11:57.834380+0000","last_became_active":"2024-09-14T04:07:55.606967+0000","last_became_peered":"2024-09-14T04:07:55.606967+0000","last_unstale":"2024-09-14T04:11:57.834380+0000","last_undegraded":"2024-09-14T04:11:57.834380+0000","last_fullsized":"2024-09-14T04:11:57.834380+0000","mapping_epoch":199,"log_start":"211'8400","ondisk_log_start":"211'8400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:36.618823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14996,"num_write_kb":483712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":301,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.053705+0000","last_change":"2024-09-14T04:05:55.695825+0000","last_active":"2024-09-14T04:11:49.053705+0000","last_peered":"2024-09-14T04:11:49.053705+0000","last_clean":"2024-09-14T04:11:49.053705+0000","last_became_active":"2024-09-14T04:05:55.680668+0000","last_became_peered":"2024-09-14T04:05:55.680668+0000","last_unstale":"2024-09-14T04:11:49.053705+0000","last_undegraded":"2024-09-14T04:11:49.053705+0000","last_fullsized":"2024-09-14T04:11:49.053705+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:30.262947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":221,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.059609+0000","last_change":"2024-09-14T04:05:55.677723+0000","last_active":"2024-09-14T04:11:49.059609+0000","last_peered":"2024-09-14T04:11:49.059609+0000","last_clean":"2024-09-14T04:11:49.059609+0000","last_became_active":"2024-09-14T04:05:55.677254+0000","last_became_peered":"2024-09-14T04:05:55.677254+0000","last_unstale":"2024-09-14T04:11:49.059609+0000","last_undegraded":"2024-09-14T04:11:49.059609+0000","last_fullsized":"2024-09-14T04:11:49.059609+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:53.163120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"227'14742","reported_seq":14810,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:55.812802+0000","last_change":"2024-09-14T04:07:55.607260+0000","last_active":"2024-09-14T04:11:55.812802+0000","last_peered":"2024-09-14T04:11:55.812802+0000","last_clean":"2024-09-14T04:11:55.812802+0000","last_became_active":"2024-09-14T04:07:55.606973+0000","last_became_peered":"2024-09-14T04:07:55.606973+0000","last_unstale":"2024-09-14T04:11:55.812802+0000","last_undegraded":"2024-09-14T04:11:55.812802+0000","last_fullsized":"2024-09-14T04:11:55.812802+0000","mapping_epoch":199,"log_start":"207'4700","ondisk_log_start":"207'4700","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:59:15.715573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14417920,"num_objects":220,"num_object_clones":0,"num_object_copies":660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14742,"num_write_kb":478784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":203,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.971643+0000","last_change":"2024-09-14T04:05:55.679201+0000","last_active":"2024-09-14T04:11:38.971643+0000","last_peered":"2024-09-14T04:11:38.971643+0000","last_clean":"2024-09-14T04:11:38.971643+0000","last_became_active":"2024-09-14T04:05:55.678843+0000","last_became_peered":"2024-09-14T04:05:55.678843+0000","last_unstale":"2024-09-14T04:11:38.971643+0000","last_undegraded":"2024-09-14T04:11:38.971643+0000","last_fullsized":"2024-09-14T04:11:38.971643+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:12:57.418514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":212,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.039556+0000","last_change":"2024-09-14T04:05:55.673650+0000","last_active":"2024-09-14T04:11:49.039556+0000","last_peered":"2024-09-14T04:11:49.039556+0000","last_clean":"2024-09-14T04:11:49.039556+0000","last_became_active":"2024-09-14T04:05:55.673258+0000","last_became_peered":"2024-09-14T04:05:55.673258+0000","last_unstale":"2024-09-14T04:11:49.039556+0000","last_undegraded":"2024-09-14T04:11:49.039556+0000","last_fullsized":"2024-09-14T04:11:49.039556+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:21.089797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"227'14901","reported_seq":14937,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:58.737712+0000","last_change":"2024-09-14T04:07:55.365272+0000","last_active":"2024-09-14T04:11:58.737712+0000","last_peered":"2024-09-14T04:11:58.737712+0000","last_clean":"2024-09-14T04:11:58.737712+0000","last_became_active":"2024-09-14T04:07:55.364722+0000","last_became_peered":"2024-09-14T04:07:55.364722+0000","last_unstale":"2024-09-14T04:11:58.737712+0000","last_undegraded":"2024-09-14T04:11:58.737712+0000","last_fullsized":"2024-09-14T04:11:58.737712+0000","mapping_epoch":199,"log_start":"210'7400","ondisk_log_start":"210'7400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:40:55.470085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14901,"num_write_kb":478656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":196,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.036851+0000","last_change":"2024-09-14T04:05:55.671329+0000","last_active":"2024-09-14T04:11:49.036851+0000","last_peered":"2024-09-14T04:11:49.036851+0000","last_clean":"2024-09-14T04:11:49.036851+0000","last_became_active":"2024-09-14T04:05:55.671032+0000","last_became_peered":"2024-09-14T04:05:55.671032+0000","last_unstale":"2024-09-14T04:11:49.036851+0000","last_undegraded":"2024-09-14T04:11:49.036851+0000","last_fullsized":"2024-09-14T04:11:49.036851+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:58:10.779452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":124,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.971511+0000","last_change":"2024-09-14T04:05:55.679870+0000","last_active":"2024-09-14T04:11:38.971511+0000","last_peered":"2024-09-14T04:11:38.971511+0000","last_clean":"2024-09-14T04:11:38.971511+0000","last_became_active":"2024-09-14T04:05:55.679574+0000","last_became_peered":"2024-09-14T04:05:55.679574+0000","last_unstale":"2024-09-14T04:11:38.971511+0000","last_undegraded":"2024-09-14T04:11:38.971511+0000","last_fullsized":"2024-09-14T04:11:38.971511+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:51:09.931574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"227'14701","reported_seq":14737,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:54.180326+0000","last_change":"2024-09-14T04:07:55.367126+0000","last_active":"2024-09-14T04:11:54.180326+0000","last_peered":"2024-09-14T04:11:54.180326+0000","last_clean":"2024-09-14T04:11:54.180326+0000","last_became_active":"2024-09-14T04:07:55.366798+0000","last_became_peered":"2024-09-14T04:07:55.366798+0000","last_unstale":"2024-09-14T04:11:54.180326+0000","last_undegraded":"2024-09-14T04:11:54.180326+0000","last_fullsized":"2024-09-14T04:11:54.180326+0000","mapping_epoch":199,"log_start":"209'6500","ondisk_log_start":"209'6500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:24:51.791784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18677760,"num_objects":285,"num_object_clones":0,"num_object_copies":855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14701,"num_write_kb":479552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.050427+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:11:49.050427+0000","last_peered":"2024-09-14T04:11:49.050427+0000","last_clean":"2024-09-14T04:11:49.050427+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:11:49.050427+0000","last_undegraded":"2024-09-14T04:11:49.050427+0000","last_fullsized":"2024-09-14T04:11:49.050427+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":305,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.038641+0000","last_change":"2024-09-14T04:05:55.678520+0000","last_active":"2024-09-14T04:11:49.038641+0000","last_peered":"2024-09-14T04:11:49.038641+0000","last_clean":"2024-09-14T04:11:49.038641+0000","last_became_active":"2024-09-14T04:05:55.678008+0000","last_became_peered":"2024-09-14T04:05:55.678008+0000","last_unstale":"2024-09-14T04:11:49.038641+0000","last_undegraded":"2024-09-14T04:11:49.038641+0000","last_fullsized":"2024-09-14T04:11:49.038641+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:40:26.778212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"227'14696","reported_seq":14732,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:58.850443+0000","last_change":"2024-09-14T04:07:55.367659+0000","last_active":"2024-09-14T04:11:58.850443+0000","last_peered":"2024-09-14T04:11:58.850443+0000","last_clean":"2024-09-14T04:11:58.850443+0000","last_became_active":"2024-09-14T04:07:55.367247+0000","last_became_peered":"2024-09-14T04:07:55.367247+0000","last_unstale":"2024-09-14T04:11:58.850443+0000","last_undegraded":"2024-09-14T04:11:58.850443+0000","last_fullsized":"2024-09-14T04:11:58.850443+0000","mapping_epoch":199,"log_start":"210'7100","ondisk_log_start":"210'7100","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7596,"log_dups_size":0,"ondisk_log_size":7596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:58:08.638370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":78,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14696,"num_write_kb":472768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":309,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.983830+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:11:38.983830+0000","last_peered":"2024-09-14T04:11:38.983830+0000","last_clean":"2024-09-14T04:11:38.983830+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:11:38.983830+0000","last_undegraded":"2024-09-14T04:11:38.983830+0000","last_fullsized":"2024-09-14T04:11:38.983830+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":206,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.973572+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:11:38.973572+0000","last_peered":"2024-09-14T04:11:38.973572+0000","last_clean":"2024-09-14T04:11:38.973572+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:11:38.973572+0000","last_undegraded":"2024-09-14T04:11:38.973572+0000","last_fullsized":"2024-09-14T04:11:38.973572+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":76,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.049053+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:11:49.049053+0000","last_peered":"2024-09-14T04:11:49.049053+0000","last_clean":"2024-09-14T04:11:49.049053+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:11:49.049053+0000","last_undegraded":"2024-09-14T04:11:49.049053+0000","last_fullsized":"2024-09-14T04:11:49.049053+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":185,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.972180+0000","last_change":"2024-09-14T04:05:55.676445+0000","last_active":"2024-09-14T04:11:38.972180+0000","last_peered":"2024-09-14T04:11:38.972180+0000","last_clean":"2024-09-14T04:11:38.972180+0000","last_became_active":"2024-09-14T04:05:55.674141+0000","last_became_peered":"2024-09-14T04:05:55.674141+0000","last_unstale":"2024-09-14T04:11:38.972180+0000","last_undegraded":"2024-09-14T04:11:38.972180+0000","last_fullsized":"2024-09-14T04:11:38.972180+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:28:50.348309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"227'15034","reported_seq":15102,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:58.686264+0000","last_change":"2024-09-14T04:07:55.366975+0000","last_active":"2024-09-14T04:11:58.686264+0000","last_peered":"2024-09-14T04:11:58.686264+0000","last_clean":"2024-09-14T04:11:58.686264+0000","last_became_active":"2024-09-14T04:07:55.366645+0000","last_became_peered":"2024-09-14T04:07:55.366645+0000","last_unstale":"2024-09-14T04:11:58.686264+0000","last_undegraded":"2024-09-14T04:11:58.686264+0000","last_fullsized":"2024-09-14T04:11:58.686264+0000","mapping_epoch":199,"log_start":"211'8000","ondisk_log_start":"211'8000","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7034,"log_dups_size":0,"ondisk_log_size":7034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:39:56.637518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15034,"num_write_kb":483008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":221,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.983831+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:11:38.983831+0000","last_peered":"2024-09-14T04:11:38.983831+0000","last_clean":"2024-09-14T04:11:38.983831+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:11:38.983831+0000","last_undegraded":"2024-09-14T04:11:38.983831+0000","last_fullsized":"2024-09-14T04:11:38.983831+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":355,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.037608+0000","last_change":"2024-09-14T04:07:44.337314+0000","last_active":"2024-09-14T04:11:49.037608+0000","last_peered":"2024-09-14T04:11:49.037608+0000","last_clean":"2024-09-14T04:11:49.037608+0000","last_became_active":"2024-09-14T04:07:44.336916+0000","last_became_peered":"2024-09-14T04:07:44.336916+0000","last_unstale":"2024-09-14T04:11:49.037608+0000","last_undegraded":"2024-09-14T04:11:49.037608+0000","last_fullsized":"2024-09-14T04:11:49.037608+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:23.153578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":345,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.046598+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:11:49.046598+0000","last_peered":"2024-09-14T04:11:49.046598+0000","last_clean":"2024-09-14T04:11:49.046598+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:11:49.046598+0000","last_undegraded":"2024-09-14T04:11:49.046598+0000","last_fullsized":"2024-09-14T04:11:49.046598+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"227'14772","reported_seq":14808,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:54.360485+0000","last_change":"2024-09-14T04:07:55.365448+0000","last_active":"2024-09-14T04:11:54.360485+0000","last_peered":"2024-09-14T04:11:54.360485+0000","last_clean":"2024-09-14T04:11:54.360485+0000","last_became_active":"2024-09-14T04:07:55.365168+0000","last_became_peered":"2024-09-14T04:07:55.365168+0000","last_unstale":"2024-09-14T04:11:54.360485+0000","last_undegraded":"2024-09-14T04:11:54.360485+0000","last_fullsized":"2024-09-14T04:11:54.360485+0000","mapping_epoch":199,"log_start":"207'4700","ondisk_log_start":"207'4700","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:25.997258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18087936,"num_objects":276,"num_object_clones":0,"num_object_copies":828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14772,"num_write_kb":481536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":338,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.036990+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:11:49.036990+0000","last_peered":"2024-09-14T04:11:49.036990+0000","last_clean":"2024-09-14T04:11:49.036990+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:11:49.036990+0000","last_undegraded":"2024-09-14T04:11:49.036990+0000","last_fullsized":"2024-09-14T04:11:49.036990+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":299,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.031782+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:11:49.031782+0000","last_peered":"2024-09-14T04:11:49.031782+0000","last_clean":"2024-09-14T04:11:49.031782+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:11:49.031782+0000","last_undegraded":"2024-09-14T04:11:49.031782+0000","last_fullsized":"2024-09-14T04:11:49.031782+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":69,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.036849+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:11:49.036849+0000","last_peered":"2024-09-14T04:11:49.036849+0000","last_clean":"2024-09-14T04:11:49.036849+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:11:49.036849+0000","last_undegraded":"2024-09-14T04:11:49.036849+0000","last_fullsized":"2024-09-14T04:11:49.036849+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"227'15204","reported_seq":15239,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:58.842685+0000","last_change":"2024-09-14T04:07:55.367851+0000","last_active":"2024-09-14T04:11:58.842685+0000","last_peered":"2024-09-14T04:11:58.842685+0000","last_clean":"2024-09-14T04:11:58.842685+0000","last_became_active":"2024-09-14T04:07:55.367146+0000","last_became_peered":"2024-09-14T04:07:55.367146+0000","last_unstale":"2024-09-14T04:11:58.842685+0000","last_undegraded":"2024-09-14T04:11:58.842685+0000","last_fullsized":"2024-09-14T04:11:58.842685+0000","mapping_epoch":199,"log_start":"210'7700","ondisk_log_start":"210'7700","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7504,"log_dups_size":0,"ondisk_log_size":7504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:26:46.495508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3276800,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15204,"num_write_kb":488128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":207,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969697+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:11:38.969697+0000","last_peered":"2024-09-14T04:11:38.969697+0000","last_clean":"2024-09-14T04:11:38.969697+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:11:38.969697+0000","last_undegraded":"2024-09-14T04:11:38.969697+0000","last_fullsized":"2024-09-14T04:11:38.969697+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":167,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.042441+0000","last_change":"2024-09-14T04:05:55.676693+0000","last_active":"2024-09-14T04:11:49.042441+0000","last_peered":"2024-09-14T04:11:49.042441+0000","last_clean":"2024-09-14T04:11:49.042441+0000","last_became_active":"2024-09-14T04:05:55.676335+0000","last_became_peered":"2024-09-14T04:05:55.676335+0000","last_unstale":"2024-09-14T04:11:49.042441+0000","last_undegraded":"2024-09-14T04:11:49.042441+0000","last_fullsized":"2024-09-14T04:11:49.042441+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:19:43.626016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":324,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.031942+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:11:49.031942+0000","last_peered":"2024-09-14T04:11:49.031942+0000","last_clean":"2024-09-14T04:11:49.031942+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:11:49.031942+0000","last_undegraded":"2024-09-14T04:11:49.031942+0000","last_fullsized":"2024-09-14T04:11:49.031942+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"227'14656","reported_seq":14692,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:54.199417+0000","last_change":"2024-09-14T04:07:55.607796+0000","last_active":"2024-09-14T04:11:54.199417+0000","last_peered":"2024-09-14T04:11:54.199417+0000","last_clean":"2024-09-14T04:11:54.199417+0000","last_became_active":"2024-09-14T04:07:55.607447+0000","last_became_peered":"2024-09-14T04:07:55.607447+0000","last_unstale":"2024-09-14T04:11:54.199417+0000","last_undegraded":"2024-09-14T04:11:54.199417+0000","last_fullsized":"2024-09-14T04:11:54.199417+0000","mapping_epoch":199,"log_start":"209'6500","ondisk_log_start":"209'6500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:09:44.307466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19922944,"num_objects":304,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14656,"num_write_kb":478720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":230,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.970183+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:11:38.970183+0000","last_peered":"2024-09-14T04:11:38.970183+0000","last_clean":"2024-09-14T04:11:38.970183+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:11:38.970183+0000","last_undegraded":"2024-09-14T04:11:38.970183+0000","last_fullsized":"2024-09-14T04:11:38.970183+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":338,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.032028+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:11:49.032028+0000","last_peered":"2024-09-14T04:11:49.032028+0000","last_clean":"2024-09-14T04:11:49.032028+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:11:49.032028+0000","last_undegraded":"2024-09-14T04:11:49.032028+0000","last_fullsized":"2024-09-14T04:11:49.032028+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":295,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.035988+0000","last_change":"2024-09-14T04:05:55.674511+0000","last_active":"2024-09-14T04:11:49.035988+0000","last_peered":"2024-09-14T04:11:49.035988+0000","last_clean":"2024-09-14T04:11:49.035988+0000","last_became_active":"2024-09-14T04:05:55.674182+0000","last_became_peered":"2024-09-14T04:05:55.674182+0000","last_unstale":"2024-09-14T04:11:49.035988+0000","last_undegraded":"2024-09-14T04:11:49.035988+0000","last_fullsized":"2024-09-14T04:11:49.035988+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:12:34.453647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"227'15001","reported_seq":15038,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:58.805689+0000","last_change":"2024-09-14T04:07:55.607679+0000","last_active":"2024-09-14T04:11:58.805689+0000","last_peered":"2024-09-14T04:11:58.805689+0000","last_clean":"2024-09-14T04:11:58.805689+0000","last_became_active":"2024-09-14T04:07:55.607338+0000","last_became_peered":"2024-09-14T04:07:55.607338+0000","last_unstale":"2024-09-14T04:11:58.805689+0000","last_undegraded":"2024-09-14T04:11:58.805689+0000","last_fullsized":"2024-09-14T04:11:58.805689+0000","mapping_epoch":199,"log_start":"210'7500","ondisk_log_start":"210'7500","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7501,"log_dups_size":0,"ondisk_log_size":7501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:02:20.945987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4325400,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15001,"num_write_kb":482113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":299,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.032153+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:11:49.032153+0000","last_peered":"2024-09-14T04:11:49.032153+0000","last_clean":"2024-09-14T04:11:49.032153+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:11:49.032153+0000","last_undegraded":"2024-09-14T04:11:49.032153+0000","last_fullsized":"2024-09-14T04:11:49.032153+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":214,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981522+0000","last_change":"2024-09-14T04:05:55.681238+0000","last_active":"2024-09-14T04:11:38.981522+0000","last_peered":"2024-09-14T04:11:38.981522+0000","last_clean":"2024-09-14T04:11:38.981522+0000","last_became_active":"2024-09-14T04:05:55.680946+0000","last_became_peered":"2024-09-14T04:05:55.680946+0000","last_unstale":"2024-09-14T04:11:38.981522+0000","last_undegraded":"2024-09-14T04:11:38.981522+0000","last_fullsized":"2024-09-14T04:11:38.981522+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:13:39.006545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":323,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.042222+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:11:49.042222+0000","last_peered":"2024-09-14T04:11:49.042222+0000","last_clean":"2024-09-14T04:11:49.042222+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:11:49.042222+0000","last_undegraded":"2024-09-14T04:11:49.042222+0000","last_fullsized":"2024-09-14T04:11:49.042222+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"227'14981","reported_seq":15017,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:58.815947+0000","last_change":"2024-09-14T04:07:55.366077+0000","last_active":"2024-09-14T04:11:58.815947+0000","last_peered":"2024-09-14T04:11:58.815947+0000","last_clean":"2024-09-14T04:11:58.815947+0000","last_became_active":"2024-09-14T04:07:55.365753+0000","last_became_peered":"2024-09-14T04:07:55.365753+0000","last_unstale":"2024-09-14T04:11:58.815947+0000","last_undegraded":"2024-09-14T04:11:58.815947+0000","last_fullsized":"2024-09-14T04:11:58.815947+0000","mapping_epoch":199,"log_start":"210'7400","ondisk_log_start":"210'7400","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7581,"log_dups_size":0,"ondisk_log_size":7581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:04:18.784725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14981,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":291,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.058178+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:11:49.058178+0000","last_peered":"2024-09-14T04:11:49.058178+0000","last_clean":"2024-09-14T04:11:49.058178+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:11:49.058178+0000","last_undegraded":"2024-09-14T04:11:49.058178+0000","last_fullsized":"2024-09-14T04:11:49.058178+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":306,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.031653+0000","last_change":"2024-09-14T04:05:55.676617+0000","last_active":"2024-09-14T04:11:49.031653+0000","last_peered":"2024-09-14T04:11:49.031653+0000","last_clean":"2024-09-14T04:11:49.031653+0000","last_became_active":"2024-09-14T04:05:55.676318+0000","last_became_peered":"2024-09-14T04:05:55.676318+0000","last_unstale":"2024-09-14T04:11:49.031653+0000","last_undegraded":"2024-09-14T04:11:49.031653+0000","last_fullsized":"2024-09-14T04:11:49.031653+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:38.529611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":38,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969445+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:11:38.969445+0000","last_peered":"2024-09-14T04:11:38.969445+0000","last_clean":"2024-09-14T04:11:38.969445+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:11:38.969445+0000","last_undegraded":"2024-09-14T04:11:38.969445+0000","last_fullsized":"2024-09-14T04:11:38.969445+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.f","version":"227'14868","reported_seq":14904,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:58.835254+0000","last_change":"2024-09-14T04:07:55.607496+0000","last_active":"2024-09-14T04:11:58.835254+0000","last_peered":"2024-09-14T04:11:58.835254+0000","last_clean":"2024-09-14T04:11:58.835254+0000","last_became_active":"2024-09-14T04:07:55.607124+0000","last_became_peered":"2024-09-14T04:07:55.607124+0000","last_unstale":"2024-09-14T04:11:58.835254+0000","last_undegraded":"2024-09-14T04:11:58.835254+0000","last_fullsized":"2024-09-14T04:11:58.835254+0000","mapping_epoch":199,"log_start":"210'7300","ondisk_log_start":"210'7300","created":199,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:07:54.347074+0000","last_clean_scrub_stamp":"2024-09-14T04:07:54.347074+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:15:10.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14868,"num_write_kb":477504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":348,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.046916+0000","last_change":"2024-09-14T04:05:55.690010+0000","last_active":"2024-09-14T04:11:49.046916+0000","last_peered":"2024-09-14T04:11:49.046916+0000","last_clean":"2024-09-14T04:11:49.046916+0000","last_became_active":"2024-09-14T04:05:55.670892+0000","last_became_peered":"2024-09-14T04:05:55.670892+0000","last_unstale":"2024-09-14T04:11:49.046916+0000","last_undegraded":"2024-09-14T04:11:49.046916+0000","last_fullsized":"2024-09-14T04:11:49.046916+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:40:33.024617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":205,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.972235+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:11:38.972235+0000","last_peered":"2024-09-14T04:11:38.972235+0000","last_clean":"2024-09-14T04:11:38.972235+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:11:38.972235+0000","last_undegraded":"2024-09-14T04:11:38.972235+0000","last_fullsized":"2024-09-14T04:11:38.972235+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":70,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969847+0000","last_change":"2024-09-14T04:05:55.674629+0000","last_active":"2024-09-14T04:11:38.969847+0000","last_peered":"2024-09-14T04:11:38.969847+0000","last_clean":"2024-09-14T04:11:38.969847+0000","last_became_active":"2024-09-14T04:05:55.674242+0000","last_became_peered":"2024-09-14T04:05:55.674242+0000","last_unstale":"2024-09-14T04:11:38.969847+0000","last_undegraded":"2024-09-14T04:11:38.969847+0000","last_fullsized":"2024-09-14T04:11:38.969847+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:06:16.827114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":83,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.971733+0000","last_change":"2024-09-14T04:05:55.677636+0000","last_active":"2024-09-14T04:11:38.971733+0000","last_peered":"2024-09-14T04:11:38.971733+0000","last_clean":"2024-09-14T04:11:38.971733+0000","last_became_active":"2024-09-14T04:05:55.677313+0000","last_became_peered":"2024-09-14T04:05:55.677313+0000","last_unstale":"2024-09-14T04:11:38.971733+0000","last_undegraded":"2024-09-14T04:11:38.971733+0000","last_fullsized":"2024-09-14T04:11:38.971733+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:00:13.489641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":123,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981276+0000","last_change":"2024-09-14T04:05:55.686083+0000","last_active":"2024-09-14T04:11:38.981276+0000","last_peered":"2024-09-14T04:11:38.981276+0000","last_clean":"2024-09-14T04:11:38.981276+0000","last_became_active":"2024-09-14T04:05:55.673710+0000","last_became_peered":"2024-09-14T04:05:55.673710+0000","last_unstale":"2024-09-14T04:11:38.981276+0000","last_undegraded":"2024-09-14T04:11:38.981276+0000","last_fullsized":"2024-09-14T04:11:38.981276+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:25.169218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":298,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.035829+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:11:49.035829+0000","last_peered":"2024-09-14T04:11:49.035829+0000","last_clean":"2024-09-14T04:11:49.035829+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:11:49.035829+0000","last_undegraded":"2024-09-14T04:11:49.035829+0000","last_fullsized":"2024-09-14T04:11:49.035829+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":211,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.981398+0000","last_change":"2024-09-14T04:05:55.698766+0000","last_active":"2024-09-14T04:11:38.981398+0000","last_peered":"2024-09-14T04:11:38.981398+0000","last_clean":"2024-09-14T04:11:38.981398+0000","last_became_active":"2024-09-14T04:05:55.674923+0000","last_became_peered":"2024-09-14T04:05:55.674923+0000","last_unstale":"2024-09-14T04:11:38.981398+0000","last_undegraded":"2024-09-14T04:11:38.981398+0000","last_fullsized":"2024-09-14T04:11:38.981398+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:52:37.679728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":193,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.971915+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:11:38.971915+0000","last_peered":"2024-09-14T04:11:38.971915+0000","last_clean":"2024-09-14T04:11:38.971915+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:11:38.971915+0000","last_undegraded":"2024-09-14T04:11:38.971915+0000","last_fullsized":"2024-09-14T04:11:38.971915+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":250,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.049122+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:11:49.049122+0000","last_peered":"2024-09-14T04:11:49.049122+0000","last_clean":"2024-09-14T04:11:49.049122+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:11:49.049122+0000","last_undegraded":"2024-09-14T04:11:49.049122+0000","last_fullsized":"2024-09-14T04:11:49.049122+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.046343+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:11:49.046343+0000","last_peered":"2024-09-14T04:11:49.046343+0000","last_clean":"2024-09-14T04:11:49.046343+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:11:49.046343+0000","last_undegraded":"2024-09-14T04:11:49.046343+0000","last_fullsized":"2024-09-14T04:11:49.046343+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":262,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.050503+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:11:49.050503+0000","last_peered":"2024-09-14T04:11:49.050503+0000","last_clean":"2024-09-14T04:11:49.050503+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:11:49.050503+0000","last_undegraded":"2024-09-14T04:11:49.050503+0000","last_fullsized":"2024-09-14T04:11:49.050503+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":202,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.971981+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:11:38.971981+0000","last_peered":"2024-09-14T04:11:38.971981+0000","last_clean":"2024-09-14T04:11:38.971981+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:11:38.971981+0000","last_undegraded":"2024-09-14T04:11:38.971981+0000","last_fullsized":"2024-09-14T04:11:38.971981+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.050337+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:11:49.050337+0000","last_peered":"2024-09-14T04:11:49.050337+0000","last_clean":"2024-09-14T04:11:49.050337+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:11:49.050337+0000","last_undegraded":"2024-09-14T04:11:49.050337+0000","last_fullsized":"2024-09-14T04:11:49.050337+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":348,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.046698+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:11:49.046698+0000","last_peered":"2024-09-14T04:11:49.046698+0000","last_clean":"2024-09-14T04:11:49.046698+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:11:49.046698+0000","last_undegraded":"2024-09-14T04:11:49.046698+0000","last_fullsized":"2024-09-14T04:11:49.046698+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":305,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.032087+0000","last_change":"2024-09-14T04:05:55.677962+0000","last_active":"2024-09-14T04:11:49.032087+0000","last_peered":"2024-09-14T04:11:49.032087+0000","last_clean":"2024-09-14T04:11:49.032087+0000","last_became_active":"2024-09-14T04:05:55.677277+0000","last_became_peered":"2024-09-14T04:05:55.677277+0000","last_unstale":"2024-09-14T04:11:49.032087+0000","last_undegraded":"2024-09-14T04:11:49.032087+0000","last_fullsized":"2024-09-14T04:11:49.032087+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:24.908568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.057620+0000","last_change":"2024-09-14T04:07:44.338546+0000","last_active":"2024-09-14T04:11:49.057620+0000","last_peered":"2024-09-14T04:11:49.057620+0000","last_clean":"2024-09-14T04:11:49.057620+0000","last_became_active":"2024-09-14T04:07:44.338210+0000","last_became_peered":"2024-09-14T04:07:44.338210+0000","last_unstale":"2024-09-14T04:11:49.057620+0000","last_undegraded":"2024-09-14T04:11:49.057620+0000","last_fullsized":"2024-09-14T04:11:49.057620+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:56.100013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":129,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.036077+0000","last_change":"2024-09-14T04:05:55.671091+0000","last_active":"2024-09-14T04:11:49.036077+0000","last_peered":"2024-09-14T04:11:49.036077+0000","last_clean":"2024-09-14T04:11:49.036077+0000","last_became_active":"2024-09-14T04:05:55.670180+0000","last_became_peered":"2024-09-14T04:05:55.670180+0000","last_unstale":"2024-09-14T04:11:49.036077+0000","last_undegraded":"2024-09-14T04:11:49.036077+0000","last_fullsized":"2024-09-14T04:11:49.036077+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:33.489272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.037032+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:11:49.037032+0000","last_peered":"2024-09-14T04:11:49.037032+0000","last_clean":"2024-09-14T04:11:49.037032+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:11:49.037032+0000","last_undegraded":"2024-09-14T04:11:49.037032+0000","last_fullsized":"2024-09-14T04:11:49.037032+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":331,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.042297+0000","last_change":"2024-09-14T04:05:55.671536+0000","last_active":"2024-09-14T04:11:49.042297+0000","last_peered":"2024-09-14T04:11:49.042297+0000","last_clean":"2024-09-14T04:11:49.042297+0000","last_became_active":"2024-09-14T04:05:55.671029+0000","last_became_peered":"2024-09-14T04:05:55.671029+0000","last_unstale":"2024-09-14T04:11:49.042297+0000","last_undegraded":"2024-09-14T04:11:49.042297+0000","last_fullsized":"2024-09-14T04:11:49.042297+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:35.745580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":238,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.053608+0000","last_change":"2024-09-14T04:05:55.681242+0000","last_active":"2024-09-14T04:11:49.053608+0000","last_peered":"2024-09-14T04:11:49.053608+0000","last_clean":"2024-09-14T04:11:49.053608+0000","last_became_active":"2024-09-14T04:05:55.680502+0000","last_became_peered":"2024-09-14T04:05:55.680502+0000","last_unstale":"2024-09-14T04:11:49.053608+0000","last_undegraded":"2024-09-14T04:11:49.053608+0000","last_fullsized":"2024-09-14T04:11:49.053608+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:23.760294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":209,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.969717+0000","last_change":"2024-09-14T04:05:55.687662+0000","last_active":"2024-09-14T04:11:38.969717+0000","last_peered":"2024-09-14T04:11:38.969717+0000","last_clean":"2024-09-14T04:11:38.969717+0000","last_became_active":"2024-09-14T04:05:55.673561+0000","last_became_peered":"2024-09-14T04:05:55.673561+0000","last_unstale":"2024-09-14T04:11:38.969717+0000","last_undegraded":"2024-09-14T04:11:38.969717+0000","last_fullsized":"2024-09-14T04:11:38.969717+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:14:51.708356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":238,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.053930+0000","last_change":"2024-09-14T04:05:55.679885+0000","last_active":"2024-09-14T04:11:49.053930+0000","last_peered":"2024-09-14T04:11:49.053930+0000","last_clean":"2024-09-14T04:11:49.053930+0000","last_became_active":"2024-09-14T04:05:55.678589+0000","last_became_peered":"2024-09-14T04:05:55.678589+0000","last_unstale":"2024-09-14T04:11:49.053930+0000","last_undegraded":"2024-09-14T04:11:49.053930+0000","last_fullsized":"2024-09-14T04:11:49.053930+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:51:32.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":192,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.970087+0000","last_change":"2024-09-14T04:05:55.697832+0000","last_active":"2024-09-14T04:11:38.970087+0000","last_peered":"2024-09-14T04:11:38.970087+0000","last_clean":"2024-09-14T04:11:38.970087+0000","last_became_active":"2024-09-14T04:05:55.673923+0000","last_became_peered":"2024-09-14T04:05:55.673923+0000","last_unstale":"2024-09-14T04:11:38.970087+0000","last_undegraded":"2024-09-14T04:11:38.970087+0000","last_fullsized":"2024-09-14T04:11:38.970087+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:44.697302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.049045+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:11:49.049045+0000","last_peered":"2024-09-14T04:11:49.049045+0000","last_clean":"2024-09-14T04:11:49.049045+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:11:49.049045+0000","last_undegraded":"2024-09-14T04:11:49.049045+0000","last_fullsized":"2024-09-14T04:11:49.049045+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":76,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.057180+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:11:49.057180+0000","last_peered":"2024-09-14T04:11:49.057180+0000","last_clean":"2024-09-14T04:11:49.057180+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:11:49.057180+0000","last_undegraded":"2024-09-14T04:11:49.057180+0000","last_fullsized":"2024-09-14T04:11:49.057180+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":144,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-14T04:11:38.971189+0000","last_change":"2024-09-14T04:07:44.338957+0000","last_active":"2024-09-14T04:11:38.971189+0000","last_peered":"2024-09-14T04:11:38.971189+0000","last_clean":"2024-09-14T04:11:38.971189+0000","last_became_active":"2024-09-14T04:07:44.338190+0000","last_became_peered":"2024-09-14T04:07:44.338190+0000","last_unstale":"2024-09-14T04:11:38.971189+0000","last_undegraded":"2024-09-14T04:11:38.971189+0000","last_fullsized":"2024-09-14T04:11:38.971189+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:36:27.488115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":337,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-14T04:11:49.046062+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:11:49.046062+0000","last_peered":"2024-09-14T04:11:49.046062+0000","last_clean":"2024-09-14T04:11:49.046062+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:11:49.046062+0000","last_undegraded":"2024-09-14T04:11:49.046062+0000","last_fullsized":"2024-09-14T04:11:49.046062+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":147062808,"num_objects":2245,"num_object_clones":0,"num_object_copies":6735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238143,"num_write_kb":7692353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544944128,"data_stored":544931912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130043,"ondisk_log_size":130043,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4890,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430578,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1007596,"kb_used_data":100388,"kb_used_omap":1,"kb_used_meta":907198,"kb_avail":92737556,"statfs":{"total":95995035648,"available":94963257344,"internally_reserved":0,"allocated":102797312,"data_stored":101732606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":928971096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626808,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":945896,"kb_used_data":40224,"kb_used_omap":1,"kb_used_meta":905662,"kb_avail":92799256,"statfs":{"total":95995035648,"available":95026438144,"internally_reserved":0,"allocated":41189376,"data_stored":40128382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927398232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823035,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1029136,"kb_used_data":123784,"kb_used_omap":1,"kb_used_meta":905278,"kb_avail":92716016,"statfs":{"total":95995035648,"available":94941200384,"internally_reserved":0,"allocated":126754816,"data_stored":125713307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927005016},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":798863917118,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552416,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":652372,"kb_used_data":149772,"kb_used_omap":4,"kb_used_meta":502395,"kb_avail":93092780,"statfs":{"total":95995035648,"available":95327006720,"internally_reserved":0,"allocated":153366528,"data_stored":152321028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":514452925},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435458,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":522328,"kb_used_data":86480,"kb_used_omap":6,"kb_used_meta":435833,"kb_avail":93222824,"statfs":{"total":95995035648,"available":95460171776,"internally_reserved":0,"allocated":88555520,"data_stored":87509696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":446293453},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827928,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":711892,"kb_used_data":52876,"kb_used_omap":5,"kb_used_meta":659002,"kb_avail":93033260,"statfs":{"total":95995035648,"available":95266058240,"internally_reserved":0,"allocated":54145024,"data_stored":53099451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":674818287},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383916,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776988,"kb_used_data":52436,"kb_used_omap":5,"kb_used_meta":724410,"kb_avail":92968164,"statfs":{"total":95995035648,"available":95199399936,"internally_reserved":0,"allocated":53694464,"data_stored":52641609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":741795900},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,8,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44503040,"data_stored":44498968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":44957696,"data_stored":44957696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":80150528,"data_stored":80150528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":141627392,"data_stored":141623320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":110100480,"data_stored":110100480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28446720,"data_stored":28442648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":95158272,"data_stored":95158272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:12:00.770 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-14T04:12:00.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_5 pg_num 26 2024-09-14T04:12:00.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:00.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:00.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:00.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:00.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:00.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:01.091 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:01.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:01.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:01.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:01.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:01.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:01.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:01.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:01.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:01.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:01.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:01.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:01 smithi114 ceph-mon[103029]: from='client.36198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:12:01.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:01 smithi114 ceph-mon[106969]: from='client.36198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:12:01.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:01 smithi160 ceph-mon[88407]: from='client.36198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:12:01.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:01.482 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:01.486 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:01.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:01.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:01.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:01.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:01.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:12:01.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:01.737 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:01.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:01.748 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:12:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:12:01] "GET /metrics HTTP/1.1" 200 36478 "" "Prometheus/2.51.0" 2024-09-14T04:12:01.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:01.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:01.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:01.911 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:12:01.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:01.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:01.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:02.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:12:02.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:02.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:02.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:02.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:02.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:02.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:02.324 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:02 smithi114 ceph-mon[103029]: pgmap v849: 89 pgs: 89 active+clean; 53 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 797 op/s 2024-09-14T04:12:02.325 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:02 smithi114 ceph-mon[106969]: pgmap v849: 89 pgs: 89 active+clean; 53 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 797 op/s 2024-09-14T04:12:02.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:02.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:02.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:02 smithi160 ceph-mon[88407]: pgmap v849: 89 pgs: 89 active+clean; 53 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 797 op/s 2024-09-14T04:12:02.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:02.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:02.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:02.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:02.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:02.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:02.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:02.710 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:12:02.710 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:12:02.719 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:12:02.719 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:12:02.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:02.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:02.735 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:12:02.735 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:12:02.752 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:12:02.752 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:12:02.767 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:12:02.768 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:12:02.782 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:12:02.782 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:12:02.797 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:12:02.798 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:12:02.813 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:12:02.813 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:12:02.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:02.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:02.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:02.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:02.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:02.973 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:02.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:03.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:03.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:03.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:03.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:03.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:03 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/979628571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:03 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/979628571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:03 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/979628571' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:03.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:03.220 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:03.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:03.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:03.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:03.343 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:03.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:03.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:03.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:03.467 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:03.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:03.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:03.590 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:03.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:03.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:03.713 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:03.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:03.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:03.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:03.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:03.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:03.938 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:03.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:03.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:04.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:04.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:04.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:04.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:04.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:04.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:04.273 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:04 smithi160 ceph-mon[88407]: pgmap v850: 89 pgs: 89 active+clean; 53 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 797 op/s 2024-09-14T04:12:04.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:04.336 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:04.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:04.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:04 smithi114 ceph-mon[103029]: pgmap v850: 89 pgs: 89 active+clean; 53 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 797 op/s 2024-09-14T04:12:04.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:04 smithi114 ceph-mon[106969]: pgmap v850: 89 pgs: 89 active+clean; 53 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 797 op/s 2024-09-14T04:12:04.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:04.462 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:04.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:04.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:04.586 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:04.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:04.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:04.710 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:04.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:04.801 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:12:04.801 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:12:04.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:04.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:04.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:04.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:04.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:04.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:04.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:05.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:05.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:05.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:05.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:05.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:05.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:05.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:05.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:05.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:05.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:05.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-14T04:12:05.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:05.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:05.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:05.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:05.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:05.678 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:05.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:05.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:05.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:05.709 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:12:05.737 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:05.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:05.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:05.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:05.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:05.952 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:05.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:05.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:06.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:06.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:06.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:06.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:06.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:06.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:06.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:06 smithi114 ceph-mon[103029]: pgmap v851: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 530 op/s 2024-09-14T04:12:06.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:06 smithi114 ceph-mon[106969]: pgmap v851: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 530 op/s 2024-09-14T04:12:06.301 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:06 smithi160 ceph-mon[88407]: pgmap v851: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 530 op/s 2024-09-14T04:12:06.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:06.323 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:06.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:06.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:06.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:06.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:06.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:06.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:06.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:06.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:06.695 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:06.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:06.796 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:06.817 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:06.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:06.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:06.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:06.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:07.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:07.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:07.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:07.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:07.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:07.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:07.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:07.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:07.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:07.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:07.439 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:07.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:07.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:07.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:07.562 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:12:07.562 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:12:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:07.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:07.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:07.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:07.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:07.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:12:07.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:07.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:07.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:07.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:07.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:07.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:08.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:08.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:08 smithi160 ceph-mon[88407]: pgmap v852: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 478 op/s 2024-09-14T04:12:08.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:08.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:08.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:08.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:08.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:08.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-14T04:12:08.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:08.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:08 smithi114 ceph-mon[103029]: pgmap v852: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 478 op/s 2024-09-14T04:12:08.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:08.285 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:08 smithi114 ceph-mon[106969]: pgmap v852: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 478 op/s 2024-09-14T04:12:08.285 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:08.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:08.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:08.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:08.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:08.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:08.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:08.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:08.448 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:12:08.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:08.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:08.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:08.567 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:08.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:08.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:08.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:08.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:08.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:08.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:08.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:08.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:08.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:09.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:09.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:09.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:09.095 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 10 pg_num to 26 2024-09-14T04:12:09.096 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:12:09.113 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:09 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2528200823' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:12:09.113 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:12:09.113 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:09 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4285941722' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:12:09.114 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:12:09.114 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:09 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2528200823' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:12:09.114 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:12:09.114 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:09 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4285941722' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:12:09.114 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:12:09.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:09 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2528200823' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:12:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:12:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:09 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4285941722' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:12:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:12:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:09.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:09.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:09.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:09.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:09.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:09.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:09.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:09.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:09.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:09.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:09.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:09.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:09.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:09.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:09.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:09.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:09.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:09.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:09.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:09.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:09.912 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:09.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:10.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:10.037 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:10.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:10.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:10.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:10.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:10.209 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:12:10.209 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:12:10.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:10 smithi160 ceph-mon[88407]: pgmap v853: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 398 op/s 2024-09-14T04:12:10.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:10 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:10.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:10 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-09-14T04:12:10.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:10 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:12:10.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:10 smithi160 ceph-mon[88407]: osdmap e229: 8 total, 8 up, 7 in 2024-09-14T04:12:10.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:10.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:10.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:10.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:10 smithi114 ceph-mon[106969]: pgmap v853: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 398 op/s 2024-09-14T04:12:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:10 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:10 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-09-14T04:12:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:10 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:12:10.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:10 smithi114 ceph-mon[106969]: osdmap e229: 8 total, 8 up, 7 in 2024-09-14T04:12:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:10 smithi114 ceph-mon[103029]: pgmap v853: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 398 op/s 2024-09-14T04:12:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:10 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:10 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-09-14T04:12:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:10 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:12:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:10 smithi114 ceph-mon[103029]: osdmap e229: 8 total, 8 up, 7 in 2024-09-14T04:12:10.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:10.407 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:10.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:10.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:10.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:10.552 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:12:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:12:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:12:10.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:10.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:10.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:10.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:10.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:10.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:10.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:10.800 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-14T04:12:10.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-14T04:12:10.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:10.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:10.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:10.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-14T04:12:11.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:11.027 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:11.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:11.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:11.154 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:11.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:11 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-14T04:12:11.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:11.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:11.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:11.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:11.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:11 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-14T04:12:11.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-14T04:12:11.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:11.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:11.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:11.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:11.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:11.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:11.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:11.730 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:11.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:11.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:11.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:11.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:11.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:12:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:12:11] "GET /metrics HTTP/1.1" 200 36434 "" "Prometheus/2.51.0" 2024-09-14T04:12:11.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:12.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:12.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:12.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:12.165 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:12.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:12.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:12.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:12 smithi114 ceph-mon[103029]: pgmap v855: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 83 op/s 2024-09-14T04:12:12.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-14T04:12:12.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:12 smithi114 ceph-mon[103029]: osdmap e230: 8 total, 8 up, 7 in 2024-09-14T04:12:12.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:12 smithi114 ceph-mon[106969]: pgmap v855: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 83 op/s 2024-09-14T04:12:12.277 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-14T04:12:12.278 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:12 smithi114 ceph-mon[106969]: osdmap e230: 8 total, 8 up, 7 in 2024-09-14T04:12:12.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:12.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:12.372 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:12:12.372 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:12:12.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:12.395 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:12:12.396 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:12:12.396 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:12.402 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:12.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:12 smithi160 ceph-mon[88407]: pgmap v855: 89 pgs: 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 83 op/s 2024-09-14T04:12:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-14T04:12:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:12 smithi160 ceph-mon[88407]: osdmap e230: 8 total, 8 up, 7 in 2024-09-14T04:12:12.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:12.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:12.414 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:12:12.414 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:12:12.431 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:12:12.432 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:12:12.448 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:12:12.448 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:12:12.464 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:12:12.464 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:12:12.479 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:12:12.480 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:12:12.495 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:12:12.495 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:12:12.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:12.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:12.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:12.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:12.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:12.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:12.785 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:12.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:12.886 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:12.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:12.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:13.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:13.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:13.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:13.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:13.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:13.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:13.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:13.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:13.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:13.345 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:13 smithi114 ceph-mon[103029]: osdmap e231: 8 total, 8 up, 7 in 2024-09-14T04:12:13.345 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:13 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1693438809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:13.346 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:13 smithi114 ceph-mon[106969]: osdmap e231: 8 total, 8 up, 7 in 2024-09-14T04:12:13.346 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:13 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1693438809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:13.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:13.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:13.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:13.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:13.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:13 smithi160 ceph-mon[88407]: osdmap e231: 8 total, 8 up, 7 in 2024-09-14T04:12:13.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:13 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1693438809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:13.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:13.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:13.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:13.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:13.651 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:13.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:13.753 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:13.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:13.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:13.877 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:13.900 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:13.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:13.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:12:14.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:14.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:14.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:14.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:14.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:14.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:14.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:14.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:14.275 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:14.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[103029]: 10.4 scrub starts 2024-09-14T04:12:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[103029]: 10.4 scrub ok 2024-09-14T04:12:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[103029]: 10.2 scrub starts 2024-09-14T04:12:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[103029]: 10.2 scrub ok 2024-09-14T04:12:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[103029]: 10.3 scrub starts 2024-09-14T04:12:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[103029]: 10.3 scrub ok 2024-09-14T04:12:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[103029]: pgmap v858: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[103029]: 10.7 scrub starts 2024-09-14T04:12:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[103029]: 10.7 scrub ok 2024-09-14T04:12:14.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[106969]: 10.4 scrub starts 2024-09-14T04:12:14.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[106969]: 10.4 scrub ok 2024-09-14T04:12:14.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[106969]: 10.2 scrub starts 2024-09-14T04:12:14.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[106969]: 10.2 scrub ok 2024-09-14T04:12:14.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[106969]: 10.3 scrub starts 2024-09-14T04:12:14.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[106969]: 10.3 scrub ok 2024-09-14T04:12:14.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[106969]: pgmap v858: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:14.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[106969]: 10.7 scrub starts 2024-09-14T04:12:14.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:14 smithi114 ceph-mon[106969]: 10.7 scrub ok 2024-09-14T04:12:14.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:14.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:14.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:14 smithi160 ceph-mon[88407]: 10.4 scrub starts 2024-09-14T04:12:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:14 smithi160 ceph-mon[88407]: 10.4 scrub ok 2024-09-14T04:12:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:14 smithi160 ceph-mon[88407]: 10.2 scrub starts 2024-09-14T04:12:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:14 smithi160 ceph-mon[88407]: 10.2 scrub ok 2024-09-14T04:12:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:14 smithi160 ceph-mon[88407]: 10.3 scrub starts 2024-09-14T04:12:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:14 smithi160 ceph-mon[88407]: 10.3 scrub ok 2024-09-14T04:12:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:14 smithi160 ceph-mon[88407]: pgmap v858: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:14 smithi160 ceph-mon[88407]: 10.7 scrub starts 2024-09-14T04:12:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:14 smithi160 ceph-mon[88407]: 10.7 scrub ok 2024-09-14T04:12:14.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:14.525 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:14.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:14.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:14.656 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:14.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:14.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:14.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:14.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:14.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:14.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:14.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:15.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:15.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:15.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:15.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:15.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:15.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:15.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[103029]: 10.8 scrub starts 2024-09-14T04:12:15.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[103029]: 10.8 scrub ok 2024-09-14T04:12:15.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[103029]: 10.12 scrub starts 2024-09-14T04:12:15.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[103029]: 10.12 scrub ok 2024-09-14T04:12:15.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[103029]: 10.5 scrub starts 2024-09-14T04:12:15.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[103029]: 10.5 scrub ok 2024-09-14T04:12:15.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[103029]: 10.6 scrub starts 2024-09-14T04:12:15.241 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[103029]: 10.6 scrub ok 2024-09-14T04:12:15.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[106969]: 10.8 scrub starts 2024-09-14T04:12:15.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[106969]: 10.8 scrub ok 2024-09-14T04:12:15.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[106969]: 10.12 scrub starts 2024-09-14T04:12:15.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[106969]: 10.12 scrub ok 2024-09-14T04:12:15.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[106969]: 10.5 scrub starts 2024-09-14T04:12:15.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[106969]: 10.5 scrub ok 2024-09-14T04:12:15.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[106969]: 10.6 scrub starts 2024-09-14T04:12:15.242 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:15 smithi114 ceph-mon[106969]: 10.6 scrub ok 2024-09-14T04:12:15.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:15.275 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:15.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:15.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:15.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:15.400 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:15.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:15 smithi160 ceph-mon[88407]: 10.8 scrub starts 2024-09-14T04:12:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:15 smithi160 ceph-mon[88407]: 10.8 scrub ok 2024-09-14T04:12:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:15 smithi160 ceph-mon[88407]: 10.12 scrub starts 2024-09-14T04:12:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:15 smithi160 ceph-mon[88407]: 10.12 scrub ok 2024-09-14T04:12:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:15 smithi160 ceph-mon[88407]: 10.5 scrub starts 2024-09-14T04:12:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:15 smithi160 ceph-mon[88407]: 10.5 scrub ok 2024-09-14T04:12:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:15 smithi160 ceph-mon[88407]: 10.6 scrub starts 2024-09-14T04:12:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:15 smithi160 ceph-mon[88407]: 10.6 scrub ok 2024-09-14T04:12:15.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:15.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:15.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:15.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:15.649 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:15.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:15.678 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:15.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:15.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:15.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:15.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:12:15.801 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:12:15.801 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:12:15.801 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-14T04:12:15.801 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:12:15.801 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:12:15.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-14T04:12:15.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:15.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:15.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:15.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:16.020 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:16.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:16.053 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:12:16.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:16.143 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:16.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:16.172 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[106969]: 10.14 scrub starts 2024-09-14T04:12:16.172 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[106969]: 10.14 scrub ok 2024-09-14T04:12:16.172 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[106969]: 10.13 scrub starts 2024-09-14T04:12:16.173 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[106969]: 10.13 scrub ok 2024-09-14T04:12:16.173 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[106969]: pgmap v859: 99 pgs: 2 unknown, 97 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:16.173 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[106969]: 10.0 scrub starts 2024-09-14T04:12:16.173 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[106969]: 10.0 scrub ok 2024-09-14T04:12:16.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[103029]: 10.14 scrub starts 2024-09-14T04:12:16.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[103029]: 10.14 scrub ok 2024-09-14T04:12:16.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[103029]: 10.13 scrub starts 2024-09-14T04:12:16.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[103029]: 10.13 scrub ok 2024-09-14T04:12:16.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[103029]: pgmap v859: 99 pgs: 2 unknown, 97 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:16.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[103029]: 10.0 scrub starts 2024-09-14T04:12:16.174 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:16 smithi114 ceph-mon[103029]: 10.0 scrub ok 2024-09-14T04:12:16.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:16.267 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:16.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:16.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:16.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:16.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:16.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:16 smithi160 ceph-mon[88407]: 10.14 scrub starts 2024-09-14T04:12:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:16 smithi160 ceph-mon[88407]: 10.14 scrub ok 2024-09-14T04:12:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:16 smithi160 ceph-mon[88407]: 10.13 scrub starts 2024-09-14T04:12:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:16 smithi160 ceph-mon[88407]: 10.13 scrub ok 2024-09-14T04:12:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:16 smithi160 ceph-mon[88407]: pgmap v859: 99 pgs: 2 unknown, 97 active+clean; 582 KiB data, 4.8 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:16.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:16 smithi160 ceph-mon[88407]: 10.0 scrub starts 2024-09-14T04:12:16.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:16 smithi160 ceph-mon[88407]: 10.0 scrub ok 2024-09-14T04:12:16.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:16.513 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:16.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:16.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:16.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:16.637 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:16.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:16.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:16.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:16.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:16.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:16.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:16.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:16.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:16.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:17.007 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:17.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:17.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:17.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:17.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:17.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:17.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:17.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:17.261 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:17 smithi160 ceph-mon[88407]: 10.18 scrub starts 2024-09-14T04:12:17.261 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:17 smithi160 ceph-mon[88407]: 10.18 scrub ok 2024-09-14T04:12:17.261 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:17 smithi160 ceph-mon[88407]: 10.15 scrub starts 2024-09-14T04:12:17.261 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:17 smithi160 ceph-mon[88407]: 10.15 scrub ok 2024-09-14T04:12:17.261 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:17 smithi160 ceph-mon[88407]: 10.1 scrub starts 2024-09-14T04:12:17.261 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:17 smithi160 ceph-mon[88407]: 10.1 scrub ok 2024-09-14T04:12:17.261 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-14T04:12:17.286 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[106969]: 10.18 scrub starts 2024-09-14T04:12:17.286 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[106969]: 10.18 scrub ok 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[106969]: 10.15 scrub starts 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[106969]: 10.15 scrub ok 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[106969]: 10.1 scrub starts 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[106969]: 10.1 scrub ok 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[103029]: 10.18 scrub starts 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[103029]: 10.18 scrub ok 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[103029]: 10.15 scrub starts 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[103029]: 10.15 scrub ok 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[103029]: 10.1 scrub starts 2024-09-14T04:12:17.287 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[103029]: 10.1 scrub ok 2024-09-14T04:12:17.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-14T04:12:17.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:17.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:17.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:17.383 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:12:17.383 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:12:17.383 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-14T04:12:17.383 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:12:17.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:17.504 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:17.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:17.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:17.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:17.627 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:17.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:17.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:17.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:17.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:17.783 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:12:17.783 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-14T04:12:17.783 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:12:17.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:17.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:17.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:17.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:18.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:18.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:18.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:18.119 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_5' removed 2024-09-14T04:12:18.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:18.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:18.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:18.264 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:18.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:18.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:18.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[103029]: 10.9 scrub starts 2024-09-14T04:12:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[103029]: 10.9 scrub ok 2024-09-14T04:12:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[103029]: pgmap v860: 99 pgs: 99 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-14T04:12:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[103029]: osdmap e232: 8 total, 8 up, 7 in 2024-09-14T04:12:18.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/28806426' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:12:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[106969]: 10.9 scrub starts 2024-09-14T04:12:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[106969]: 10.9 scrub ok 2024-09-14T04:12:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[106969]: pgmap v860: 99 pgs: 99 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-14T04:12:18.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[106969]: osdmap e232: 8 total, 8 up, 7 in 2024-09-14T04:12:18.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:18 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/28806426' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:12:18.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:18.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:18.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:18.395 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:18 smithi160 ceph-mon[88407]: 10.9 scrub starts 2024-09-14T04:12:18.395 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:18 smithi160 ceph-mon[88407]: 10.9 scrub ok 2024-09-14T04:12:18.395 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:18 smithi160 ceph-mon[88407]: pgmap v860: 99 pgs: 99 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:18.395 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-14T04:12:18.395 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:18 smithi160 ceph-mon[88407]: osdmap e232: 8 total, 8 up, 7 in 2024-09-14T04:12:18.395 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:18 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/28806426' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:12:18.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:18.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:18.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:18.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:18.638 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:18.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:18.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:18.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:18.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:18.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:18.886 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:18.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:18.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:19.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:19.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:19.095 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-14T04:12:19.096 INFO:tasks.radosbench:Beginning radosbench... 2024-09-14T04:12:19.096 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-09-14T04:12:19.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-09-14T04:12:19.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:19.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:19.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:19.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:19.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:19.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:19.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:19.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/28806426' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:12:19.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:19 smithi114 ceph-mon[106969]: osdmap e233: 8 total, 8 up, 7 in 2024-09-14T04:12:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:19 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/28806426' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:12:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:19 smithi114 ceph-mon[103029]: osdmap e233: 8 total, 8 up, 7 in 2024-09-14T04:12:19.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:19.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:19 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/28806426' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:12:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:19 smithi160 ceph-mon[88407]: osdmap e233: 8 total, 8 up, 7 in 2024-09-14T04:12:19.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:19.507 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:19.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:19.609 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:19.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:19.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:19.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:19.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:19.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-14T04:12:19.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:19.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:19.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:19.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:19.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:19.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:19.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:20.000 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:20.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:20.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:20.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:20.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:20.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:20.229 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:20.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:20.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:20.268 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:20.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:20.354 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:20 smithi114 ceph-mon[103029]: pgmap v863: 73 pgs: 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:20.354 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:20 smithi114 ceph-mon[106969]: pgmap v863: 73 pgs: 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:20.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:20.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:20 smithi160 ceph-mon[88407]: pgmap v863: 73 pgs: 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:20.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:12:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:12:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:12:20.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:20.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:20.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:20.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:20.619 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:20.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:20.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:20.721 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:20.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:20.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:20.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:20.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:20.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:20.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:20.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:21.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:21.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:21.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:21.142 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:12:21.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:21.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:21.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:21.340 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:21.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:21.365 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:21.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2857640532' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:12:21.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2857640532' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:12:21.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2857640532' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:12:21.467 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:21.489 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:21.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:21.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:21.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:21.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:21.737 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:21.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:21.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:21.852 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:12:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:12:21] "GET /metrics HTTP/1.1" 200 36434 "" "Prometheus/2.51.0" 2024-09-14T04:12:21.860 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:21.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:21.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:21.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:21.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:22.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:22.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:22.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:22.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:22.230 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:22.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:22.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:22.356 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:22.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:22 smithi160 ceph-mon[88407]: pgmap v864: 73 pgs: 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:22 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:12:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:22 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:12:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:22 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2857640532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:12:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:22 smithi160 ceph-mon[88407]: osdmap e234: 8 total, 8 up, 7 in 2024-09-14T04:12:22.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:22.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:22.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:22.585 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:22.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:22.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:22 smithi114 ceph-mon[103029]: pgmap v864: 73 pgs: 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:22 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:12:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:22 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:12:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:22 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2857640532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:12:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:22 smithi114 ceph-mon[103029]: osdmap e234: 8 total, 8 up, 7 in 2024-09-14T04:12:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:22 smithi114 ceph-mon[106969]: pgmap v864: 73 pgs: 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:22 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:12:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:22 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:12:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:22 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2857640532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:12:22.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:22 smithi114 ceph-mon[106969]: osdmap e234: 8 total, 8 up, 7 in 2024-09-14T04:12:22.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:22.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:22.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:22.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:22.858 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:22.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:22.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:22.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:22.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:22.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:22.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:23.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:23.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:23.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:23.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:23.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:23.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:23.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:23.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:23.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:23.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:23.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:23.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:23.491 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:23.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:23.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:23.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:23.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:23.710 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:12:23.711 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:12:23.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:23.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:23.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:23.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:23.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:23.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:23.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:12:23.902 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:23.902 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:23.902 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:23 smithi114 ceph-mon[103029]: pgmap v866: 73 pgs: 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:23.902 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:23.903 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:23.903 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:23.903 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:23 smithi114 ceph-mon[106969]: pgmap v866: 73 pgs: 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:23.903 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:23 smithi160 ceph-mon[88407]: pgmap v866: 73 pgs: 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:23.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:23.992 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:23.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:23.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:24.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:24.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:24.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:24.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:24.239 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:24.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:24.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:24.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:24.369 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:24.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:24.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:24.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:24.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:24.615 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:24.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:24.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:24.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:24.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:24.820 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_6' created 2024-09-14T04:12:24.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:24.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:24 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/395494995' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-14T04:12:24.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:24 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/395494995' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-14T04:12:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:24.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:24.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:24.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:24.994 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:24.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:25.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:25.121 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:25.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:24 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/395494995' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-14T04:12:25.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:25.243 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:25.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:25.320 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:12:25.320 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:12:25.320 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:12:25.320 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:12:25.320 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:12:25.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:25.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:25.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:25.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:25.474 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:25.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:25.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:25.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:25.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:25.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:25.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:25.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:25.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:25.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:25.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:25.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:25.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:25.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:25.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:25.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:26.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:26.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/395494995' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-14T04:12:26.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:25 smithi114 ceph-mon[103029]: osdmap e235: 8 total, 8 up, 7 in 2024-09-14T04:12:26.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:25 smithi114 ceph-mon[103029]: pgmap v868: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:26.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:25 smithi114 ceph-mon[103029]: osdmap e236: 8 total, 8 up, 7 in 2024-09-14T04:12:26.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/395494995' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-14T04:12:26.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:25 smithi114 ceph-mon[106969]: osdmap e235: 8 total, 8 up, 7 in 2024-09-14T04:12:26.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:25 smithi114 ceph-mon[106969]: pgmap v868: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:26.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:25 smithi114 ceph-mon[106969]: osdmap e236: 8 total, 8 up, 7 in 2024-09-14T04:12:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:26.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:26.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:26.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-14T04:12:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/395494995' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-14T04:12:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:25 smithi160 ceph-mon[88407]: osdmap e235: 8 total, 8 up, 7 in 2024-09-14T04:12:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:25 smithi160 ceph-mon[88407]: pgmap v868: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:25 smithi160 ceph-mon[88407]: osdmap e236: 8 total, 8 up, 7 in 2024-09-14T04:12:26.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:26.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:26.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:26.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:26.373 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:26.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:26.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:26.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:26.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:26.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-09-14T04:12:26.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:26.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:26.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:26.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:26.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:26.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:26.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:26.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:26.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:26.986 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:27.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:27.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:27.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:27.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:26 smithi114 ceph-mon[103029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:12:27.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:26 smithi114 ceph-mon[103029]: osdmap e237: 8 total, 8 up, 7 in 2024-09-14T04:12:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:26 smithi114 ceph-mon[106969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:12:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:26 smithi114 ceph-mon[106969]: osdmap e237: 8 total, 8 up, 7 in 2024-09-14T04:12:27.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:27.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:26 smithi160 ceph-mon[88407]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:12:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:26 smithi160 ceph-mon[88407]: osdmap e237: 8 total, 8 up, 7 in 2024-09-14T04:12:27.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:27.265 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:27.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:27.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:27.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:27.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:27.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:27.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:27.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:27.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:27.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:27.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:27.763 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:27.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:27.793 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:12:27.793 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:12:27.793 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:12:27.793 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:12:27.799 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:27.810 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:27.815 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:12:27.815 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:12:27.843 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:12:27.843 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:12:27.859 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:12:27.860 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:12:27.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:27.875 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:12:27.875 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:12:27.891 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:12:27.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:27.891 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:12:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:27.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:27.906 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:12:27.907 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:12:27.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:28.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:28.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:28.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:28.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:27 smithi114 ceph-mon[103029]: pgmap v871: 89 pgs: 2 creating+activating, 14 unknown, 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:28.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:27 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2759686559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:28.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:27 smithi114 ceph-mon[106969]: pgmap v871: 89 pgs: 2 creating+activating, 14 unknown, 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:28.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:27 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2759686559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:28.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:28.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:28.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:27 smithi160 ceph-mon[88407]: pgmap v871: 89 pgs: 2 creating+activating, 14 unknown, 73 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:27 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2759686559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:28.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:28.267 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:28.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:28.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:28.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:28.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:28.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:28.514 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:28.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:28.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:28.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:28.642 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:28.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:28.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:28.769 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:28.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:28.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:28.894 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:28.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:28.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:29.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:29.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:29.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:29.145 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:29.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:29.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:29.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:29.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:29.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:29.392 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:29.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:29.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:29.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:29.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:29.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:29.639 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:29.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:29.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:29.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:29.762 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:29.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:29.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:29.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:29.886 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:29.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:29.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:30.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:30.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:30.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:29 smithi114 ceph-mon[103029]: pgmap v872: 89 pgs: 2 creating+activating, 10 unknown, 77 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:30.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:12:30.030 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:29 smithi114 ceph-mon[106969]: pgmap v872: 89 pgs: 2 creating+activating, 10 unknown, 77 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:30.030 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:12:30.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:30.133 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:30.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:30.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:29 smithi160 ceph-mon[88407]: pgmap v872: 89 pgs: 2 creating+activating, 10 unknown, 77 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:30.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:12:30.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:30.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:30.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:30.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:30.380 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:30.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:30.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:30.503 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:30.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:30.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:30.550 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:12:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:12:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:12:30.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:30.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:30.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:30.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:30.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:30.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:30.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:30.863 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:12:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:30.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:30.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:30.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:31.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:31.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:31.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:31.081 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:30 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1721751755' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:12:31.081 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:30 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:12:31.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:30 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1721751755' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:12:31.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:30 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:12:31.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:30 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1721751755' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:12:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:30 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:12:31.165 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:31.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:31.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:31.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:31.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:31.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:31.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:31.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:31.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:31.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:31.536 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:31.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:31.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:31.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:31.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:31.716 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:12:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:12:31] "GET /metrics HTTP/1.1" 200 34071 "" "Prometheus/2.51.0" 2024-09-14T04:12:31.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:31.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:31.882 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:31.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:31.907 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:31.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:31.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:31 smithi114 ceph-mon[106969]: pgmap v873: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:31.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:31 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:31.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:31 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:12:31.988 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:31 smithi114 ceph-mon[106969]: osdmap e238: 8 total, 8 up, 7 in 2024-09-14T04:12:31.988 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:31 smithi114 ceph-mon[103029]: pgmap v873: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:31.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:31 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:31.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:31 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:12:31.989 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:31 smithi114 ceph-mon[103029]: osdmap e238: 8 total, 8 up, 7 in 2024-09-14T04:12:32.008 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:32.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:32.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:32.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:32.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:32.140 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:31 smithi160 ceph-mon[88407]: pgmap v873: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:32.140 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:31 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:32.140 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:31 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:12:32.141 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:31 smithi160 ceph-mon[88407]: osdmap e238: 8 total, 8 up, 7 in 2024-09-14T04:12:32.154 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:32.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:32.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:32.278 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:32.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:32.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:32.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:32.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:32.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:32.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:32.526 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:32.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:32.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:32.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:32.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:32.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:32.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:32.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:32.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:32.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:32.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:33.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:33.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:33.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:33.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:12:33.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:33.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:33.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:33.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:33.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:33.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:33.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:33.394 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:33.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:33.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:33.520 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:33.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:33.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:33.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:33.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:33.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:33.765 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:33.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:33.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:33.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:12:33.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:33.920 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:33.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:33.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:34.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:34.041 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:33 smithi114 ceph-mon[106969]: pgmap v875: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:34.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:33 smithi114 ceph-mon[103029]: pgmap v875: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:34.044 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:34.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:34.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:33 smithi160 ceph-mon[88407]: pgmap v875: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:34.168 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:34.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:34.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:34.291 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:34.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:34.334 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:12:34.334 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:12:34.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:34.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:34.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:34.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:34.539 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:34.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:34.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:34.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:34.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:34.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:34.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:34.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:34.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:34.908 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:34.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:35.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:35.040 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:35.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:35.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-14T04:12:35.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:35.168 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:35.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:35.270 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:35.294 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:35.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:35.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:35 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:35.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:35 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:35 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:35 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:35.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:35.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:35.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:35.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:35.541 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:35.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:35.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:35.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:35 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:35 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:35.668 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:35.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:35.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:35.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:35.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:35.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:35.894 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:35.919 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:35.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:36.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:36.043 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:36.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:36.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:36.171 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:36.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:36.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:36.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:36 smithi114 ceph-mon[106969]: pgmap v876: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:36.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:36 smithi114 ceph-mon[103029]: pgmap v876: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:36.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:36.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:36.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:36.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:36.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:36.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:36.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:36.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:36.655 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:36.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:36 smithi160 ceph-mon[88407]: pgmap v876: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:36.681 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:36.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:36.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:36.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:36.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:36.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:36.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:36.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:37.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:37.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:37.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:37.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:37.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:37.191 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:37.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:37.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:37.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:37.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:37.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:37.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:37.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:37.546 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:37.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:37.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:37.620 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-09-14T04:12:37.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:37 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/446724147' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:12:37.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:37 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/446724147' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:12:37.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:37 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/446724147' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:12:37.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:37.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:37.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:37.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:37.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:37.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:37.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:37.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:37.943 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:37.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:38.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:38.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:38.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:38.178 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:38.179 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:38.196 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:38.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:38.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:38.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:38.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:38.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:38.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:38.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:38.486 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[106969]: pgmap v877: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:38.486 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/446724147' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:12:38.486 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[106969]: osdmap e239: 8 total, 8 up, 7 in 2024-09-14T04:12:38.486 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:38.486 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:38.487 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:38.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[103029]: pgmap v877: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:38.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/446724147' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:12:38.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[103029]: osdmap e239: 8 total, 8 up, 7 in 2024-09-14T04:12:38.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:38.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:38.487 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:38.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:38.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:38.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:38 smithi160 ceph-mon[88407]: pgmap v877: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/446724147' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:12:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:38 smithi160 ceph-mon[88407]: osdmap e239: 8 total, 8 up, 7 in 2024-09-14T04:12:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:38.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:38.693 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:38.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:38.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:38.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:38.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:38.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:38.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:38.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:38.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:38.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:39.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:39.071 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:39.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:39.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:39.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:39.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:39.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:39.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:39.321 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:39.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:39.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:39.444 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:39.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:39.546 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:39.573 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:39.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:39.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:39 smithi114 ceph-mon[103029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:12:39.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:39 smithi114 ceph-mon[106969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:12:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:39 smithi160 ceph-mon[88407]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:12:39.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:39.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:39.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:39.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:39.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:39.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:39.925 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:39.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:39.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:39.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:40.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:40.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:40.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:40.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:40.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:40.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:40.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:40.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:40.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:40.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:40.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:40.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:40.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:40.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:40 smithi114 ceph-mon[106969]: pgmap v879: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:40.515 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:40 smithi114 ceph-mon[103029]: pgmap v879: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:40.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:40.578 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:40.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:40 smithi160 ceph-mon[88407]: pgmap v879: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:12:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:12:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:12:40.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:40.701 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:40.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:40.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:40.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:40.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:40.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:40.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:40.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:41.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:41.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:41.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:41.186 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:41.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:41.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:41.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:41.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:41.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:41.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:41.462 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:41.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:41.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:41.549 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:12:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:12:41] "GET /metrics HTTP/1.1" 200 36489 "" "Prometheus/2.51.0" 2024-09-14T04:12:41.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:41.570 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:12:41.570 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:12:41.586 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:41.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:41.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:41.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:41.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:41.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:41.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:41.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:41.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:41.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:41.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:41.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:42.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:42.090 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:42.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:42.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:42.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:42.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:42.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:42.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:42.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:42.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:42.467 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:42.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:42.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:42.590 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:42.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:42 smithi160 ceph-mon[88407]: pgmap v880: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:42.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:42.718 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:42.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:42.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:42.842 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:42.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:42.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:42 smithi114 ceph-mon[103029]: pgmap v880: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:42.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:42 smithi114 ceph-mon[106969]: pgmap v880: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:42.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:42.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:42.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:43.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-14T04:12:43.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:43.096 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:43.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:43.129 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-09-14T04:12:43.130 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_6 2024-09-14T04:12:43.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:12:43.133 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-14T04:12:43.133 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-14T04:12:43.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:43.220 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-14T04:12:43.220 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-14T04:12:43.220 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_223345 2024-09-14T04:12:43.221 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:12:43.221 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-14T04:12:43.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:43.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:43.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:43.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:43.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:43.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:43.526 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:43.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:43.558 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:12:43.558 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:12:43.602 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:12:43.602 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:12:43.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:43.643 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:12:43.643 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:12:43.658 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:43.684 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:43.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:43.688 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:12:43.688 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:12:43.723 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:12:43.723 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:12:43.757 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:12:43.757 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:12:43.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:43.789 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:12:43.789 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:12:43.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:43.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:43.823 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:12:43.824 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:12:43.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:43.946 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:43.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:44.048 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:44.077 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:44.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:44.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:44.210 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:44.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:44.221 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 2970 2954 184.605 184.625 0.0042668 0.0053999 2024-09-14T04:12:44.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:44.349 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:44.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:44.407 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:12:44.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:44.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:44.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:44.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:44.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:44 smithi114 ceph-mon[106969]: pgmap v881: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:44.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:44 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/536748688' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:44 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2444541027' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:12:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:44 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:12:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:44 smithi114 ceph-mon[103029]: pgmap v881: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:44 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/536748688' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:44 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2444541027' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:12:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:44 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:12:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:44.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:44.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:44.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:44.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:44.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:44.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:44 smithi160 ceph-mon[88407]: pgmap v881: 89 pgs: 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail 2024-09-14T04:12:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:44 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/536748688' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:12:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:44 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2444541027' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:12:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:44 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:12:44.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:44.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:45.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:45.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:45.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:45.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:45.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:45.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:45.222 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 6323 6307 197.074 209.562 0.00327293 0.00506458 2024-09-14T04:12:45.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:45.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:45.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:45.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:45.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:45.502 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:45.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:45.590 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:45.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:45.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:45.606 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:45 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:45.606 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:45 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:12:45.606 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:45 smithi114 ceph-mon[103029]: osdmap e240: 8 total, 8 up, 7 in 2024-09-14T04:12:45.606 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:45 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:45.606 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:45 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:12:45.606 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:45 smithi114 ceph-mon[106969]: osdmap e240: 8 total, 8 up, 7 in 2024-09-14T04:12:45.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:45.730 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:45.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:45.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:45.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:45.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:45 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:12:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:45 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:12:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:45 smithi160 ceph-mon[88407]: osdmap e240: 8 total, 8 up, 7 in 2024-09-14T04:12:45.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:46.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:46.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:46.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:46.143 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:46.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:46.221 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 9600 9584 199.647 204.812 0.00429237 0.00500066 2024-09-14T04:12:46.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:46.281 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:46.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:46.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:46.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:46.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:46.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:46.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:46.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:46.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:46.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:46 smithi114 ceph-mon[106969]: pgmap v883: 89 pgs: 89 active+clean; 143 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 18 MiB/s wr, 285 op/s 2024-09-14T04:12:46.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:46 smithi114 ceph-mon[103029]: pgmap v883: 89 pgs: 89 active+clean; 143 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 18 MiB/s wr, 285 op/s 2024-09-14T04:12:46.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:46.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:46.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:46.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:46.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:46.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:12:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:46 smithi160 ceph-mon[88407]: pgmap v883: 89 pgs: 89 active+clean; 143 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 18 MiB/s wr, 285 op/s 2024-09-14T04:12:46.925 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:46.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:46.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:47.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:47.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:47.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:47.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:47.222 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 12510 12494 195.157 181.875 0.00364419 0.00511685 2024-09-14T04:12:47.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:47.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:47.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:47.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:47.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:47.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:47.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:47.505 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:47.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:47.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:47.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:47.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:47.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:47.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:47.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:47.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:47.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:47.917 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:47.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:48.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:48.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:48.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:48.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:48.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:12:48.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:48.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:48.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:48.223 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 15154 15138 189.172 165.25 0.00788737 0.00527928 2024-09-14T04:12:48.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:48.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:48.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:48.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:48.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:48.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:48.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:48.577 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:48.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:48.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:48.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:48.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:48.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:48.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:48 smithi114 ceph-mon[103029]: pgmap v884: 89 pgs: 89 active+clean; 206 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 21 MiB/s wr, 342 op/s 2024-09-14T04:12:48.735 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:48 smithi114 ceph-mon[106969]: pgmap v884: 89 pgs: 89 active+clean; 206 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 21 MiB/s wr, 342 op/s 2024-09-14T04:12:48.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:48.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:48.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:48.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:48 smithi160 ceph-mon[88407]: pgmap v884: 89 pgs: 89 active+clean; 206 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 21 MiB/s wr, 342 op/s 2024-09-14T04:12:48.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:48.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:48.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:49.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:49.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:49.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:49.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:49.222 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 17445 17429 181.507 143.188 0.00816767 0.00550113 2024-09-14T04:12:49.237 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:49.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:49.271 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:12:49.272 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:12:49.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:49.369 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:49.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:49.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:49.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:49.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:49.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:49.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:49.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:49.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:49.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:49.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:49.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:49.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:49.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:49.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:50.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:50.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:50.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:50.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:50.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:50.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:50.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:50.222 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 20092 20076 179.209 165.438 0.00277897 0.00557397 2024-09-14T04:12:50.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:50.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:50.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:50.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:50.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:50.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:50.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:50.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:50.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:50.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:50.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:50.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:50 smithi114 ceph-mon[106969]: pgmap v885: 89 pgs: 89 active+clean; 435 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s 2024-09-14T04:12:50.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:50 smithi114 ceph-mon[106969]: from='client.46241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:12:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:50 smithi114 ceph-mon[103029]: pgmap v885: 89 pgs: 89 active+clean; 435 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s 2024-09-14T04:12:50.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:50 smithi114 ceph-mon[103029]: from='client.46241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:12:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:50 smithi160 ceph-mon[88407]: pgmap v885: 89 pgs: 89 active+clean; 435 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s 2024-09-14T04:12:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:50 smithi160 ceph-mon[88407]: from='client.46241 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:12:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:12:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:12:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:12:50.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:50.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:50.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:50.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:50.870 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":885,"stamp":"2024-09-14T04:12:48.852880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":456333261,"num_objects":6980,"num_object_clones":0,"num_object_copies":20940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6980,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":7166,"num_write_kb":446715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":595258,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7112,"ondisk_log_size":7112,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6452084,"kb_used_data":1339960,"kb_used_omap":27,"kb_used_meta":5112036,"kb_avail":649763980,"statfs":{"total":671965249536,"available":665358315520,"internally_reserved":0,"allocated":1372119040,"data_stored":1364409083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28483,"internal_metadata":5234725053},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":455737344,"num_objects":6954,"num_object_clones":0,"num_object_copies":20862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6954,"num_write_kb":445056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006600"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":289,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.409023+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:12:44.409023+0000","last_peered":"2024-09-14T04:12:44.409023+0000","last_clean":"2024-09-14T04:12:44.409023+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:12:44.409023+0000","last_undegraded":"2024-09-14T04:12:44.409023+0000","last_fullsized":"2024-09-14T04:12:44.409023+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":222,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.762514+0000","last_change":"2024-09-14T04:05:55.701296+0000","last_active":"2024-09-14T04:12:31.762514+0000","last_peered":"2024-09-14T04:12:31.762514+0000","last_clean":"2024-09-14T04:12:31.762514+0000","last_became_active":"2024-09-14T04:05:55.676634+0000","last_became_peered":"2024-09-14T04:05:55.676634+0000","last_unstale":"2024-09-14T04:12:31.762514+0000","last_undegraded":"2024-09-14T04:12:31.762514+0000","last_fullsized":"2024-09-14T04:12:31.762514+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:44.187804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":205,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.272163+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:12:37.272163+0000","last_peered":"2024-09-14T04:12:37.272163+0000","last_clean":"2024-09-14T04:12:37.272163+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:12:37.272163+0000","last_undegraded":"2024-09-14T04:12:37.272163+0000","last_fullsized":"2024-09-14T04:12:37.272163+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":373,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.405087+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:12:44.405087+0000","last_peered":"2024-09-14T04:12:44.405087+0000","last_clean":"2024-09-14T04:12:44.405087+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:12:44.405087+0000","last_undegraded":"2024-09-14T04:12:44.405087+0000","last_fullsized":"2024-09-14T04:12:44.405087+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":359,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.405973+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:12:44.405973+0000","last_peered":"2024-09-14T04:12:44.405973+0000","last_clean":"2024-09-14T04:12:44.405973+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:12:44.405973+0000","last_undegraded":"2024-09-14T04:12:44.405973+0000","last_fullsized":"2024-09-14T04:12:44.405973+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":197,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.762367+0000","last_change":"2024-09-14T04:05:55.676850+0000","last_active":"2024-09-14T04:12:31.762367+0000","last_peered":"2024-09-14T04:12:31.762367+0000","last_clean":"2024-09-14T04:12:31.762367+0000","last_became_active":"2024-09-14T04:05:55.676337+0000","last_became_peered":"2024-09-14T04:05:55.676337+0000","last_unstale":"2024-09-14T04:12:31.762367+0000","last_undegraded":"2024-09-14T04:12:31.762367+0000","last_fullsized":"2024-09-14T04:12:31.762367+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:54:12.377665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":102,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.409260+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:12:44.409260+0000","last_peered":"2024-09-14T04:12:44.409260+0000","last_clean":"2024-09-14T04:12:44.409260+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:12:44.409260+0000","last_undegraded":"2024-09-14T04:12:44.409260+0000","last_fullsized":"2024-09-14T04:12:44.409260+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":213,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.761173+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:12:31.761173+0000","last_peered":"2024-09-14T04:12:31.761173+0000","last_clean":"2024-09-14T04:12:31.761173+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:12:31.761173+0000","last_undegraded":"2024-09-14T04:12:31.761173+0000","last_fullsized":"2024-09-14T04:12:31.761173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"240'230","reported_seq":243,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.415144+0000","last_change":"2024-09-14T04:12:25.832585+0000","last_active":"2024-09-14T04:12:44.415144+0000","last_peered":"2024-09-14T04:12:44.415144+0000","last_clean":"2024-09-14T04:12:44.415144+0000","last_became_active":"2024-09-14T04:12:25.832280+0000","last_became_peered":"2024-09-14T04:12:25.832280+0000","last_unstale":"2024-09-14T04:12:44.415144+0000","last_undegraded":"2024-09-14T04:12:44.415144+0000","last_fullsized":"2024-09-14T04:12:44.415144+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:07:45.490307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15073280,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":230,"num_write_kb":14720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":219,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.763135+0000","last_change":"2024-09-14T04:05:55.704897+0000","last_active":"2024-09-14T04:12:31.763135+0000","last_peered":"2024-09-14T04:12:31.763135+0000","last_clean":"2024-09-14T04:12:31.763135+0000","last_became_active":"2024-09-14T04:05:55.681093+0000","last_became_peered":"2024-09-14T04:05:55.681093+0000","last_unstale":"2024-09-14T04:12:31.763135+0000","last_undegraded":"2024-09-14T04:12:31.763135+0000","last_fullsized":"2024-09-14T04:12:31.763135+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:05:49.647801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":188,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.271757+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:12:37.271757+0000","last_peered":"2024-09-14T04:12:37.271757+0000","last_clean":"2024-09-14T04:12:37.271757+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:12:37.271757+0000","last_undegraded":"2024-09-14T04:12:37.271757+0000","last_fullsized":"2024-09-14T04:12:37.271757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"240'1074","reported_seq":1095,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:48.691879+0000","last_change":"2024-09-14T04:12:25.836189+0000","last_active":"2024-09-14T04:12:48.691879+0000","last_peered":"2024-09-14T04:12:48.691879+0000","last_clean":"2024-09-14T04:12:48.691879+0000","last_became_active":"2024-09-14T04:12:25.835681+0000","last_became_peered":"2024-09-14T04:12:25.835681+0000","last_unstale":"2024-09-14T04:12:48.691879+0000","last_undegraded":"2024-09-14T04:12:48.691879+0000","last_fullsized":"2024-09-14T04:12:48.691879+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:28:43.799936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70385664,"num_objects":1074,"num_object_clones":0,"num_object_copies":3222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1074,"num_write_kb":68736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":220,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.762923+0000","last_change":"2024-09-14T04:05:55.702981+0000","last_active":"2024-09-14T04:12:31.762923+0000","last_peered":"2024-09-14T04:12:31.762923+0000","last_clean":"2024-09-14T04:12:31.762923+0000","last_became_active":"2024-09-14T04:05:55.679789+0000","last_became_peered":"2024-09-14T04:05:55.679789+0000","last_unstale":"2024-09-14T04:12:31.762923+0000","last_undegraded":"2024-09-14T04:12:31.762923+0000","last_fullsized":"2024-09-14T04:12:31.762923+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:34:24.358644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":279,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.411025+0000","last_change":"2024-09-14T04:07:44.340689+0000","last_active":"2024-09-14T04:12:44.411025+0000","last_peered":"2024-09-14T04:12:44.411025+0000","last_clean":"2024-09-14T04:12:44.411025+0000","last_became_active":"2024-09-14T04:07:44.340112+0000","last_became_peered":"2024-09-14T04:07:44.340112+0000","last_unstale":"2024-09-14T04:12:44.411025+0000","last_undegraded":"2024-09-14T04:12:44.411025+0000","last_fullsized":"2024-09-14T04:12:44.411025+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:04.263233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.5","version":"239'181","reported_seq":193,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.213118+0000","last_change":"2024-09-14T04:12:25.834686+0000","last_active":"2024-09-14T04:12:44.213118+0000","last_peered":"2024-09-14T04:12:44.213118+0000","last_clean":"2024-09-14T04:12:44.213118+0000","last_became_active":"2024-09-14T04:12:25.834355+0000","last_became_peered":"2024-09-14T04:12:25.834355+0000","last_unstale":"2024-09-14T04:12:44.213118+0000","last_undegraded":"2024-09-14T04:12:44.213118+0000","last_fullsized":"2024-09-14T04:12:44.213118+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:23:29.553246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11862016,"num_objects":181,"num_object_clones":0,"num_object_copies":543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":181,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":188,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.762014+0000","last_change":"2024-09-14T03:43:11.270388+0000","last_active":"2024-09-14T04:12:31.762014+0000","last_peered":"2024-09-14T04:12:31.762014+0000","last_clean":"2024-09-14T04:12:31.762014+0000","last_became_active":"2024-09-14T03:43:11.270122+0000","last_became_peered":"2024-09-14T03:43:11.270122+0000","last_unstale":"2024-09-14T04:12:31.762014+0000","last_undegraded":"2024-09-14T04:12:31.762014+0000","last_fullsized":"2024-09-14T04:12:31.762014+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:30:31.830488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":214,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.761512+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:12:31.761512+0000","last_peered":"2024-09-14T04:12:31.761512+0000","last_clean":"2024-09-14T04:12:31.761512+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:12:31.761512+0000","last_undegraded":"2024-09-14T04:12:31.761512+0000","last_fullsized":"2024-09-14T04:12:31.761512+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"239'188","reported_seq":200,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.208643+0000","last_change":"2024-09-14T04:12:25.838840+0000","last_active":"2024-09-14T04:12:44.208643+0000","last_peered":"2024-09-14T04:12:44.208643+0000","last_clean":"2024-09-14T04:12:44.208643+0000","last_became_active":"2024-09-14T04:12:25.838346+0000","last_became_peered":"2024-09-14T04:12:25.838346+0000","last_unstale":"2024-09-14T04:12:44.208643+0000","last_undegraded":"2024-09-14T04:12:44.208643+0000","last_fullsized":"2024-09-14T04:12:44.208643+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:02:35.537032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12320768,"num_objects":188,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":188,"num_write_kb":12032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":226,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.272186+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:12:37.272186+0000","last_peered":"2024-09-14T04:12:37.272186+0000","last_clean":"2024-09-14T04:12:37.272186+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:12:37.272186+0000","last_undegraded":"2024-09-14T04:12:37.272186+0000","last_fullsized":"2024-09-14T04:12:37.272186+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":388,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.404274+0000","last_change":"2024-09-14T04:07:54.363033+0000","last_active":"2024-09-14T04:12:44.404274+0000","last_peered":"2024-09-14T04:12:44.404274+0000","last_clean":"2024-09-14T04:12:44.404274+0000","last_became_active":"2024-09-14T04:07:54.362706+0000","last_became_peered":"2024-09-14T04:07:54.362706+0000","last_unstale":"2024-09-14T04:12:44.404274+0000","last_undegraded":"2024-09-14T04:12:44.404274+0000","last_fullsized":"2024-09-14T04:12:44.404274+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:27:31.193597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"239'93","reported_seq":104,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:43.856726+0000","last_change":"2024-09-14T04:12:25.833063+0000","last_active":"2024-09-14T04:12:43.856726+0000","last_peered":"2024-09-14T04:12:43.856726+0000","last_clean":"2024-09-14T04:12:43.856726+0000","last_became_active":"2024-09-14T04:12:25.832255+0000","last_became_peered":"2024-09-14T04:12:25.832255+0000","last_unstale":"2024-09-14T04:12:43.856726+0000","last_undegraded":"2024-09-14T04:12:43.856726+0000","last_fullsized":"2024-09-14T04:12:43.856726+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:25:44.212992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6094848,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":5952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":328,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.408792+0000","last_change":"2024-09-14T04:05:55.695825+0000","last_active":"2024-09-14T04:12:44.408792+0000","last_peered":"2024-09-14T04:12:44.408792+0000","last_clean":"2024-09-14T04:12:44.408792+0000","last_became_active":"2024-09-14T04:05:55.680668+0000","last_became_peered":"2024-09-14T04:05:55.680668+0000","last_unstale":"2024-09-14T04:12:44.408792+0000","last_undegraded":"2024-09-14T04:12:44.408792+0000","last_fullsized":"2024-09-14T04:12:44.408792+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:30.262947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":248,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.409104+0000","last_change":"2024-09-14T04:05:55.677723+0000","last_active":"2024-09-14T04:12:44.409104+0000","last_peered":"2024-09-14T04:12:44.409104+0000","last_clean":"2024-09-14T04:12:44.409104+0000","last_became_active":"2024-09-14T04:05:55.677254+0000","last_became_peered":"2024-09-14T04:05:55.677254+0000","last_unstale":"2024-09-14T04:12:44.409104+0000","last_undegraded":"2024-09-14T04:12:44.409104+0000","last_fullsized":"2024-09-14T04:12:44.409104+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:53.163120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"240'487","reported_seq":508,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:45.815345+0000","last_change":"2024-09-14T04:12:25.836247+0000","last_active":"2024-09-14T04:12:45.815345+0000","last_peered":"2024-09-14T04:12:45.815345+0000","last_clean":"2024-09-14T04:12:45.815345+0000","last_became_active":"2024-09-14T04:12:25.835921+0000","last_became_peered":"2024-09-14T04:12:25.835921+0000","last_unstale":"2024-09-14T04:12:45.815345+0000","last_undegraded":"2024-09-14T04:12:45.815345+0000","last_fullsized":"2024-09-14T04:12:45.815345+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:51:27.528823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31916032,"num_objects":487,"num_object_clones":0,"num_object_copies":1461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":487,"num_write_kb":31168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":215,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.761768+0000","last_change":"2024-09-14T04:05:55.679201+0000","last_active":"2024-09-14T04:12:31.761768+0000","last_peered":"2024-09-14T04:12:31.761768+0000","last_clean":"2024-09-14T04:12:31.761768+0000","last_became_active":"2024-09-14T04:05:55.678843+0000","last_became_peered":"2024-09-14T04:05:55.678843+0000","last_unstale":"2024-09-14T04:12:31.761768+0000","last_undegraded":"2024-09-14T04:12:31.761768+0000","last_fullsized":"2024-09-14T04:12:31.761768+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:12:57.418514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":239,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.407152+0000","last_change":"2024-09-14T04:05:55.673650+0000","last_active":"2024-09-14T04:12:44.407152+0000","last_peered":"2024-09-14T04:12:44.407152+0000","last_clean":"2024-09-14T04:12:44.407152+0000","last_became_active":"2024-09-14T04:05:55.673258+0000","last_became_peered":"2024-09-14T04:05:55.673258+0000","last_unstale":"2024-09-14T04:12:44.407152+0000","last_undegraded":"2024-09-14T04:12:44.407152+0000","last_fullsized":"2024-09-14T04:12:44.407152+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:21.089797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"239'98","reported_seq":110,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:43.845500+0000","last_change":"2024-09-14T04:12:25.836150+0000","last_active":"2024-09-14T04:12:43.845500+0000","last_peered":"2024-09-14T04:12:43.845500+0000","last_clean":"2024-09-14T04:12:43.845500+0000","last_became_active":"2024-09-14T04:12:25.835919+0000","last_became_peered":"2024-09-14T04:12:25.835919+0000","last_unstale":"2024-09-14T04:12:43.845500+0000","last_undegraded":"2024-09-14T04:12:43.845500+0000","last_fullsized":"2024-09-14T04:12:43.845500+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:42:02.273888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6422528,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":6272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":223,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.406163+0000","last_change":"2024-09-14T04:05:55.671329+0000","last_active":"2024-09-14T04:12:44.406163+0000","last_peered":"2024-09-14T04:12:44.406163+0000","last_clean":"2024-09-14T04:12:44.406163+0000","last_became_active":"2024-09-14T04:05:55.671032+0000","last_became_peered":"2024-09-14T04:05:55.671032+0000","last_unstale":"2024-09-14T04:12:44.406163+0000","last_undegraded":"2024-09-14T04:12:44.406163+0000","last_fullsized":"2024-09-14T04:12:44.406163+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:58:10.779452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":136,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.761226+0000","last_change":"2024-09-14T04:05:55.679870+0000","last_active":"2024-09-14T04:12:31.761226+0000","last_peered":"2024-09-14T04:12:31.761226+0000","last_clean":"2024-09-14T04:12:31.761226+0000","last_became_active":"2024-09-14T04:05:55.679574+0000","last_became_peered":"2024-09-14T04:05:55.679574+0000","last_unstale":"2024-09-14T04:12:31.761226+0000","last_undegraded":"2024-09-14T04:12:31.761226+0000","last_fullsized":"2024-09-14T04:12:31.761226+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:51:09.931574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"240'881","reported_seq":902,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:47.871933+0000","last_change":"2024-09-14T04:12:25.831277+0000","last_active":"2024-09-14T04:12:47.871933+0000","last_peered":"2024-09-14T04:12:47.871933+0000","last_clean":"2024-09-14T04:12:47.871933+0000","last_became_active":"2024-09-14T04:12:25.830993+0000","last_became_peered":"2024-09-14T04:12:25.830993+0000","last_unstale":"2024-09-14T04:12:47.871933+0000","last_undegraded":"2024-09-14T04:12:47.871933+0000","last_fullsized":"2024-09-14T04:12:47.871933+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":881,"log_dups_size":0,"ondisk_log_size":881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:56:20.982511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57737216,"num_objects":881,"num_object_clones":0,"num_object_copies":2643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":881,"num_write_kb":56384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":102,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.409281+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:12:44.409281+0000","last_peered":"2024-09-14T04:12:44.409281+0000","last_clean":"2024-09-14T04:12:44.409281+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:12:44.409281+0000","last_undegraded":"2024-09-14T04:12:44.409281+0000","last_fullsized":"2024-09-14T04:12:44.409281+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":332,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.412435+0000","last_change":"2024-09-14T04:05:55.678520+0000","last_active":"2024-09-14T04:12:44.412435+0000","last_peered":"2024-09-14T04:12:44.412435+0000","last_clean":"2024-09-14T04:12:44.412435+0000","last_became_active":"2024-09-14T04:05:55.678008+0000","last_became_peered":"2024-09-14T04:05:55.678008+0000","last_unstale":"2024-09-14T04:12:44.412435+0000","last_undegraded":"2024-09-14T04:12:44.412435+0000","last_fullsized":"2024-09-14T04:12:44.412435+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:40:26.778212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"240'347","reported_seq":368,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.744103+0000","last_change":"2024-09-14T04:12:25.833053+0000","last_active":"2024-09-14T04:12:44.744103+0000","last_peered":"2024-09-14T04:12:44.744103+0000","last_clean":"2024-09-14T04:12:44.744103+0000","last_became_active":"2024-09-14T04:12:25.832051+0000","last_became_peered":"2024-09-14T04:12:25.832051+0000","last_unstale":"2024-09-14T04:12:44.744103+0000","last_undegraded":"2024-09-14T04:12:44.744103+0000","last_fullsized":"2024-09-14T04:12:44.744103+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:40:32.903707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":347,"num_write_kb":22208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":321,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.762179+0000","last_change":"2024-09-14T03:52:28.194320+0000","last_active":"2024-09-14T04:12:31.762179+0000","last_peered":"2024-09-14T04:12:31.762179+0000","last_clean":"2024-09-14T04:12:31.762179+0000","last_became_active":"2024-09-14T03:52:28.193716+0000","last_became_peered":"2024-09-14T03:52:28.193716+0000","last_unstale":"2024-09-14T04:12:31.762179+0000","last_undegraded":"2024-09-14T04:12:31.762179+0000","last_fullsized":"2024-09-14T04:12:31.762179+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:31.176444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":219,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.272141+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:12:37.272141+0000","last_peered":"2024-09-14T04:12:37.272141+0000","last_clean":"2024-09-14T04:12:37.272141+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:12:37.272141+0000","last_undegraded":"2024-09-14T04:12:37.272141+0000","last_fullsized":"2024-09-14T04:12:37.272141+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":102,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.409029+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:12:44.409029+0000","last_peered":"2024-09-14T04:12:44.409029+0000","last_clean":"2024-09-14T04:12:44.409029+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:12:44.409029+0000","last_undegraded":"2024-09-14T04:12:44.409029+0000","last_fullsized":"2024-09-14T04:12:44.409029+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":197,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.761698+0000","last_change":"2024-09-14T04:05:55.676445+0000","last_active":"2024-09-14T04:12:31.761698+0000","last_peered":"2024-09-14T04:12:31.761698+0000","last_clean":"2024-09-14T04:12:31.761698+0000","last_became_active":"2024-09-14T04:05:55.674141+0000","last_became_peered":"2024-09-14T04:05:55.674141+0000","last_unstale":"2024-09-14T04:12:31.761698+0000","last_undegraded":"2024-09-14T04:12:31.761698+0000","last_fullsized":"2024-09-14T04:12:31.761698+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:28:50.348309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"240'310","reported_seq":331,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.748498+0000","last_change":"2024-09-14T04:12:25.832752+0000","last_active":"2024-09-14T04:12:44.748498+0000","last_peered":"2024-09-14T04:12:44.748498+0000","last_clean":"2024-09-14T04:12:44.748498+0000","last_became_active":"2024-09-14T04:12:25.832479+0000","last_became_peered":"2024-09-14T04:12:25.832479+0000","last_unstale":"2024-09-14T04:12:44.748498+0000","last_undegraded":"2024-09-14T04:12:44.748498+0000","last_fullsized":"2024-09-14T04:12:44.748498+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:39:44.882371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":310,"num_write_kb":19840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":233,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.762166+0000","last_change":"2024-09-14T03:54:09.085780+0000","last_active":"2024-09-14T04:12:31.762166+0000","last_peered":"2024-09-14T04:12:31.762166+0000","last_clean":"2024-09-14T04:12:31.762166+0000","last_became_active":"2024-09-14T03:54:09.085358+0000","last_became_peered":"2024-09-14T03:54:09.085358+0000","last_unstale":"2024-09-14T04:12:31.762166+0000","last_undegraded":"2024-09-14T04:12:31.762166+0000","last_fullsized":"2024-09-14T04:12:31.762166+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:51.395999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":381,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.405539+0000","last_change":"2024-09-14T04:07:44.337314+0000","last_active":"2024-09-14T04:12:44.405539+0000","last_peered":"2024-09-14T04:12:44.405539+0000","last_clean":"2024-09-14T04:12:44.405539+0000","last_became_active":"2024-09-14T04:07:44.336916+0000","last_became_peered":"2024-09-14T04:07:44.336916+0000","last_unstale":"2024-09-14T04:12:44.405539+0000","last_undegraded":"2024-09-14T04:12:44.405539+0000","last_fullsized":"2024-09-14T04:12:44.405539+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:23.153578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":371,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.405749+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:12:44.405749+0000","last_peered":"2024-09-14T04:12:44.405749+0000","last_clean":"2024-09-14T04:12:44.405749+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:12:44.405749+0000","last_undegraded":"2024-09-14T04:12:44.405749+0000","last_fullsized":"2024-09-14T04:12:44.405749+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"240'299","reported_seq":320,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.741943+0000","last_change":"2024-09-14T04:12:25.832363+0000","last_active":"2024-09-14T04:12:44.741943+0000","last_peered":"2024-09-14T04:12:44.741943+0000","last_clean":"2024-09-14T04:12:44.741943+0000","last_became_active":"2024-09-14T04:12:25.831825+0000","last_became_peered":"2024-09-14T04:12:25.831825+0000","last_unstale":"2024-09-14T04:12:44.741943+0000","last_undegraded":"2024-09-14T04:12:44.741943+0000","last_fullsized":"2024-09-14T04:12:44.741943+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":299,"log_dups_size":0,"ondisk_log_size":299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:00.413726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19595264,"num_objects":299,"num_object_clones":0,"num_object_copies":897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":299,"num_write_kb":19136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":365,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.406315+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:12:44.406315+0000","last_peered":"2024-09-14T04:12:44.406315+0000","last_clean":"2024-09-14T04:12:44.406315+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:12:44.406315+0000","last_undegraded":"2024-09-14T04:12:44.406315+0000","last_fullsized":"2024-09-14T04:12:44.406315+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":326,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.411785+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:12:44.411785+0000","last_peered":"2024-09-14T04:12:44.411785+0000","last_clean":"2024-09-14T04:12:44.411785+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:12:44.411785+0000","last_undegraded":"2024-09-14T04:12:44.411785+0000","last_fullsized":"2024-09-14T04:12:44.411785+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":96,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.407041+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:12:44.407041+0000","last_peered":"2024-09-14T04:12:44.407041+0000","last_clean":"2024-09-14T04:12:44.407041+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:12:44.407041+0000","last_undegraded":"2024-09-14T04:12:44.407041+0000","last_fullsized":"2024-09-14T04:12:44.407041+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"240'258","reported_seq":279,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.730715+0000","last_change":"2024-09-14T04:12:25.834700+0000","last_active":"2024-09-14T04:12:44.730715+0000","last_peered":"2024-09-14T04:12:44.730715+0000","last_clean":"2024-09-14T04:12:44.730715+0000","last_became_active":"2024-09-14T04:12:25.834414+0000","last_became_peered":"2024-09-14T04:12:25.834414+0000","last_unstale":"2024-09-14T04:12:44.730715+0000","last_undegraded":"2024-09-14T04:12:44.730715+0000","last_fullsized":"2024-09-14T04:12:44.730715+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:10:46.003185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":258,"num_object_clones":0,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":258,"num_write_kb":16512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":220,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.272015+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:12:37.272015+0000","last_peered":"2024-09-14T04:12:37.272015+0000","last_clean":"2024-09-14T04:12:37.272015+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:12:37.272015+0000","last_undegraded":"2024-09-14T04:12:37.272015+0000","last_fullsized":"2024-09-14T04:12:37.272015+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":194,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.404701+0000","last_change":"2024-09-14T04:05:55.676693+0000","last_active":"2024-09-14T04:12:44.404701+0000","last_peered":"2024-09-14T04:12:44.404701+0000","last_clean":"2024-09-14T04:12:44.404701+0000","last_became_active":"2024-09-14T04:05:55.676335+0000","last_became_peered":"2024-09-14T04:05:55.676335+0000","last_unstale":"2024-09-14T04:12:44.404701+0000","last_undegraded":"2024-09-14T04:12:44.404701+0000","last_fullsized":"2024-09-14T04:12:44.404701+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:19:43.626016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":350,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.411820+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:12:44.411820+0000","last_peered":"2024-09-14T04:12:44.411820+0000","last_clean":"2024-09-14T04:12:44.411820+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:12:44.411820+0000","last_undegraded":"2024-09-14T04:12:44.411820+0000","last_fullsized":"2024-09-14T04:12:44.411820+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"240'515","reported_seq":536,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:45.819695+0000","last_change":"2024-09-14T04:12:25.832376+0000","last_active":"2024-09-14T04:12:45.819695+0000","last_peered":"2024-09-14T04:12:45.819695+0000","last_clean":"2024-09-14T04:12:45.819695+0000","last_became_active":"2024-09-14T04:12:25.832104+0000","last_became_peered":"2024-09-14T04:12:25.832104+0000","last_unstale":"2024-09-14T04:12:45.819695+0000","last_undegraded":"2024-09-14T04:12:45.819695+0000","last_fullsized":"2024-09-14T04:12:45.819695+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:33:00.041656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":515,"num_object_clones":0,"num_object_copies":1545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":32960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":243,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.272331+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:12:37.272331+0000","last_peered":"2024-09-14T04:12:37.272331+0000","last_clean":"2024-09-14T04:12:37.272331+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:12:37.272331+0000","last_undegraded":"2024-09-14T04:12:37.272331+0000","last_fullsized":"2024-09-14T04:12:37.272331+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":364,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.411915+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:12:44.411915+0000","last_peered":"2024-09-14T04:12:44.411915+0000","last_clean":"2024-09-14T04:12:44.411915+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:12:44.411915+0000","last_undegraded":"2024-09-14T04:12:44.411915+0000","last_fullsized":"2024-09-14T04:12:44.411915+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":321,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.410696+0000","last_change":"2024-09-14T04:05:55.674511+0000","last_active":"2024-09-14T04:12:44.410696+0000","last_peered":"2024-09-14T04:12:44.410696+0000","last_clean":"2024-09-14T04:12:44.410696+0000","last_became_active":"2024-09-14T04:05:55.674182+0000","last_became_peered":"2024-09-14T04:05:55.674182+0000","last_unstale":"2024-09-14T04:12:44.410696+0000","last_undegraded":"2024-09-14T04:12:44.410696+0000","last_fullsized":"2024-09-14T04:12:44.410696+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:12:34.453647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"240'878","reported_seq":899,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:47.882120+0000","last_change":"2024-09-14T04:12:25.834204+0000","last_active":"2024-09-14T04:12:47.882120+0000","last_peered":"2024-09-14T04:12:47.882120+0000","last_clean":"2024-09-14T04:12:47.882120+0000","last_became_active":"2024-09-14T04:12:25.833870+0000","last_became_peered":"2024-09-14T04:12:25.833870+0000","last_unstale":"2024-09-14T04:12:47.882120+0000","last_undegraded":"2024-09-14T04:12:47.882120+0000","last_fullsized":"2024-09-14T04:12:47.882120+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":878,"log_dups_size":0,"ondisk_log_size":878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:59.400608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57540608,"num_objects":878,"num_object_clones":0,"num_object_copies":2634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":878,"num_write_kb":56192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":325,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.412221+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:12:44.412221+0000","last_peered":"2024-09-14T04:12:44.412221+0000","last_clean":"2024-09-14T04:12:44.412221+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:12:44.412221+0000","last_undegraded":"2024-09-14T04:12:44.412221+0000","last_fullsized":"2024-09-14T04:12:44.412221+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":226,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.762779+0000","last_change":"2024-09-14T04:05:55.681238+0000","last_active":"2024-09-14T04:12:31.762779+0000","last_peered":"2024-09-14T04:12:31.762779+0000","last_clean":"2024-09-14T04:12:31.762779+0000","last_became_active":"2024-09-14T04:05:55.680946+0000","last_became_peered":"2024-09-14T04:05:55.680946+0000","last_unstale":"2024-09-14T04:12:31.762779+0000","last_undegraded":"2024-09-14T04:12:31.762779+0000","last_fullsized":"2024-09-14T04:12:31.762779+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:13:39.006545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":349,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.404393+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:12:44.404393+0000","last_peered":"2024-09-14T04:12:44.404393+0000","last_clean":"2024-09-14T04:12:44.404393+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:12:44.404393+0000","last_undegraded":"2024-09-14T04:12:44.404393+0000","last_fullsized":"2024-09-14T04:12:44.404393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"239'194","reported_seq":205,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.221723+0000","last_change":"2024-09-14T04:12:25.838838+0000","last_active":"2024-09-14T04:12:44.221723+0000","last_peered":"2024-09-14T04:12:44.221723+0000","last_clean":"2024-09-14T04:12:44.221723+0000","last_became_active":"2024-09-14T04:12:25.838336+0000","last_became_peered":"2024-09-14T04:12:25.838336+0000","last_unstale":"2024-09-14T04:12:44.221723+0000","last_undegraded":"2024-09-14T04:12:44.221723+0000","last_fullsized":"2024-09-14T04:12:44.221723+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:13:06.383742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12713984,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":194,"num_write_kb":12416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":317,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.409352+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:12:44.409352+0000","last_peered":"2024-09-14T04:12:44.409352+0000","last_clean":"2024-09-14T04:12:44.409352+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:12:44.409352+0000","last_undegraded":"2024-09-14T04:12:44.409352+0000","last_fullsized":"2024-09-14T04:12:44.409352+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":332,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.412820+0000","last_change":"2024-09-14T04:05:55.676617+0000","last_active":"2024-09-14T04:12:44.412820+0000","last_peered":"2024-09-14T04:12:44.412820+0000","last_clean":"2024-09-14T04:12:44.412820+0000","last_became_active":"2024-09-14T04:05:55.676318+0000","last_became_peered":"2024-09-14T04:05:55.676318+0000","last_unstale":"2024-09-14T04:12:44.412820+0000","last_undegraded":"2024-09-14T04:12:44.412820+0000","last_fullsized":"2024-09-14T04:12:44.412820+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:38.529611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":51,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.271631+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:12:37.271631+0000","last_peered":"2024-09-14T04:12:37.271631+0000","last_clean":"2024-09-14T04:12:37.271631+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:12:37.271631+0000","last_undegraded":"2024-09-14T04:12:37.271631+0000","last_fullsized":"2024-09-14T04:12:37.271631+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.e","version":"240'921","reported_seq":942,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:47.878156+0000","last_change":"2024-09-14T04:12:25.835349+0000","last_active":"2024-09-14T04:12:47.878156+0000","last_peered":"2024-09-14T04:12:47.878156+0000","last_clean":"2024-09-14T04:12:47.878156+0000","last_became_active":"2024-09-14T04:12:25.835042+0000","last_became_peered":"2024-09-14T04:12:25.835042+0000","last_unstale":"2024-09-14T04:12:47.878156+0000","last_undegraded":"2024-09-14T04:12:47.878156+0000","last_fullsized":"2024-09-14T04:12:47.878156+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":235,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:12:24.817148+0000","last_clean_scrub_stamp":"2024-09-14T04:12:24.817148+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:06:41.980408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60358656,"num_objects":921,"num_object_clones":0,"num_object_copies":2763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":921,"num_write_kb":58944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":374,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.406046+0000","last_change":"2024-09-14T04:05:55.690010+0000","last_active":"2024-09-14T04:12:44.406046+0000","last_peered":"2024-09-14T04:12:44.406046+0000","last_clean":"2024-09-14T04:12:44.406046+0000","last_became_active":"2024-09-14T04:05:55.670892+0000","last_became_peered":"2024-09-14T04:05:55.670892+0000","last_unstale":"2024-09-14T04:12:44.406046+0000","last_undegraded":"2024-09-14T04:12:44.406046+0000","last_fullsized":"2024-09-14T04:12:44.406046+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:40:33.024617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":217,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.761685+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:12:31.761685+0000","last_peered":"2024-09-14T04:12:31.761685+0000","last_clean":"2024-09-14T04:12:31.761685+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:12:31.761685+0000","last_undegraded":"2024-09-14T04:12:31.761685+0000","last_fullsized":"2024-09-14T04:12:31.761685+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":83,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.271656+0000","last_change":"2024-09-14T04:05:55.674629+0000","last_active":"2024-09-14T04:12:37.271656+0000","last_peered":"2024-09-14T04:12:37.271656+0000","last_clean":"2024-09-14T04:12:37.271656+0000","last_became_active":"2024-09-14T04:05:55.674242+0000","last_became_peered":"2024-09-14T04:05:55.674242+0000","last_unstale":"2024-09-14T04:12:37.271656+0000","last_undegraded":"2024-09-14T04:12:37.271656+0000","last_fullsized":"2024-09-14T04:12:37.271656+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:06:16.827114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":95,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.761648+0000","last_change":"2024-09-14T04:05:55.677636+0000","last_active":"2024-09-14T04:12:31.761648+0000","last_peered":"2024-09-14T04:12:31.761648+0000","last_clean":"2024-09-14T04:12:31.761648+0000","last_became_active":"2024-09-14T04:05:55.677313+0000","last_became_peered":"2024-09-14T04:05:55.677313+0000","last_unstale":"2024-09-14T04:12:31.761648+0000","last_undegraded":"2024-09-14T04:12:31.761648+0000","last_fullsized":"2024-09-14T04:12:31.761648+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:00:13.489641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":135,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.762624+0000","last_change":"2024-09-14T04:05:55.686083+0000","last_active":"2024-09-14T04:12:31.762624+0000","last_peered":"2024-09-14T04:12:31.762624+0000","last_clean":"2024-09-14T04:12:31.762624+0000","last_became_active":"2024-09-14T04:05:55.673710+0000","last_became_peered":"2024-09-14T04:05:55.673710+0000","last_unstale":"2024-09-14T04:12:31.762624+0000","last_undegraded":"2024-09-14T04:12:31.762624+0000","last_fullsized":"2024-09-14T04:12:31.762624+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:25.169218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":325,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.410587+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:12:44.410587+0000","last_peered":"2024-09-14T04:12:44.410587+0000","last_clean":"2024-09-14T04:12:44.410587+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:12:44.410587+0000","last_undegraded":"2024-09-14T04:12:44.410587+0000","last_fullsized":"2024-09-14T04:12:44.410587+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":223,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.762712+0000","last_change":"2024-09-14T04:05:55.698766+0000","last_active":"2024-09-14T04:12:31.762712+0000","last_peered":"2024-09-14T04:12:31.762712+0000","last_clean":"2024-09-14T04:12:31.762712+0000","last_became_active":"2024-09-14T04:05:55.674923+0000","last_became_peered":"2024-09-14T04:05:55.674923+0000","last_unstale":"2024-09-14T04:12:31.762712+0000","last_undegraded":"2024-09-14T04:12:31.762712+0000","last_fullsized":"2024-09-14T04:12:31.762712+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:52:37.679728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":206,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.272340+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:12:37.272340+0000","last_peered":"2024-09-14T04:12:37.272340+0000","last_clean":"2024-09-14T04:12:37.272340+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:12:37.272340+0000","last_undegraded":"2024-09-14T04:12:37.272340+0000","last_fullsized":"2024-09-14T04:12:37.272340+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":277,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.410915+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:12:44.410915+0000","last_peered":"2024-09-14T04:12:44.410915+0000","last_clean":"2024-09-14T04:12:44.410915+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:12:44.410915+0000","last_undegraded":"2024-09-14T04:12:44.410915+0000","last_fullsized":"2024-09-14T04:12:44.410915+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":373,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.404670+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:12:44.404670+0000","last_peered":"2024-09-14T04:12:44.404670+0000","last_clean":"2024-09-14T04:12:44.404670+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:12:44.404670+0000","last_undegraded":"2024-09-14T04:12:44.404670+0000","last_fullsized":"2024-09-14T04:12:44.404670+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":289,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.409412+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:12:44.409412+0000","last_peered":"2024-09-14T04:12:44.409412+0000","last_clean":"2024-09-14T04:12:44.409412+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:12:44.409412+0000","last_undegraded":"2024-09-14T04:12:44.409412+0000","last_fullsized":"2024-09-14T04:12:44.409412+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":214,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.761611+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:12:31.761611+0000","last_peered":"2024-09-14T04:12:31.761611+0000","last_clean":"2024-09-14T04:12:31.761611+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:12:31.761611+0000","last_undegraded":"2024-09-14T04:12:31.761611+0000","last_fullsized":"2024-09-14T04:12:31.761611+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":102,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.408603+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:12:44.408603+0000","last_peered":"2024-09-14T04:12:44.408603+0000","last_clean":"2024-09-14T04:12:44.408603+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:12:44.408603+0000","last_undegraded":"2024-09-14T04:12:44.408603+0000","last_fullsized":"2024-09-14T04:12:44.408603+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":375,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.405955+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:12:44.405955+0000","last_peered":"2024-09-14T04:12:44.405955+0000","last_clean":"2024-09-14T04:12:44.405955+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:12:44.405955+0000","last_undegraded":"2024-09-14T04:12:44.405955+0000","last_fullsized":"2024-09-14T04:12:44.405955+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":331,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.411110+0000","last_change":"2024-09-14T04:05:55.677962+0000","last_active":"2024-09-14T04:12:44.411110+0000","last_peered":"2024-09-14T04:12:44.411110+0000","last_clean":"2024-09-14T04:12:44.411110+0000","last_became_active":"2024-09-14T04:05:55.677277+0000","last_became_peered":"2024-09-14T04:05:55.677277+0000","last_unstale":"2024-09-14T04:12:44.411110+0000","last_undegraded":"2024-09-14T04:12:44.411110+0000","last_fullsized":"2024-09-14T04:12:44.411110+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:24.908568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":102,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.409508+0000","last_change":"2024-09-14T04:07:44.338546+0000","last_active":"2024-09-14T04:12:44.409508+0000","last_peered":"2024-09-14T04:12:44.409508+0000","last_clean":"2024-09-14T04:12:44.409508+0000","last_became_active":"2024-09-14T04:07:44.338210+0000","last_became_peered":"2024-09-14T04:07:44.338210+0000","last_unstale":"2024-09-14T04:12:44.409508+0000","last_undegraded":"2024-09-14T04:12:44.409508+0000","last_fullsized":"2024-09-14T04:12:44.409508+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:56.100013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":155,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.410788+0000","last_change":"2024-09-14T04:05:55.671091+0000","last_active":"2024-09-14T04:12:44.410788+0000","last_peered":"2024-09-14T04:12:44.410788+0000","last_clean":"2024-09-14T04:12:44.410788+0000","last_became_active":"2024-09-14T04:05:55.670180+0000","last_became_peered":"2024-09-14T04:05:55.670180+0000","last_unstale":"2024-09-14T04:12:44.410788+0000","last_undegraded":"2024-09-14T04:12:44.410788+0000","last_fullsized":"2024-09-14T04:12:44.410788+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:33.489272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":367,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.406485+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:12:44.406485+0000","last_peered":"2024-09-14T04:12:44.406485+0000","last_clean":"2024-09-14T04:12:44.406485+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:12:44.406485+0000","last_undegraded":"2024-09-14T04:12:44.406485+0000","last_fullsized":"2024-09-14T04:12:44.406485+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":358,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.404480+0000","last_change":"2024-09-14T04:05:55.671536+0000","last_active":"2024-09-14T04:12:44.404480+0000","last_peered":"2024-09-14T04:12:44.404480+0000","last_clean":"2024-09-14T04:12:44.404480+0000","last_became_active":"2024-09-14T04:05:55.671029+0000","last_became_peered":"2024-09-14T04:05:55.671029+0000","last_unstale":"2024-09-14T04:12:44.404480+0000","last_undegraded":"2024-09-14T04:12:44.404480+0000","last_fullsized":"2024-09-14T04:12:44.404480+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:35.745580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":264,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.408771+0000","last_change":"2024-09-14T04:05:55.681242+0000","last_active":"2024-09-14T04:12:44.408771+0000","last_peered":"2024-09-14T04:12:44.408771+0000","last_clean":"2024-09-14T04:12:44.408771+0000","last_became_active":"2024-09-14T04:05:55.680502+0000","last_became_peered":"2024-09-14T04:05:55.680502+0000","last_unstale":"2024-09-14T04:12:44.408771+0000","last_undegraded":"2024-09-14T04:12:44.408771+0000","last_fullsized":"2024-09-14T04:12:44.408771+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:23.760294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":222,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.272108+0000","last_change":"2024-09-14T04:05:55.687662+0000","last_active":"2024-09-14T04:12:37.272108+0000","last_peered":"2024-09-14T04:12:37.272108+0000","last_clean":"2024-09-14T04:12:37.272108+0000","last_became_active":"2024-09-14T04:05:55.673561+0000","last_became_peered":"2024-09-14T04:05:55.673561+0000","last_unstale":"2024-09-14T04:12:37.272108+0000","last_undegraded":"2024-09-14T04:12:37.272108+0000","last_fullsized":"2024-09-14T04:12:37.272108+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:14:51.708356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":264,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.409400+0000","last_change":"2024-09-14T04:05:55.679885+0000","last_active":"2024-09-14T04:12:44.409400+0000","last_peered":"2024-09-14T04:12:44.409400+0000","last_clean":"2024-09-14T04:12:44.409400+0000","last_became_active":"2024-09-14T04:05:55.678589+0000","last_became_peered":"2024-09-14T04:05:55.678589+0000","last_unstale":"2024-09-14T04:12:44.409400+0000","last_undegraded":"2024-09-14T04:12:44.409400+0000","last_fullsized":"2024-09-14T04:12:44.409400+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:51:32.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":205,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-09-14T04:12:37.272258+0000","last_change":"2024-09-14T04:05:55.697832+0000","last_active":"2024-09-14T04:12:37.272258+0000","last_peered":"2024-09-14T04:12:37.272258+0000","last_clean":"2024-09-14T04:12:37.272258+0000","last_became_active":"2024-09-14T04:05:55.673923+0000","last_became_peered":"2024-09-14T04:05:55.673923+0000","last_unstale":"2024-09-14T04:12:37.272258+0000","last_undegraded":"2024-09-14T04:12:37.272258+0000","last_fullsized":"2024-09-14T04:12:37.272258+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:44.697302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":102,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.410775+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:12:44.410775+0000","last_peered":"2024-09-14T04:12:44.410775+0000","last_clean":"2024-09-14T04:12:44.410775+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:12:44.410775+0000","last_undegraded":"2024-09-14T04:12:44.410775+0000","last_fullsized":"2024-09-14T04:12:44.410775+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":102,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.408895+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:12:44.408895+0000","last_peered":"2024-09-14T04:12:44.408895+0000","last_clean":"2024-09-14T04:12:44.408895+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:12:44.408895+0000","last_undegraded":"2024-09-14T04:12:44.408895+0000","last_fullsized":"2024-09-14T04:12:44.408895+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":156,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-14T04:12:31.761021+0000","last_change":"2024-09-14T04:07:44.338957+0000","last_active":"2024-09-14T04:12:31.761021+0000","last_peered":"2024-09-14T04:12:31.761021+0000","last_clean":"2024-09-14T04:12:31.761021+0000","last_became_active":"2024-09-14T04:07:44.338190+0000","last_became_peered":"2024-09-14T04:07:44.338190+0000","last_unstale":"2024-09-14T04:12:31.761021+0000","last_undegraded":"2024-09-14T04:12:31.761021+0000","last_fullsized":"2024-09-14T04:12:31.761021+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:36:27.488115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":363,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-09-14T04:12:44.404263+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:12:44.404263+0000","last_peered":"2024-09-14T04:12:44.404263+0000","last_clean":"2024-09-14T04:12:44.404263+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:12:44.404263+0000","last_undegraded":"2024-09-14T04:12:44.404263+0000","last_fullsized":"2024-09-14T04:12:44.404263+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":455737344,"num_objects":6954,"num_object_clones":0,"num_object_copies":20862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6954,"num_write_kb":445056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1543307264,"data_stored":1543307264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6954,"ondisk_log_size":6954,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4890,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13709,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430588,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":963264,"kb_used_data":60216,"kb_used_omap":1,"kb_used_meta":903038,"kb_avail":92781888,"statfs":{"total":95995035648,"available":95008653312,"internally_reserved":0,"allocated":61661184,"data_stored":60546582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924711256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626817,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903736,"kb_used_data":10544,"kb_used_omap":1,"kb_used_meta":893182,"kb_avail":92841416,"statfs":{"total":95995035648,"available":95069609984,"internally_reserved":0,"allocated":10797056,"data_stored":9690238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914618712},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823045,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917220,"kb_used_data":20252,"kb_used_omap":1,"kb_used_meta":896958,"kb_avail":92827932,"statfs":{"total":95995035648,"available":95055802368,"internally_reserved":0,"allocated":20738048,"data_stored":19646643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918485336},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":798863917128,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552426,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":621108,"kb_used_data":104808,"kb_used_omap":4,"kb_used_meta":516283,"kb_avail":93124044,"statfs":{"total":95995035648,"available":95359021056,"internally_reserved":0,"allocated":107323392,"data_stored":106228083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":528674237},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435468,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":658616,"kb_used_data":190832,"kb_used_omap":6,"kb_used_meta":467769,"kb_avail":93086536,"statfs":{"total":95995035648,"available":95320612864,"internally_reserved":0,"allocated":195411968,"data_stored":194312263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":478995917},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827938,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1157044,"kb_used_data":492460,"kb_used_omap":5,"kb_used_meta":664570,"kb_avail":92588108,"statfs":{"total":95995035648,"available":94810222592,"internally_reserved":0,"allocated":504279040,"data_stored":503179586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":680519919},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383926,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1231096,"kb_used_data":460848,"kb_used_omap":5,"kb_used_meta":770234,"kb_avail":92514056,"statfs":{"total":95995035648,"available":94734393344,"internally_reserved":0,"allocated":471908352,"data_stored":470805688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":788719676},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471138304,"data_stored":471138304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":531693568,"data_stored":531693568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":202571776,"data_stored":202571776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":137494528,"data_stored":137494528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":62717952,"data_stored":62717952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53870592,"data_stored":53870592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":83820544,"data_stored":83820544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:12:50.874 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-14T04:12:50.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_6 pg_num 26 2024-09-14T04:12:50.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:50.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:50.984 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:51.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:51.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:51.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:51.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:51.223 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 22477 22461 175.433 149.062 0.00685175 0.00569304 2024-09-14T04:12:51.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:51.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:51.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:51.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:51.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:51.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:51.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:51.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:51.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:51.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:51.696 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:12:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:12:51] "GET /metrics HTTP/1.1" 200 36489 "" "Prometheus/2.51.0" 2024-09-14T04:12:51.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:51.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:51.743 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:12:51.743 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:12:51.743 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:12:51.743 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:12:51.743 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:12:51.837 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:51.867 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:51.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:51.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:51.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:52.007 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:52.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:52.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:52.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:52.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:52.223 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 24852 24836 172.432 148.438 0.00901886 0.00579381 2024-09-14T04:12:52.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:52.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:52.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:52.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:52.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:52.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:52.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:52.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:52.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:52.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:52.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:52.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:52.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:52.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:52.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:52.813 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:52.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:52.827 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:52 smithi114 ceph-mon[103029]: pgmap v886: 89 pgs: 89 active+clean; 1.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-14T04:12:52.827 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:52 smithi114 ceph-mon[106969]: pgmap v886: 89 pgs: 89 active+clean; 1.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-14T04:12:52.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-14T04:12:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:52 smithi160 ceph-mon[88407]: pgmap v886: 89 pgs: 89 active+clean; 1.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-14T04:12:52.915 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:52.935 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:52.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:52.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:53.019 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:53.038 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:12:53.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:53.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:53.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:53.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:53.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:53.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:53.223 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 26936 26920 168.212 130.25 0.00571644 0.00593898 2024-09-14T04:12:53.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:53.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:53.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:53.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:53.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:53.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:53.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:53.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:53.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:53.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:53.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:53.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:53.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:53.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:53.872 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:53.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:53 smithi160 ceph-mon[88407]: pgmap v887: 89 pgs: 89 active+clean; 1.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-14T04:12:53.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:54.008 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:54.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:54.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:54.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:53 smithi114 ceph-mon[103029]: pgmap v887: 89 pgs: 89 active+clean; 1.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-14T04:12:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:12:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:12:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:53 smithi114 ceph-mon[106969]: pgmap v887: 89 pgs: 89 active+clean; 1.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 110 MiB/s wr, 1.76k op/s 2024-09-14T04:12:54.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:54.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:54.223 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 28941 28925 164.312 125.312 0.00189947 0.00607448 2024-09-14T04:12:54.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:54.276 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:54.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:54.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:54.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:54.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:54.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:54.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:54.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:54.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:54.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:54.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:54.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:54.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:54.809 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:54.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:54.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:54.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:54.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:55.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:55.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:55.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:55.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:55.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:55.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:55.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:55.223 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 30882 30866 160.727 121.312 0.00399205 0.00620744 2024-09-14T04:12:55.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:55.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:55.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:55.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:55.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:55.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:55.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:55.627 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:55.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:55.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:55.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:12:55.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:55.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:12:55.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:55.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:55.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:55.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:55.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:56.028 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:56.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:56.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:55 smithi114 ceph-mon[106969]: pgmap v888: 89 pgs: 89 active+clean; 1.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-09-14T04:12:56.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:55 smithi114 ceph-mon[103029]: pgmap v888: 89 pgs: 89 active+clean; 1.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-09-14T04:12:56.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:56.160 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:56.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:56.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:56.223 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 32423 32407 155.772 96.3125 0.00798479 0.00641424 2024-09-14T04:12:56.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:56.286 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:56.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:56.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:55 smithi160 ceph-mon[88407]: pgmap v888: 89 pgs: 89 active+clean; 1.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 160 MiB/s wr, 2.56k op/s 2024-09-14T04:12:56.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:56.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:56.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:56.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:56.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:56.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:56.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:56.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:56.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:56.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:56.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:56.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:56.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:56.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:56.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:57.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:57.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:57.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:57.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:57.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:57.224 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 33765 33749 150.636 83.875 0.00533668 0.00662533 2024-09-14T04:12:57.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:12:57.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:57.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:12:57.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:57.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:57.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:57.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:57.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:57.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:57.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:57.710 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:57.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:57.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:57.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:57.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:57.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:57.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:57.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:57.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:58.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:58.099 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:58.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:58.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:58.224 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 35316 35300 147.056 96.9375 0.00444632 0.00679458 2024-09-14T04:12:58.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:58.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:58.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:57 smithi114 ceph-mon[103029]: pgmap v889: 89 pgs: 89 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-09-14T04:12:58.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:57 smithi114 ceph-mon[106969]: pgmap v889: 89 pgs: 89 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-09-14T04:12:58.371 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:58.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:58.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:57 smithi160 ceph-mon[88407]: pgmap v889: 89 pgs: 89 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 139 MiB/s wr, 2.22k op/s 2024-09-14T04:12:58.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:12:58.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:58.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:12:58.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:12:58.625 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:58.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:12:58.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:58.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:58.764 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:58.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:58.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:58.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:58.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:58.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:59.024 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:59.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:59.126 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:59.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:59.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:59.224 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 36585 36569 142.822 79.3125 0.00327805 0.00695826 2024-09-14T04:12:59.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:12:59.276 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:12:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:59.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:12:59.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:59.420 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:59.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:59.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:12:59.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:59.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:12:59.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:59.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:59.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:59.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:12:59.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:12:59.838 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:59.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:59.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:12:59.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:12:59.979 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:12:59.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:12:59.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:00.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:00.115 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:00.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:00.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:00.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:12:59 smithi160 ceph-mon[88407]: pgmap v890: 89 pgs: 89 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T04:13:00.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:00.224 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 38208 38192 140.387 101.438 0.0130619 0.00711729 2024-09-14T04:13:00.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:00.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:00.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:00.263 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:12:59 smithi114 ceph-mon[106969]: pgmap v890: 89 pgs: 89 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T04:13:00.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:12:59 smithi114 ceph-mon[103029]: pgmap v890: 89 pgs: 89 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T04:13:00.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:00.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:00.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:00.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:00.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:00.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:00.576 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:13:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:13:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:13:00.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:00.642 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:00.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:00.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:00.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:00.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:00.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:00.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:00.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:00.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:01.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:01.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:01.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:01.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:01.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:01.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:01.224 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 39550 39534 137.247 83.875 0.00229627 0.00727697 2024-09-14T04:13:01.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:01.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:01.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:01.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:01.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:01.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:01.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:01.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:01.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:01.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:01.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:01.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:01.724 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:13:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:13:01] "GET /metrics HTTP/1.1" 200 36556 "" "Prometheus/2.51.0" 2024-09-14T04:13:01.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:01.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:01.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:01.937 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:01.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:01.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:02.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:02.081 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:01 smithi114 ceph-mon[106969]: pgmap v891: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-14T04:13:02.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:01 smithi114 ceph-mon[103029]: pgmap v891: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-14T04:13:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:02.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:02.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:02.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:02.224 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 40952 40936 134.635 87.625 0.0060807 0.00742195 2024-09-14T04:13:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:02.241 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:02.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:02.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:02.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:02.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:01 smithi160 ceph-mon[88407]: pgmap v891: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-09-14T04:13:02.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:02.513 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:02.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:02.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:02.643 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:02.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:02.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:02.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:02.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:02.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:02.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:02.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:02.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:03.005 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:03.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:03.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:03.134 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:03.159 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:03.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:03.224 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:13:03.224214+0000 min lat: 0.00112332 max lat: 0.189793 avg lat: 0.00749041 2024-09-14T04:13:03.225 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:13:03.225 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 42709 42693 133.393 109.812 0.0057805 0.00749041 2024-09-14T04:13:03.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:03.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:03.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:03.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:03.434 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:03.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:03.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:03.576 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:03.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:03.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:03.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:03.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:03.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:03.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:03.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:03.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:03.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:03.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:04.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:04.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:04.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:04.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:03 smithi114 ceph-mon[106969]: pgmap v892: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-09-14T04:13:04.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:03 smithi114 ceph-mon[103029]: pgmap v892: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-09-14T04:13:04.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:04.225 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 44229 44213 131.562 95 0.00450784 0.00759472 2024-09-14T04:13:04.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:04.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:04.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:04.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:04.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:04.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:04.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:03 smithi160 ceph-mon[88407]: pgmap v892: 89 pgs: 89 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-09-14T04:13:04.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:04.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:04.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:04.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:04.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:04.670 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:04.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:04.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:04.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:04.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:04.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:04.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:04.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:05.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:05.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:05.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:05.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:05.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:05.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:05.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:05.224 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 45770 45754 129.96 96.3125 0.0058186 0.00768957 2024-09-14T04:13:05.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:05.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:05.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:05.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:05.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:05.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:05.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:05.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:05.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:05.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:05.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:05.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:05.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:05.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:05.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:05.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:05.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:06.020 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:06.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:05 smithi114 ceph-mon[106969]: pgmap v893: 89 pgs: 89 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-09-14T04:13:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:05 smithi114 ceph-mon[103029]: pgmap v893: 89 pgs: 89 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-09-14T04:13:06.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:06.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:06.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:06.225 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 47454 47438 128.885 105.25 0.00589267 0.00775364 2024-09-14T04:13:06.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:06.291 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:06.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:06.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:06.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:05 smithi160 ceph-mon[88407]: pgmap v893: 89 pgs: 89 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-09-14T04:13:06.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:06.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:06.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:06.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:06.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:06.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:06.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:06.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:06.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:13:06.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:13:06.728 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:13:06.729 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:13:06.729 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:13:06.729 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:13:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:13:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:13:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:13:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:13:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:13:06.729 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:13:06.730 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:13:06.730 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:13:06.730 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:13:06.730 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:13:06.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:06.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:06.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:06.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:06.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:06.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:07.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:07.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:07.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:07.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:07.226 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 49001 48985 127.537 96.6875 0.0139666 0.00783556 2024-09-14T04:13:07.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:07.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:07.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:07.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1381468531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:13:07.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:06 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:07.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:06 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:07.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1381468531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:13:07.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:07.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:07.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:07.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:07.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:07.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1381468531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:13:07.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:06 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:07.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:06 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:07.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:07.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:07.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:07.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:07.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:07.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:07.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:07.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:07.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:07.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:07.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:07.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:07.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:07.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:08.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:08.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:08.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:08.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:08.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:08.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:08.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:08.226 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 50096 50080 125.172 68.4375 0.00285752 0.00791474 2024-09-14T04:13:08.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:08.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:08.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:08.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:07 smithi114 ceph-mon[103029]: pgmap v894: 89 pgs: 89 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T04:13:08.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:08.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:07 smithi114 ceph-mon[106969]: pgmap v894: 89 pgs: 89 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T04:13:08.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:08.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:08.403 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:08.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:08.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:08.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:07 smithi160 ceph-mon[88407]: pgmap v894: 89 pgs: 89 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T04:13:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:08.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:08.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:08.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:08.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:08.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:08.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:08.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:08.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:08.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:08.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:08.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:08.931 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:08.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:09.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:09.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:09.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:09.112 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:13:09.112 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:13:09.112 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:09.112 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:13:09.113 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:13:09.113 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:09.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:09.218 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:09.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:09.226 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 51220 51204 123.06 70.25 0.00505228 0.00812088 2024-09-14T04:13:09.312 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:13:09.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:09.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:09.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:13:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:13:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:09.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:09.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:09.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:09.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:09.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:09.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:09.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:09.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:09.742 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:09.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:09.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:09.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:09.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:09.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:10.013 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:10.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:10.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:10.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:10.145 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:10.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:10.214 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:09 smithi160 ceph-mon[88407]: pgmap v895: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-14T04:13:10.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:09 smithi114 ceph-mon[103029]: pgmap v895: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-14T04:13:10.219 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:09 smithi114 ceph-mon[106969]: pgmap v895: 89 pgs: 89 active+clean; 2.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-14T04:13:10.228 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 52784 52768 122.122 97.75 0.0128547 0.00818227 2024-09-14T04:13:10.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:10.285 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:10.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:10.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:10.426 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:10.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:10.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:10.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:10.553 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:10.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:10.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:13:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:13:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:13:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:10.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:10.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:10.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:10.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:10.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:10.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:10.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:10.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:11.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:11.092 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:11.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:11.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:11.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:11.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:11.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:11.226 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 54010 53994 120.497 76.625 0.00275915 0.00827307 2024-09-14T04:13:11.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:10 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3852181009' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:13:11.308 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:10 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:13:11.308 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:10 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3852181009' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:13:11.308 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:10 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:13:11.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:11.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:11.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:10 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3852181009' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:13:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:10 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:13:11.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:11.475 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:11.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:11.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:11.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:11.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:13:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:13:11] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-14T04:13:11.716 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:11.758 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:11.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:11.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:11.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:11.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:11.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:12.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:12.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:12.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:12.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:12.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:12.207 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:13:12.227 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 55453 55437 119.452 90.1875 0.0174762 0.0083665 2024-09-14T04:13:12.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:12.305 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:12.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:12.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:11 smithi114 ceph-mon[103029]: pgmap v896: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-09-14T04:13:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:11 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:13:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:11 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:13:12.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:11 smithi114 ceph-mon[103029]: osdmap e241: 8 total, 8 up, 7 in 2024-09-14T04:13:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:11 smithi114 ceph-mon[106969]: pgmap v896: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-09-14T04:13:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:11 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:13:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:11 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:13:12.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:11 smithi114 ceph-mon[106969]: osdmap e241: 8 total, 8 up, 7 in 2024-09-14T04:13:12.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:11 smithi160 ceph-mon[88407]: pgmap v896: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 105 MiB/s wr, 1.69k op/s 2024-09-14T04:13:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:11 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:13:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:11 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:13:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:11 smithi160 ceph-mon[88407]: osdmap e241: 8 total, 8 up, 7 in 2024-09-14T04:13:12.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:12.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:12.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:12.564 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:12.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:12.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:12.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:12.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:12.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:12.823 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:12.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:12.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:12.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:12.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:13.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:13.095 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:13.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:13.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:13.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:13.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 57017 57001 118.726 97.75 0.0142641 0.00841655 2024-09-14T04:13:13.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:13.239 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:13.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:13.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:13.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:13.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:13.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:13.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:13.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:13.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:13.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:13.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:13.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:13.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:13.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:13.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:13.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:13.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:13.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:13.990 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 11 pg_num to 26 2024-09-14T04:13:13.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:14.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:14.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:14.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:14.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:14.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:14.228 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 58439 58423 117.762 88.875 0.0100226 0.0084867 2024-09-14T04:13:14.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:14.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:14.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:14.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:13 smithi114 ceph-mon[106969]: pgmap v898: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T04:13:14.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:13 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/495716849' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:13:14.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:13 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:13:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:13 smithi114 ceph-mon[103029]: pgmap v898: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T04:13:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:13 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/495716849' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:13:14.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:13 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:13:14.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:13 smithi160 ceph-mon[88407]: pgmap v898: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-14T04:13:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:13 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/495716849' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:13:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:13 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:13:14.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:14.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:14.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:14.562 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:14.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:14.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:14.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:14.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:14.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:14.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:14.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:14.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:14.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:14.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:14.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:14.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:15.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:13:15.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:15.109 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:14 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-09-14T04:13:15.109 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:14 smithi114 ceph-mon[103029]: osdmap e242: 8 total, 8 up, 7 in 2024-09-14T04:13:15.109 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:14 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-14T04:13:15.109 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:14 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-09-14T04:13:15.110 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:14 smithi114 ceph-mon[106969]: osdmap e242: 8 total, 8 up, 7 in 2024-09-14T04:13:15.110 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:14 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-14T04:13:15.156 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:15.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:15.228 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 59612 59596 116.373 73.3125 0.0696838 0.0085351 2024-09-14T04:13:15.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:15.279 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:15.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:15.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:15.403 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:15.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:15.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:14 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-09-14T04:13:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:14 smithi160 ceph-mon[88407]: osdmap e242: 8 total, 8 up, 7 in 2024-09-14T04:13:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:14 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-14T04:13:15.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:15.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:15.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:15.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:15.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:15.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:15.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:13:15.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:15.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:15.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:15.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:15.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:15.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:15.997 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:15 smithi114 ceph-mon[103029]: pgmap v900: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-14T04:13:15.998 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:15 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-14T04:13:15.998 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:15 smithi114 ceph-mon[103029]: osdmap e243: 8 total, 8 up, 7 in 2024-09-14T04:13:15.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:16.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:16.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:16.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:16.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:16.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:16.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:16.228 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 59910 59894 113.411 18.625 0.0285925 0.00881027 2024-09-14T04:13:16.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:16.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:16.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:16.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:15 smithi114 ceph-mon[106969]: pgmap v900: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-14T04:13:16.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:15 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-14T04:13:16.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:15 smithi114 ceph-mon[106969]: osdmap e243: 8 total, 8 up, 7 in 2024-09-14T04:13:16.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:15 smithi160 ceph-mon[88407]: pgmap v900: 89 pgs: 89 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-14T04:13:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:15 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-14T04:13:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:15 smithi160 ceph-mon[88407]: osdmap e243: 8 total, 8 up, 7 in 2024-09-14T04:13:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:16.443 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:16.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:16.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:16.573 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:16.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:16.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:16.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:16.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:16.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:16.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:16.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:16.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:16.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:17.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:17.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:17.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:17.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:17 smithi114 ceph-mon[103029]: osdmap e244: 8 total, 8 up, 7 in 2024-09-14T04:13:17.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:17 smithi114 ceph-mon[103029]: 11.3 scrub starts 2024-09-14T04:13:17.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:17 smithi114 ceph-mon[103029]: 11.0 scrub starts 2024-09-14T04:13:17.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:17 smithi114 ceph-mon[103029]: 11.3 scrub ok 2024-09-14T04:13:17.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:17 smithi114 ceph-mon[106969]: osdmap e244: 8 total, 8 up, 7 in 2024-09-14T04:13:17.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:17 smithi114 ceph-mon[106969]: 11.3 scrub starts 2024-09-14T04:13:17.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:17 smithi114 ceph-mon[106969]: 11.0 scrub starts 2024-09-14T04:13:17.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:17 smithi114 ceph-mon[106969]: 11.3 scrub ok 2024-09-14T04:13:17.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:17.228 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 61060 61044 112.19 71.875 0.00259306 0.00886914 2024-09-14T04:13:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:17.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:17.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:17.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:17.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:17.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:17 smithi160 ceph-mon[88407]: osdmap e244: 8 total, 8 up, 7 in 2024-09-14T04:13:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:17 smithi160 ceph-mon[88407]: 11.3 scrub starts 2024-09-14T04:13:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:17 smithi160 ceph-mon[88407]: 11.0 scrub starts 2024-09-14T04:13:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:17 smithi160 ceph-mon[88407]: 11.3 scrub ok 2024-09-14T04:13:17.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:17.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:17.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:17.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:17.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:17.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:17.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:17.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:17.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:17.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:17.773 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:13:17.773 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:13:17.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:17.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:17.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:17.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:18.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:18.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:18.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[106969]: 11.8 scrub starts 2024-09-14T04:13:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[106969]: 11.6 scrub starts 2024-09-14T04:13:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[106969]: pgmap v903: 99 pgs: 1 peering, 10 unknown, 88 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T04:13:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[106969]: 11.0 scrub ok 2024-09-14T04:13:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[106969]: 11.2 scrub starts 2024-09-14T04:13:18.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[106969]: 11.4 scrub starts 2024-09-14T04:13:18.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[106969]: 11.8 scrub ok 2024-09-14T04:13:18.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[106969]: 11.10 scrub starts 2024-09-14T04:13:18.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[106969]: 11.4 scrub ok 2024-09-14T04:13:18.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[103029]: 11.8 scrub starts 2024-09-14T04:13:18.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[103029]: 11.6 scrub starts 2024-09-14T04:13:18.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[103029]: pgmap v903: 99 pgs: 1 peering, 10 unknown, 88 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T04:13:18.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[103029]: 11.0 scrub ok 2024-09-14T04:13:18.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[103029]: 11.2 scrub starts 2024-09-14T04:13:18.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[103029]: 11.4 scrub starts 2024-09-14T04:13:18.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[103029]: 11.8 scrub ok 2024-09-14T04:13:18.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[103029]: 11.10 scrub starts 2024-09-14T04:13:18.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:18 smithi114 ceph-mon[103029]: 11.4 scrub ok 2024-09-14T04:13:18.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:18.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:18.228 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 62181 62165 110.986 70.0625 0.0025451 0.00898973 2024-09-14T04:13:18.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:18.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:18.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:18.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:18 smithi160 ceph-mon[88407]: 11.8 scrub starts 2024-09-14T04:13:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:18 smithi160 ceph-mon[88407]: 11.6 scrub starts 2024-09-14T04:13:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:18 smithi160 ceph-mon[88407]: pgmap v903: 99 pgs: 1 peering, 10 unknown, 88 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T04:13:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:18 smithi160 ceph-mon[88407]: 11.0 scrub ok 2024-09-14T04:13:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:18 smithi160 ceph-mon[88407]: 11.2 scrub starts 2024-09-14T04:13:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:18 smithi160 ceph-mon[88407]: 11.4 scrub starts 2024-09-14T04:13:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:18 smithi160 ceph-mon[88407]: 11.8 scrub ok 2024-09-14T04:13:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:18 smithi160 ceph-mon[88407]: 11.10 scrub starts 2024-09-14T04:13:18.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:18 smithi160 ceph-mon[88407]: 11.4 scrub ok 2024-09-14T04:13:18.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:18.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:18.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:18.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:18.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:18.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:18.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:18.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:18.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:18.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:18.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:18.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:18.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:18.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:18.925 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:18.949 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:18.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:18.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:19.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:19.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:19.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:19.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:19.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-14T04:13:19.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:19.200 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:13:19.217 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:13:19.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 63701 63685 110.54 95 0.010274 0.00903941 2024-09-14T04:13:19.273 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:19.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:19.358 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[106969]: 11.10 scrub ok 2024-09-14T04:13:19.359 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[106969]: 11.9 scrub starts 2024-09-14T04:13:19.359 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[106969]: 11.6 scrub ok 2024-09-14T04:13:19.359 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[106969]: 11.16 scrub starts 2024-09-14T04:13:19.359 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[106969]: 11.2 scrub ok 2024-09-14T04:13:19.359 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[106969]: 11.12 scrub starts 2024-09-14T04:13:19.359 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[106969]: 11.9 scrub ok 2024-09-14T04:13:19.359 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[103029]: 11.10 scrub ok 2024-09-14T04:13:19.359 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[103029]: 11.9 scrub starts 2024-09-14T04:13:19.359 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[103029]: 11.6 scrub ok 2024-09-14T04:13:19.359 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[103029]: 11.16 scrub starts 2024-09-14T04:13:19.360 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[103029]: 11.2 scrub ok 2024-09-14T04:13:19.360 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[103029]: 11.12 scrub starts 2024-09-14T04:13:19.360 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:19 smithi114 ceph-mon[103029]: 11.9 scrub ok 2024-09-14T04:13:19.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:19 smithi160 ceph-mon[88407]: 11.10 scrub ok 2024-09-14T04:13:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:19 smithi160 ceph-mon[88407]: 11.9 scrub starts 2024-09-14T04:13:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:19 smithi160 ceph-mon[88407]: 11.6 scrub ok 2024-09-14T04:13:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:19 smithi160 ceph-mon[88407]: 11.16 scrub starts 2024-09-14T04:13:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:19 smithi160 ceph-mon[88407]: 11.2 scrub ok 2024-09-14T04:13:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:19 smithi160 ceph-mon[88407]: 11.12 scrub starts 2024-09-14T04:13:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:19 smithi160 ceph-mon[88407]: 11.9 scrub ok 2024-09-14T04:13:19.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:19.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:19.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:19.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:19.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:19.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:19.689 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:19.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:19.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:19.842 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:13:19.843 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:13:19.843 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-14T04:13:19.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-14T04:13:19.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:19.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:19.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:19.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:19.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:20.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:20.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:20.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:20.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:20.229 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 65127 65111 109.959 89.125 0.0072075 0.0090896 2024-09-14T04:13:20.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:20.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:20.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:20.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:20 smithi114 ceph-mon[103029]: 11.12 scrub ok 2024-09-14T04:13:20.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:20 smithi114 ceph-mon[103029]: 11.16 scrub ok 2024-09-14T04:13:20.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:20 smithi114 ceph-mon[103029]: 11.18 scrub starts 2024-09-14T04:13:20.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:20 smithi114 ceph-mon[103029]: pgmap v904: 99 pgs: 1 peering, 8 unknown, 90 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 7.0 MiB/s wr, 112 op/s 2024-09-14T04:13:20.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:20 smithi114 ceph-mon[103029]: 11.18 scrub ok 2024-09-14T04:13:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:20 smithi114 ceph-mon[106969]: 11.12 scrub ok 2024-09-14T04:13:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:20 smithi114 ceph-mon[106969]: 11.16 scrub ok 2024-09-14T04:13:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:20 smithi114 ceph-mon[106969]: 11.18 scrub starts 2024-09-14T04:13:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:20 smithi114 ceph-mon[106969]: pgmap v904: 99 pgs: 1 peering, 8 unknown, 90 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 7.0 MiB/s wr, 112 op/s 2024-09-14T04:13:20.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:20 smithi114 ceph-mon[106969]: 11.18 scrub ok 2024-09-14T04:13:20.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:20.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:20.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:13:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:13:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:13:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:20 smithi160 ceph-mon[88407]: 11.12 scrub ok 2024-09-14T04:13:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:20 smithi160 ceph-mon[88407]: 11.16 scrub ok 2024-09-14T04:13:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:20 smithi160 ceph-mon[88407]: 11.18 scrub starts 2024-09-14T04:13:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:20 smithi160 ceph-mon[88407]: pgmap v904: 99 pgs: 1 peering, 8 unknown, 90 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 7.0 MiB/s wr, 112 op/s 2024-09-14T04:13:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:20 smithi160 ceph-mon[88407]: 11.18 scrub ok 2024-09-14T04:13:20.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:20.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:20.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:20.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:20.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:20.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:20.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:20.771 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:20.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:20.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:20.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:20.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:20.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:20.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:21.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:21.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:21.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:21.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:21.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:21.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:21.230 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 66606 66590 109.498 92.4375 0.0363905 0.00911344 2024-09-14T04:13:21.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:21.288 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:21.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:21.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[103029]: 11.7 scrub starts 2024-09-14T04:13:21.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[103029]: 11.19 scrub starts 2024-09-14T04:13:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[103029]: 11.7 scrub ok 2024-09-14T04:13:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[103029]: 11.19 scrub ok 2024-09-14T04:13:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[103029]: 11.1 scrub starts 2024-09-14T04:13:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-14T04:13:21.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[106969]: 11.7 scrub starts 2024-09-14T04:13:21.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[106969]: 11.19 scrub starts 2024-09-14T04:13:21.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[106969]: 11.7 scrub ok 2024-09-14T04:13:21.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[106969]: 11.19 scrub ok 2024-09-14T04:13:21.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[106969]: 11.1 scrub starts 2024-09-14T04:13:21.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-14T04:13:21.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:21 smithi160 ceph-mon[88407]: 11.7 scrub starts 2024-09-14T04:13:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:21 smithi160 ceph-mon[88407]: 11.19 scrub starts 2024-09-14T04:13:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:21 smithi160 ceph-mon[88407]: 11.7 scrub ok 2024-09-14T04:13:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:21 smithi160 ceph-mon[88407]: 11.19 scrub ok 2024-09-14T04:13:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:21 smithi160 ceph-mon[88407]: 11.1 scrub starts 2024-09-14T04:13:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:21 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-14T04:13:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:21.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:21.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:21.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:21.549 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:21.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:21.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:21.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:21.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:21.775 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:21.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:21.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:13:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:13:21] "GET /metrics HTTP/1.1" 200 36557 "" "Prometheus/2.51.0" 2024-09-14T04:13:21.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:21.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:21.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:22.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:22.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:22.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:22.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:22.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:22.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:22.230 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 67101 67085 107.483 30.9375 0.0020179 0.00907564 2024-09-14T04:13:22.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:22.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:22.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:22.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[103029]: pgmap v905: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 19 KiB/s, 0 objects/s recovering 2024-09-14T04:13:22.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[103029]: 11.1 scrub ok 2024-09-14T04:13:22.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-14T04:13:22.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[103029]: osdmap e245: 8 total, 8 up, 7 in 2024-09-14T04:13:22.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[103029]: 11.13 scrub starts 2024-09-14T04:13:22.316 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[103029]: 11.13 scrub ok 2024-09-14T04:13:22.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[106969]: pgmap v905: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 19 KiB/s, 0 objects/s recovering 2024-09-14T04:13:22.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[106969]: 11.1 scrub ok 2024-09-14T04:13:22.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-14T04:13:22.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[106969]: osdmap e245: 8 total, 8 up, 7 in 2024-09-14T04:13:22.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[106969]: 11.13 scrub starts 2024-09-14T04:13:22.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:22 smithi114 ceph-mon[106969]: 11.13 scrub ok 2024-09-14T04:13:22.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:22 smithi160 ceph-mon[88407]: pgmap v905: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 19 KiB/s, 0 objects/s recovering 2024-09-14T04:13:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:22 smithi160 ceph-mon[88407]: 11.1 scrub ok 2024-09-14T04:13:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-14T04:13:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:22 smithi160 ceph-mon[88407]: osdmap e245: 8 total, 8 up, 7 in 2024-09-14T04:13:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:22 smithi160 ceph-mon[88407]: 11.13 scrub starts 2024-09-14T04:13:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:22 smithi160 ceph-mon[88407]: 11.13 scrub ok 2024-09-14T04:13:22.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:22.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:22.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:22.546 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:22.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:22.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:22.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:22.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:22.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:22.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:22.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:22.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:22.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:22.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:22.918 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:22.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:23.020 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:23.042 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:23.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:23.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:23.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:23.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:23.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:13:23.230864+0000 min lat: 0.00112332 max lat: 2.09936 avg lat: 0.00952558 2024-09-14T04:13:23.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:13:23.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 67173 67157 104.907 4.5 0.0133285 0.00952558 2024-09-14T04:13:23.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:23.293 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:23.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:23.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:23.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[106969]: 11.5 scrub starts 2024-09-14T04:13:23.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[106969]: osdmap e246: 8 total, 8 up, 7 in 2024-09-14T04:13:23.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[106969]: 11.5 scrub ok 2024-09-14T04:13:23.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[106969]: 11.11 scrub starts 2024-09-14T04:13:23.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[106969]: 11.11 scrub ok 2024-09-14T04:13:23.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:23.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-14T04:13:23.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[103029]: 11.5 scrub starts 2024-09-14T04:13:23.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[103029]: osdmap e246: 8 total, 8 up, 7 in 2024-09-14T04:13:23.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[103029]: 11.5 scrub ok 2024-09-14T04:13:23.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[103029]: 11.11 scrub starts 2024-09-14T04:13:23.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[103029]: 11.11 scrub ok 2024-09-14T04:13:23.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:23.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-14T04:13:23.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:23 smithi160 ceph-mon[88407]: 11.5 scrub starts 2024-09-14T04:13:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:23 smithi160 ceph-mon[88407]: osdmap e246: 8 total, 8 up, 7 in 2024-09-14T04:13:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:23 smithi160 ceph-mon[88407]: 11.5 scrub ok 2024-09-14T04:13:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:23 smithi160 ceph-mon[88407]: 11.11 scrub starts 2024-09-14T04:13:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:23 smithi160 ceph-mon[88407]: 11.11 scrub ok 2024-09-14T04:13:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-14T04:13:23.424 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:23.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:23.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:23.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:23.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:23.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:23.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:23.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:23.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:23.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:23.700 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:13:23.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:23.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:23.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:23.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:23.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:23.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:24.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:24.043 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:24.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:24.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:24.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:24.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:24.220 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:24 smithi114 ceph-mon[106969]: pgmap v908: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 19 KiB/s, 0 objects/s recovering 2024-09-14T04:13:24.220 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:24 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-14T04:13:24.221 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:24 smithi114 ceph-mon[106969]: osdmap e247: 8 total, 8 up, 7 in 2024-09-14T04:13:24.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:24 smithi114 ceph-mon[103029]: pgmap v908: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 19 KiB/s, 0 objects/s recovering 2024-09-14T04:13:24.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:24 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-14T04:13:24.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:24 smithi114 ceph-mon[103029]: osdmap e247: 8 total, 8 up, 7 in 2024-09-14T04:13:24.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 67471 67455 102.802 18.625 0.00280557 0.00951287 2024-09-14T04:13:24.270 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:24.293 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:24.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:24.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:24 smithi160 ceph-mon[88407]: pgmap v908: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 19 KiB/s, 0 objects/s recovering 2024-09-14T04:13:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:24 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-14T04:13:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:24 smithi160 ceph-mon[88407]: osdmap e247: 8 total, 8 up, 7 in 2024-09-14T04:13:24.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:24.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:24.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:24.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:24.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:24.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:24.661 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:24.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:24.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:24.785 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:24.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:24.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:24.908 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:24.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:25.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:25.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:25.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:25.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:25.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:25.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:25.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 67529 67513 100.441 3.625 1.77602 0.00992467 2024-09-14T04:13:25.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:25.308 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:25.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:25.334 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:13:25.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:25 smithi114 ceph-mon[103029]: osdmap e248: 8 total, 8 up, 7 in 2024-09-14T04:13:25.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:25 smithi114 ceph-mon[106969]: osdmap e248: 8 total, 8 up, 7 in 2024-09-14T04:13:25.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:25 smithi160 ceph-mon[88407]: osdmap e248: 8 total, 8 up, 7 in 2024-09-14T04:13:25.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:25.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:25.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:25.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:25.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:25.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:25.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:25.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:25.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:25.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:25.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:25.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:25.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:25.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:25.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:25.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:25.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:26.067 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:13:26.099 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:26.125 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:26.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:26.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:26.233 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 68843 68827 100.015 82.125 0.00519624 0.00999334 2024-09-14T04:13:26.269 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:26.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:26.337 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:26 smithi114 ceph-mon[106969]: pgmap v911: 99 pgs: 1 remapped+peering, 98 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail 2024-09-14T04:13:26.337 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:26 smithi114 ceph-mon[106969]: osdmap e249: 8 total, 8 up, 7 in 2024-09-14T04:13:26.338 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:26 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2161071788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:13:26.338 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:26 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:13:26.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:26 smithi114 ceph-mon[103029]: pgmap v911: 99 pgs: 1 remapped+peering, 98 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail 2024-09-14T04:13:26.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:26 smithi114 ceph-mon[103029]: osdmap e249: 8 total, 8 up, 7 in 2024-09-14T04:13:26.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:26 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2161071788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:13:26.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:26 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:13:26.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:26 smithi160 ceph-mon[88407]: pgmap v911: 99 pgs: 1 remapped+peering, 98 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail 2024-09-14T04:13:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:26 smithi160 ceph-mon[88407]: osdmap e249: 8 total, 8 up, 7 in 2024-09-14T04:13:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:26 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2161071788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:13:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:26 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:13:26.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:26.409 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:26.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:26.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:26.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:26.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:26.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:26.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:26.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:26.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:26.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:26.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:26.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:26.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:26.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:27.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:27.093 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:27.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:27.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:27.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:27.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:27.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:27.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 70100 70084 99.5276 78.5625 0.00354356 0.0100428 2024-09-14T04:13:27.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:27.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:27.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:27.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:27 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:13:27.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:27 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:13:27.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:27 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:13:27.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:27 smithi114 ceph-mon[106969]: osdmap e250: 8 total, 8 up, 7 in 2024-09-14T04:13:27.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:27 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:13:27.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:27 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:13:27.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:27 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:13:27.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:27 smithi114 ceph-mon[103029]: osdmap e250: 8 total, 8 up, 7 in 2024-09-14T04:13:27.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:27 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:13:27.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:27 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:13:27.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:27 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:13:27.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:27 smithi160 ceph-mon[88407]: osdmap e250: 8 total, 8 up, 7 in 2024-09-14T04:13:27.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:27.495 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:27.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:27.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:27.623 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:27.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:27.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:27.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:27.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:27.847 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:27.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:27.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:27.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:27.996 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:27.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:28.099 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:28.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:28.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:28.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:28.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:28.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 71001 70985 98.5673 56.3125 0.00759146 0.0101391 2024-09-14T04:13:28.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:28.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:28.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[106969]: 11.17 scrub starts 2024-09-14T04:13:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[106969]: pgmap v914: 99 pgs: 1 remapped+peering, 98 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail 2024-09-14T04:13:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[106969]: 11.15 scrub starts 2024-09-14T04:13:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[106969]: 11.17 scrub ok 2024-09-14T04:13:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2770534136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:13:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[103029]: 11.17 scrub starts 2024-09-14T04:13:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[103029]: pgmap v914: 99 pgs: 1 remapped+peering, 98 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail 2024-09-14T04:13:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[103029]: 11.15 scrub starts 2024-09-14T04:13:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[103029]: 11.17 scrub ok 2024-09-14T04:13:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2770534136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:13:28.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:28 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:28.389 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:28.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:28 smithi160 ceph-mon[88407]: 11.17 scrub starts 2024-09-14T04:13:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:28 smithi160 ceph-mon[88407]: pgmap v914: 99 pgs: 1 remapped+peering, 98 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail 2024-09-14T04:13:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:28 smithi160 ceph-mon[88407]: 11.15 scrub starts 2024-09-14T04:13:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:28 smithi160 ceph-mon[88407]: 11.17 scrub ok 2024-09-14T04:13:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:28 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2770534136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:13:28.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:28 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:28.488 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:13:28.488 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:13:28.489 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:13:28.489 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:13:28.489 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:13:28.489 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:13:28.489 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:13:28.489 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:13:28.489 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:13:28.489 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:13:28.489 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:13:28.490 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:13:28.490 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:13:28.490 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:13:28.490 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:13:28.490 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:13:28.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:28.529 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:28.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:28.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:28.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:28.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:28.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:28.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:28.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:28.926 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:28.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:29.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:29.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:29.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:29.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:29.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:29.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:29.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 72286 72270 98.1704 80.3125 0.0477514 0.0101804 2024-09-14T04:13:29.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:29.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:29.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:29.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:29.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:29 smithi114 ceph-mon[103029]: 11.14 scrub starts 2024-09-14T04:13:29.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:29 smithi114 ceph-mon[106969]: 11.14 scrub starts 2024-09-14T04:13:29.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:29 smithi160 ceph-mon[88407]: 11.14 scrub starts 2024-09-14T04:13:29.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:29.464 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:29.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:29.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:29.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:29.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:29.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:29.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:29.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:29.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:29.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:29.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:29.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:30.000 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:30.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:30.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:30.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:30.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:30.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:30.231 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 73561 73545 97.7769 79.6875 0.0023036 0.0102222 2024-09-14T04:13:30.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:30.287 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:30.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:30.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:30.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:30 smithi114 ceph-mon[106969]: pgmap v915: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 2101/205812 objects degraded (1.021%); 7513/205812 objects misplaced (3.650%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:13:30.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:30 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 2101/205812 objects degraded (1.021%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:13:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:30 smithi114 ceph-mon[103029]: pgmap v915: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 2101/205812 objects degraded (1.021%); 7513/205812 objects misplaced (3.650%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:13:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:30 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 2101/205812 objects degraded (1.021%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:13:30.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:30.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:13:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:13:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:13:30.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:30 smithi160 ceph-mon[88407]: pgmap v915: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 2101/205812 objects degraded (1.021%); 7513/205812 objects misplaced (3.650%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:13:30.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:30 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 2101/205812 objects degraded (1.021%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:13:30.424 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:30.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:30.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:30.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:30.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:30.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:30.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:30.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:30.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:30.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:30.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:30.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:30.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:30.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:30.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:31.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:31.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:31.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:31.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:31.232 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:31.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:31.234 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 74793 74777 97.3377 77 0.00634003 0.0102682 2024-09-14T04:13:31.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:31.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:31.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:31.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:31.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:31.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:13:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:13:31] "GET /metrics HTTP/1.1" 200 36565 "" "Prometheus/2.51.0" 2024-09-14T04:13:31.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:31.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:31.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:31.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:31.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:31.806 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:31.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:31.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:31.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:31.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:32.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:32.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:32.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:32.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:32.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:32.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:32.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 75912 75896 96.7785 69.9375 0.0315478 0.0103266 2024-09-14T04:13:32.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:32.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:32.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:32.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:32 smithi114 ceph-mon[106969]: pgmap v916: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 2101/214920 objects degraded (0.978%); 7513/214920 objects misplaced (3.496%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:13:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:32 smithi114 ceph-mon[103029]: pgmap v916: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 2101/214920 objects degraded (0.978%); 7513/214920 objects misplaced (3.496%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:13:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:32 smithi160 ceph-mon[88407]: pgmap v916: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 2101/214920 objects degraded (0.978%); 7513/214920 objects misplaced (3.496%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:13:32.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:32.456 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:32.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:32.558 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:32.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:32.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:32.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:32.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:32.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:32.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:32.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:32.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:32.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:32.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:32.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:33.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:33.115 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:33.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:33.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:33.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:33.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 76842 76826 96.0055 58.125 0.00609615 0.010407 2024-09-14T04:13:33.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:33.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:33.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:33.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:33.367 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:33.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:33.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:33.501 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:33.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:33.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:33.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:33.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:33.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:33.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:33.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:33.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:33.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:33.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:33.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:33.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:34.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:34.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:34.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:34.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:34.140 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:34.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:34.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 77875 77859 95.389 64.5625 0.0107574 0.0104783 2024-09-14T04:13:34.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:34.275 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:34.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:34.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:34 smithi114 ceph-mon[106969]: pgmap v917: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 24 MiB/s wr, 384 op/s; 2101/217212 objects degraded (0.967%); 7513/217212 objects misplaced (3.459%); 1.9 MiB/s, 29 objects/s recovering 2024-09-14T04:13:34.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:34 smithi114 ceph-mon[103029]: pgmap v917: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 24 MiB/s wr, 384 op/s; 2101/217212 objects degraded (0.967%); 7513/217212 objects misplaced (3.459%); 1.9 MiB/s, 29 objects/s recovering 2024-09-14T04:13:34.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:34 smithi160 ceph-mon[88407]: pgmap v917: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 24 MiB/s wr, 384 op/s; 2101/217212 objects degraded (0.967%); 7513/217212 objects misplaced (3.459%); 1.9 MiB/s, 29 objects/s recovering 2024-09-14T04:13:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:34.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:34.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:34.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:34.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:34.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:34.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:34.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:34.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:34.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:34.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:34.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:34.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:34.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:34.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:35.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:35.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:35.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:35.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:35.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:35.196 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:35.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:35.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 79085 79069 95.0088 75.625 0.00616608 0.0105192 2024-09-14T04:13:35.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:35.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:35.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:35.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:35 smithi114 ceph-mon[103029]: 11.15 scrub starts 2024-09-14T04:13:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:35 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2101/217212 objects degraded (0.967%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:13:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:35 smithi114 ceph-mon[106969]: 11.15 scrub starts 2024-09-14T04:13:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:35 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2101/217212 objects degraded (0.967%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:13:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:35 smithi160 ceph-mon[88407]: 11.15 scrub starts 2024-09-14T04:13:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:35 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2101/217212 objects degraded (0.967%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:13:35.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:35.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:35.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:35.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:35.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:35.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:35.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:35.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:35.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:35.720 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:35 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:13:35.545+0000 7f3162f23640 -1 mon.a@0(leader).osd e250 definitely_dead 0 2024-09-14T04:13:35.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:35.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:35.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:35.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:35.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:35.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:36.073 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:13:36.073 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:80069 11.14 e250/243)", 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:36.069278+0000", 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001358078, 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0013796749999999999, 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:13:36.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069278+0000", 2024-09-14T04:13:36.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:13:36.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069278+0000", 2024-09-14T04:13:36.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069279+0000", 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8100000000000002e-07 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069313+0000", 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3871999999999997e-05 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:13:36.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069324+0000", 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0784999999999999e-05 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069330+0000", 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.55e-06 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:80065 11.6 e250/243 11:659a0447:::benchmark_data_smithi114_223345_object80064:head v 250'4364, mlcod=250'4362)", 2024-09-14T04:13:36.077 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:36.069350+0000", 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00128662, 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0013519070000000001, 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069350+0000", 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:13:36.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069350+0000", 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069350+0000", 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9100000000000004e-07 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069371+0000", 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1033999999999999e-05 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069379+0000", 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7430000000000002e-06 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069384+0000", 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1000000000000003e-06 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069434+0000", 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0092000000000001e-05 2024-09-14T04:13:36.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.069452+0000", 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8253999999999998e-05 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80054 11.d 11:bef3d0a9:::benchmark_data_smithi114_223345_object80053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:36.057299+0000", 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.013337526000000001, 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013457142, 2024-09-14T04:13:36.081 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80054 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.057299+0000", 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.057299+0000", 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.057301+0000", 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8610000000000001e-06 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.057338+0000", 2024-09-14T04:13:36.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7759000000000002e-05 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.057349+0000", 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0501e-05 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.057356+0000", 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9990000000000002e-06 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:13:36.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.061474+0000", 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041183299999999999 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.061606+0000", 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000132077 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.061823+0000", 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021717899999999999 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.085 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80057 11.d 11:ba4b8e6c:::benchmark_data_smithi114_223345_object80056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:36.060949+0000", 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0096872469999999995, 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0098759169999999997, 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80057 2024-09-14T04:13:36.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.060949+0000", 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.060949+0000", 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:13:36.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.060951+0000", 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0640000000000001e-06 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.060982+0000", 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0981e-05 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.060994+0000", 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2319e-05 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.061002+0000", 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2969999999999998e-06 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.064765+0000", 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037628869999999999 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:13:36.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.064889+0000", 2024-09-14T04:13:36.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000124086 2024-09-14T04:13:36.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,6", 2024-09-14T04:13:36.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.065447+0000", 2024-09-14T04:13:36.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00055845899999999997 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.36273.0:80054 11.d e250/235)", 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:36.066517+0000", 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0041190949999999997, 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044077049999999996, 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:13:36.091 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066517+0000", 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066517+0000", 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066518+0000", 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0839999999999999e-06 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066523+0000", 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0590000000000002e-06 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066536+0000", 2024-09-14T04:13:36.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3178e-05 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066544+0000", 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3429999999999998e-06 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:80041 11.c e250/235 11:33e7559d:::benchmark_data_smithi114_223345_object80040:head v 250'4969, mlcod=250'4966)", 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:36.066722+0000", 2024-09-14T04:13:36.094 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0039144130000000003, 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042466539999999999, 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066722+0000", 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066722+0000", 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:13:36.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066722+0000", 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.75e-07 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066748+0000", 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5333e-05 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:13:36.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066760+0000", 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2629999999999999e-05 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.066768+0000", 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1559999999999998e-06 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.067919+0000", 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011517960000000001 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:13:36.097 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:13:36.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.067938+0000", 2024-09-14T04:13:36.098 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8533999999999999e-05 2024-09-14T04:13:36.098 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.098 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:13:36.098 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.098 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:13:36.098 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:13:36.098 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 6 2024-09-14T04:13:36.098 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:13:36.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:36.140 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:36.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:36.230 INFO:teuthology.orchestra.run.smithi114.stderr:marked in osd.4. 2024-09-14T04:13:36.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 80201 80185 94.5322 69.75 0.00235774 0.0105641 2024-09-14T04:13:36.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:36.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:36.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:36.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:36 smithi114 ceph-mon[103029]: pgmap v918: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s; 2101/232095 objects degraded (0.905%); 7108/232095 objects misplaced (3.063%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:13:36.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:36 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/413140780' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-14T04:13:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:36 smithi114 ceph-mon[106969]: pgmap v918: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s; 2101/232095 objects degraded (0.905%); 7108/232095 objects misplaced (3.063%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:13:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:36 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/413140780' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-14T04:13:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:36.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:36.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:36 smithi160 ceph-mon[88407]: pgmap v918: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s; 2101/232095 objects degraded (0.905%); 7108/232095 objects misplaced (3.063%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:13:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:36 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/413140780' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-14T04:13:36.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:36.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:36.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:36.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:36.635 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:36.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:36.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:36.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:36.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:36.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:13:36.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:36.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:36.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:36.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:36.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:37.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:37.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:37.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:37.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:37.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:37.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:37.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 80208 80192 92.79 0.4375 0.00324109 0.0105637 2024-09-14T04:13:37.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:37.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:37.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:37.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:37.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:37.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:37 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/413140780' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-14T04:13:37.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:37 smithi160 ceph-mon[88407]: osdmap e251: 8 total, 8 up, 8 in 2024-09-14T04:13:37.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:13:37.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:37 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/413140780' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-14T04:13:37.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:37 smithi114 ceph-mon[106969]: osdmap e251: 8 total, 8 up, 8 in 2024-09-14T04:13:37.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:37 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/413140780' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-14T04:13:37.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:37 smithi114 ceph-mon[103029]: osdmap e251: 8 total, 8 up, 8 in 2024-09-14T04:13:37.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:37.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:37.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:37.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:37.625 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:37.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:37.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:37.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:37.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:37.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:37.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:37.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:37.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:37.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:37.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:37.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:38.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:38.120 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:38.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:38.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:38.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 80208 80192 91.1032 0 - 0.0105637 2024-09-14T04:13:38.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:38.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:38.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:38.392 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:38.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:38.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:38.527 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:38.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:38.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:38.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:38 smithi114 ceph-mon[103029]: pgmap v920: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 53 MiB/s wr, 840 op/s; 2101/234066 objects degraded (0.898%); 7108/234066 objects misplaced (3.037%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:13:38.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:38 smithi114 ceph-mon[103029]: osdmap e252: 8 total, 8 up, 8 in 2024-09-14T04:13:38.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:38.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:38.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:38.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:38 smithi114 ceph-mon[106969]: pgmap v920: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 53 MiB/s wr, 840 op/s; 2101/234066 objects degraded (0.898%); 7108/234066 objects misplaced (3.037%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:13:38.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:38 smithi114 ceph-mon[106969]: osdmap e252: 8 total, 8 up, 8 in 2024-09-14T04:13:38.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:38.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:38.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:38.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:38 smithi160 ceph-mon[88407]: pgmap v920: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 97 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 53 MiB/s wr, 840 op/s; 2101/234066 objects degraded (0.898%); 7108/234066 objects misplaced (3.037%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:13:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:38 smithi160 ceph-mon[88407]: osdmap e252: 8 total, 8 up, 8 in 2024-09-14T04:13:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:13:38.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:38.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:38.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:38.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:38.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:38.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:38.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:38.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:38.939 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:38.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:39.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:39.088 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:39.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:39.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:39.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:39.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:39.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 81111 81095 90.4842 28.2188 0.0215377 0.0110454 2024-09-14T04:13:39.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:39.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:39.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:39.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:39.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:39.489 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:39.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:39.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:39.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:39 smithi114 ceph-mon[103029]: osdmap e253: 8 total, 8 up, 8 in 2024-09-14T04:13:39.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:39 smithi114 ceph-mon[106969]: osdmap e253: 8 total, 8 up, 8 in 2024-09-14T04:13:39.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:39.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:39 smithi160 ceph-mon[88407]: osdmap e253: 8 total, 8 up, 8 in 2024-09-14T04:13:39.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:39.764 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:39.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:39.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:39.905 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:39.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:40.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:40.040 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:40.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:40.142 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:40.178 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:40.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:40.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:40.235 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 82120 82104 90.0031 63.0625 0.00983728 0.0111038 2024-09-14T04:13:40.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:40.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:40.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:40.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:40.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:40.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:40.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:40.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:40.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:40.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:40 smithi114 ceph-mon[103029]: pgmap v923: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 13 peering, 82 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 13213/237129 objects misplaced (5.572%); 5.4 MiB/s, 85 objects/s recovering 2024-09-14T04:13:40.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:40 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2101/234066 objects degraded (0.898%), 1 pg degraded) 2024-09-14T04:13:40.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:40 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:13:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:40 smithi114 ceph-mon[106969]: pgmap v923: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 13 peering, 82 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 13213/237129 objects misplaced (5.572%); 5.4 MiB/s, 85 objects/s recovering 2024-09-14T04:13:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:40 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2101/234066 objects degraded (0.898%), 1 pg degraded) 2024-09-14T04:13:40.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:40 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:13:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:13:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:13:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:13:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:40 smithi160 ceph-mon[88407]: pgmap v923: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 13 peering, 82 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 13213/237129 objects misplaced (5.572%); 5.4 MiB/s, 85 objects/s recovering 2024-09-14T04:13:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:40 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2101/234066 objects degraded (0.898%), 1 pg degraded) 2024-09-14T04:13:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:40 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:13:40.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:40.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:40.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:40.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:40.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:40.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:40.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:40.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:40.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:41.007 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:41.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:41.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:41.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:41.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:41.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:41.236 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 83196 83180 89.6108 67.25 0.0231292 0.0111538 2024-09-14T04:13:41.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:41.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:41.275 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:41.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:41.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:41.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:41.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:41.520 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:41.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:41.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:41.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:41 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 2530/242844 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:13:41.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:41 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 2530/242844 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:13:41.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:13:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:13:41] "GET /metrics HTTP/1.1" 200 36591 "" "Prometheus/2.51.0" 2024-09-14T04:13:41.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:41 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 2530/242844 objects degraded (1.042%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:13:41.693 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:41.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:41.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:41.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:41.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:41.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:41.959 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:41.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:42.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:42.095 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:42.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:42.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:42.234 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:42.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:42.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:42.236 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 84178 84162 89.1322 61.375 0.0132187 0.0112138 2024-09-14T04:13:42.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:42.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:42.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:42.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:42.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:42 smithi114 ceph-mon[103029]: pgmap v924: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 13 peering, 80 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 37 MiB/s wr, 594 op/s; 2530/242844 objects degraded (1.042%); 15654/242844 objects misplaced (6.446%); 4.9 MiB/s, 1 keys/s, 80 objects/s recovering 2024-09-14T04:13:42.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-14T04:13:42.508 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:42 smithi114 ceph-mon[106969]: pgmap v924: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 13 peering, 80 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 37 MiB/s wr, 594 op/s; 2530/242844 objects degraded (1.042%); 15654/242844 objects misplaced (6.446%); 4.9 MiB/s, 1 keys/s, 80 objects/s recovering 2024-09-14T04:13:42.528 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:42.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:42.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:42 smithi160 ceph-mon[88407]: pgmap v924: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 13 peering, 80 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 37 MiB/s wr, 594 op/s; 2530/242844 objects degraded (1.042%); 15654/242844 objects misplaced (6.446%); 4.9 MiB/s, 1 keys/s, 80 objects/s recovering 2024-09-14T04:13:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:42.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:42.773 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:42.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:42.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:42.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:42.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:42.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:43.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:43.072 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:43.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:43.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:43.120 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-14T04:13:43.153 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:13:43.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:43.183 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:13:43.183 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:13:43.201 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:43.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:43.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:43.240 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:13:43.239974+0000 min lat: 0.00112332 max lat: 2.20623 avg lat: 0.0112793 2024-09-14T04:13:43.240 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:13:43.240 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 85097 85081 88.5978 57.4375 0.00789816 0.0112793 2024-09-14T04:13:43.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:43.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:43.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:43.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:43.467 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:43.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:43.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:43.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:43.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:43.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:43.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:43.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:43.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:43.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:43.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:43.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:43.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:43.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:44.019 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:44.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:44.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:44.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:44.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:44.240 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 86161 86145 88.2355 66.5 0.0113362 0.0113279 2024-09-14T04:13:44.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:44.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:44.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:44.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:44.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:44.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:44.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:44.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:44.559 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:44.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:44.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:44.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:44 smithi114 ceph-mon[106969]: pgmap v925: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 92 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 31 MiB/s wr, 489 op/s; 7556/243933 objects degraded (3.098%); 15654/243933 objects misplaced (6.417%); 4.4 MiB/s, 1 keys/s, 72 objects/s recovering 2024-09-14T04:13:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:44 smithi114 ceph-mon[103029]: pgmap v925: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 92 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 31 MiB/s wr, 489 op/s; 7556/243933 objects degraded (3.098%); 15654/243933 objects misplaced (6.417%); 4.4 MiB/s, 1 keys/s, 72 objects/s recovering 2024-09-14T04:13:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:44 smithi160 ceph-mon[88407]: pgmap v925: 99 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 92 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 31 MiB/s wr, 489 op/s; 7556/243933 objects degraded (3.098%); 15654/243933 objects misplaced (6.417%); 4.4 MiB/s, 1 keys/s, 72 objects/s recovering 2024-09-14T04:13:44.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:44.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:44.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:44.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:44.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:44.860 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:44.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:44.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:44.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:44.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:45.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:45.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:45.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:45.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:45.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:45.242 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 87220 87204 87.8787 66.1875 0.0140825 0.0113737 2024-09-14T04:13:45.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:45.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:45.369 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:45.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:45.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:45.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:45.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:45.536 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:45.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:45.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:45.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:45.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:45.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:45.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:45.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:45.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:45.985 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:45.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:45.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:46.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:46.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:46.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:46.219 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:46.241 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 88215 88199 87.4709 62.1875 0.0296874 0.0114267 2024-09-14T04:13:46.268 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:46.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:46.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:46.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:46.403 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:46.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:46.486 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:46 smithi114 ceph-mon[106969]: pgmap v926: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 907 op/s; 12531/255849 objects degraded (4.898%); 14163/255849 objects misplaced (5.536%); 11 MiB/s, 1 keys/s, 171 objects/s recovering 2024-09-14T04:13:46.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:46 smithi114 ceph-mon[103029]: pgmap v926: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 907 op/s; 12531/255849 objects degraded (4.898%); 14163/255849 objects misplaced (5.536%); 11 MiB/s, 1 keys/s, 171 objects/s recovering 2024-09-14T04:13:46.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:46.517 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:13:46.517 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:13:46.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:46.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:46.568 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:13:46.568 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:13:46.601 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:13:46.602 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:13:46.635 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:13:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:13:46.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:46.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:46 smithi160 ceph-mon[88407]: pgmap v926: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 907 op/s; 12531/255849 objects degraded (4.898%); 14163/255849 objects misplaced (5.536%); 11 MiB/s, 1 keys/s, 171 objects/s recovering 2024-09-14T04:13:46.667 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:13:46.668 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:13:46.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:46.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:46.720 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:13:46.721 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:13:46.749 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:13:46.749 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:13:46.794 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:13:46.795 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:13:46.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:46.850 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:46.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:46.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:46.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:46.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:47.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:47.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:47.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:47.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:47.241 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 89149 89133 87.0163 58.375 0.0101579 0.0114861 2024-09-14T04:13:47.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:47.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:47.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:47.390 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:13:47.390 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:13:47.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:47.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:47.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:47.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:47.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:47.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:47 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/185226230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:13:47.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/185226230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:13:47.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:47 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/185226230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:13:47.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:47.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:47.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:47.806 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:47.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:47.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:47.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:47.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:47.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:48.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:48.078 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:48.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:48.121 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:13:48.121 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:13:48.121 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-14T04:13:48.122 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:13:48.122 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:13:48.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-14T04:13:48.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:48.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:48.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:48.241 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 90107 90091 86.5987 59.875 0.0129568 0.0115392 2024-09-14T04:13:48.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:48.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:48.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:48.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:48.462 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:48.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:48.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:48.585 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:48.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:48.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:48 smithi114 ceph-mon[103029]: pgmap v927: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 51 MiB/s wr, 813 op/s; 12531/257736 objects degraded (4.862%); 14163/257736 objects misplaced (5.495%); 8.7 MiB/s, 1 keys/s, 141 objects/s recovering 2024-09-14T04:13:48.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:48 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 12531/257736 objects degraded (4.862%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:13:48.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:48 smithi114 ceph-mon[106969]: pgmap v927: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 51 MiB/s wr, 813 op/s; 12531/257736 objects degraded (4.862%); 14163/257736 objects misplaced (5.495%); 8.7 MiB/s, 1 keys/s, 141 objects/s recovering 2024-09-14T04:13:48.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:48 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 12531/257736 objects degraded (4.862%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:13:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:48 smithi160 ceph-mon[88407]: pgmap v927: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 51 MiB/s wr, 813 op/s; 12531/257736 objects degraded (4.862%); 14163/257736 objects misplaced (5.495%); 8.7 MiB/s, 1 keys/s, 141 objects/s recovering 2024-09-14T04:13:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:48 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 12531/257736 objects degraded (4.862%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:13:48.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:48.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:48.724 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:48.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:48.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:48.865 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:48.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:48.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:49.005 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:49.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:49.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:49.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:49.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:49.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:49.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:49.241 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 90959 90943 86.0935 53.25 0.0153895 0.0116096 2024-09-14T04:13:49.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:49.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:49.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:49.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:49.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:49.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:49.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:49.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:49.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:49.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:49.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:49.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:49.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:49.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:49.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:49.921 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:49.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:50.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:50.060 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:50.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:50.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:50.204 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:50.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:50.241 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 91707 91691 85.5063 46.75 0.0201717 0.0116899 2024-09-14T04:13:50.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:50.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:50.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:50.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:50.462 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:50.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:50.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:50.602 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:50.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:50.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:50 smithi114 ceph-mon[106969]: pgmap v928: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 872 op/s; 12531/263403 objects degraded (4.757%); 13560/263403 objects misplaced (5.148%); 7.5 MiB/s, 0 keys/s, 122 objects/s recovering 2024-09-14T04:13:50.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:50 smithi114 ceph-mon[103029]: pgmap v928: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 872 op/s; 12531/263403 objects degraded (4.757%); 13560/263403 objects misplaced (5.148%); 7.5 MiB/s, 0 keys/s, 122 objects/s recovering 2024-09-14T04:13:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:50 smithi160 ceph-mon[88407]: pgmap v928: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 54 MiB/s wr, 872 op/s; 12531/263403 objects degraded (4.757%); 13560/263403 objects misplaced (5.148%); 7.5 MiB/s, 0 keys/s, 122 objects/s recovering 2024-09-14T04:13:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:13:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:13:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:13:50.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:50.740 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:50.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:50.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:50.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:50.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:50.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:50.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:51.011 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:51.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:51.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:51.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:51.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:51.242 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 92680 92664 85.1431 60.8125 0.0104138 0.0117398 2024-09-14T04:13:51.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:51.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:51.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:51.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:51.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:51.426 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:51.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:51.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:51.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:51.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:51.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:51.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:51.706 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:51.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:51.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:51.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:51.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:51.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:13:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:13:51] "GET /metrics HTTP/1.1" 200 36591 "" "Prometheus/2.51.0" 2024-09-14T04:13:51.937 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:51.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:51.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:52.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:52.099 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:52.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:52.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:52.240 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:52.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:52.241 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 93611 93595 84.7525 58.1875 0.0164066 0.0117937 2024-09-14T04:13:52.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:52.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:52.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:52.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:52.507 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:52.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:52.609 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:52.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:52 smithi114 ceph-mon[106969]: pgmap v929: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s; 12531/270672 objects degraded (4.630%); 12542/270672 objects misplaced (4.634%); 10 MiB/s, 0 keys/s, 167 objects/s recovering 2024-09-14T04:13:52.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:52 smithi114 ceph-mon[103029]: pgmap v929: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s; 12531/270672 objects degraded (4.630%); 12542/270672 objects misplaced (4.634%); 10 MiB/s, 0 keys/s, 167 objects/s recovering 2024-09-14T04:13:52.638 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:52.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:52.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:52.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:52 smithi160 ceph-mon[88407]: pgmap v929: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s; 12531/270672 objects degraded (4.630%); 12542/270672 objects misplaced (4.634%); 10 MiB/s, 0 keys/s, 167 objects/s recovering 2024-09-14T04:13:52.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:52.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:52.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:52.884 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:52.930 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:52.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:53.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:53.060 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:53.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:53.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:53.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:53.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:53.242 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 94511 94495 84.3453 56.25 0.0131841 0.0118502 2024-09-14T04:13:53.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:53.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:53.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:53.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:53.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:53.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:53.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:53.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:53.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:53.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:53.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:13:53.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:53.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:53.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:53.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:53.867 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:53.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:53.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:54.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:54.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:54.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:54.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:54.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:54.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:54.242 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 95399 95383 83.9391 55.5 0.00295213 0.011903 2024-09-14T04:13:54.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:54.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:54.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:54.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:54.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:54.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:54.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:54.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:54.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:54.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:54.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:54 smithi114 ceph-mon[106969]: pgmap v930: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 51 MiB/s wr, 820 op/s; 12531/272349 objects degraded (4.601%); 12542/272349 objects misplaced (4.605%); 9.9 MiB/s, 158 objects/s recovering 2024-09-14T04:13:54.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:54 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 12531/272349 objects degraded (4.601%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:13:54.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:54 smithi114 ceph-mon[103029]: pgmap v930: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 51 MiB/s wr, 820 op/s; 12531/272349 objects degraded (4.601%); 12542/272349 objects misplaced (4.605%); 9.9 MiB/s, 158 objects/s recovering 2024-09-14T04:13:54.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:54 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 12531/272349 objects degraded (4.601%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:13:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:54 smithi160 ceph-mon[88407]: pgmap v930: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 51 MiB/s wr, 820 op/s; 12531/272349 objects degraded (4.601%); 12542/272349 objects misplaced (4.605%); 9.9 MiB/s, 158 objects/s recovering 2024-09-14T04:13:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:54 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 12531/272349 objects degraded (4.601%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:13:54.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:54.696 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:54.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:54.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:54.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:54.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:54.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:54.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:54.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:55.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:55.123 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:55.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:55.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:55.242 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 96343 96327 83.5924 59 0.0317653 0.0119576 2024-09-14T04:13:55.262 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:55.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:55.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:55.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:55.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:55.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:55.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:55.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:55.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:55.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:55.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:55.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:55.807 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:55.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:55.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:55.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:55.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:56.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:56.073 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:56.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:56.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:56.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:56.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:56.242 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 97250 97234 83.2238 56.6875 0.00601915 0.0120099 2024-09-14T04:13:56.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:56.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:56.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:56.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:56.486 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:56.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:56.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:56.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:56.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:56.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:56.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:56 smithi114 ceph-mon[106969]: pgmap v931: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 12531/283851 objects degraded (4.415%); 10589/283851 objects misplaced (3.730%); 17 MiB/s, 268 objects/s recovering 2024-09-14T04:13:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:56 smithi114 ceph-mon[103029]: pgmap v931: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 12531/283851 objects degraded (4.415%); 10589/283851 objects misplaced (3.730%); 17 MiB/s, 268 objects/s recovering 2024-09-14T04:13:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:56 smithi160 ceph-mon[88407]: pgmap v931: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 12531/283851 objects degraded (4.415%); 10589/283851 objects misplaced (3.730%); 17 MiB/s, 268 objects/s recovering 2024-09-14T04:13:56.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:56.764 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:56.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:56.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:56.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:56.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:56.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:56.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:57.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-14T04:13:57.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:57.030 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:57.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:57.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:57.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:57.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:57.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 97969 97953 82.7065 44.9375 0.00397807 0.0120039 2024-09-14T04:13:57.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:57.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:57.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:57.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:57.426 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:57.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:57.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:57.553 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:57.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:57.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:57 smithi114 ceph-mon[103029]: osdmap e254: 8 total, 8 up, 8 in 2024-09-14T04:13:57.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:57 smithi114 ceph-mon[103029]: osdmap e255: 8 total, 8 up, 8 in 2024-09-14T04:13:57.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:57 smithi114 ceph-mon[106969]: osdmap e254: 8 total, 8 up, 8 in 2024-09-14T04:13:57.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:57 smithi114 ceph-mon[106969]: osdmap e255: 8 total, 8 up, 8 in 2024-09-14T04:13:57.655 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:57.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:57 smithi160 ceph-mon[88407]: osdmap e254: 8 total, 8 up, 8 in 2024-09-14T04:13:57.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:57 smithi160 ceph-mon[88407]: osdmap e255: 8 total, 8 up, 8 in 2024-09-14T04:13:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:57.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:57.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:57.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:57.837 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:57.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:57.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:57.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:57.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:13:58.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:58.115 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:58.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:58.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:58.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 98914 98898 82.3912 59.0625 0.0209444 0.0121307 2024-09-14T04:13:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:58.263 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:58.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:58.292 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:13:58.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:13:58.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:58.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:13:58.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:58.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:58 smithi114 ceph-mon[103029]: pgmap v933: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s; 12531/285675 objects degraded (4.386%); 10589/285675 objects misplaced (3.707%); 15 MiB/s, 233 objects/s recovering 2024-09-14T04:13:58.521 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:58 smithi114 ceph-mon[106969]: pgmap v933: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s; 12531/285675 objects degraded (4.386%); 10589/285675 objects misplaced (3.707%); 15 MiB/s, 233 objects/s recovering 2024-09-14T04:13:58.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:58.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:58.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:58.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:58 smithi160 ceph-mon[88407]: pgmap v933: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s; 12531/285675 objects degraded (4.386%); 10589/285675 objects misplaced (3.707%); 15 MiB/s, 233 objects/s recovering 2024-09-14T04:13:58.673 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:58.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:58.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:58.767 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:13:58.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:13:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:58.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:58.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:13:58.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:58.955 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:58.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:59.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:13:59.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:59.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:59.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:13:59.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:13:59.228 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:59.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:13:59.242 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 99853 99837 82.0793 58.6875 0.0247936 0.0121775 2024-09-14T04:13:59.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:13:59.381 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:59.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:13:59.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:59.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:59.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:59.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:13:59.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:13:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:59.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:13:59.657 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:13:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 12531/291789 objects degraded (4.295%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:13:59.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:13:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 12531/291789 objects degraded (4.295%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:13:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:13:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 12531/291789 objects degraded (4.295%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:13:59.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:13:59.785 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:59.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:13:59.887 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:13:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:13:59.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:13:59.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:00.018 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:00.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:00.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:00.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:00.179 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:00.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:00.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:14:00.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 100846 100830 81.8193 62.0625 0.03749 0.0122165 2024-09-14T04:14:00.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:00.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:00.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:00.390 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:14:00.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:00.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:00.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:00.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:00.558 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:14:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:14:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:14:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:00 smithi160 ceph-mon[88407]: pgmap v935: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 55 MiB/s wr, 879 op/s; 12531/291789 objects degraded (4.295%); 9701/291789 objects misplaced (3.325%); 13 MiB/s, 201 objects/s recovering 2024-09-14T04:14:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4188860340' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:14:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:00 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:14:00.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:00.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:00.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:00.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:00.684 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:00.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:00.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:00.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:00.821 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:00.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:00.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:00 smithi114 ceph-mon[106969]: pgmap v935: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 55 MiB/s wr, 879 op/s; 12531/291789 objects degraded (4.295%); 9701/291789 objects misplaced (3.325%); 13 MiB/s, 201 objects/s recovering 2024-09-14T04:14:00.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4188860340' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:14:00.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:00 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:14:00.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:00 smithi114 ceph-mon[103029]: pgmap v935: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 92 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 55 MiB/s wr, 879 op/s; 12531/291789 objects degraded (4.295%); 9701/291789 objects misplaced (3.325%); 13 MiB/s, 201 objects/s recovering 2024-09-14T04:14:00.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4188860340' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:14:00.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:00 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:14:00.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:00.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:00.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:00.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:01.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:01.096 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:01.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:01.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:01.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:01.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:01.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 101648 101632 81.413 50.125 0.0158451 0.0122775 2024-09-14T04:14:01.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:01.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:01.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:01.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:01.487 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:01.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:01.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:01.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:01.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:01 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:14:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:01 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:14:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:01 smithi160 ceph-mon[88407]: osdmap e256: 8 total, 8 up, 8 in 2024-09-14T04:14:01.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:01.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:01 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:14:01.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:01 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:14:01.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:01 smithi114 ceph-mon[103029]: osdmap e256: 8 total, 8 up, 8 in 2024-09-14T04:14:01.735 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:14:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:14:01] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-09-14T04:14:01.735 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:01 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:14:01.735 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:01 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:14:01.735 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:01 smithi114 ceph-mon[106969]: osdmap e256: 8 total, 8 up, 8 in 2024-09-14T04:14:01.760 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:01.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:01.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:01.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:01.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:01.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:01.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:02.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:02.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:02.138 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:02.159 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:02.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:02.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 102491 102475 81.0494 52.6875 0.0043647 0.0123089 2024-09-14T04:14:02.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:02.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:02.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:02.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:02.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:02.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:02.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:02.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:02.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:02.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:02 smithi160 ceph-mon[88407]: pgmap v937: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 744 op/s; 12531/297249 objects degraded (4.216%); 8750/297249 objects misplaced (2.944%); 10 MiB/s, 165 objects/s recovering 2024-09-14T04:14:02.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:02.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:02.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:02.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:02.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:02.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:02 smithi114 ceph-mon[106969]: pgmap v937: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 744 op/s; 12531/297249 objects degraded (4.216%); 8750/297249 objects misplaced (2.944%); 10 MiB/s, 165 objects/s recovering 2024-09-14T04:14:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:02 smithi114 ceph-mon[103029]: pgmap v937: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 744 op/s; 12531/297249 objects degraded (4.216%); 8750/297249 objects misplaced (2.944%); 10 MiB/s, 165 objects/s recovering 2024-09-14T04:14:02.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:02.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:02.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:03.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:03.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:03.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:03.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:03.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:03.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:03.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:03.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:14:03.242913+0000 min lat: 0.00112332 max lat: 2.20623 avg lat: 0.0123946 2024-09-14T04:14:03.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:14:03.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 103226 103210 80.6106 45.9375 0.067314 0.0123946 2024-09-14T04:14:03.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:03.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:03.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:03.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:03.471 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:03.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:03.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:03.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:03.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:03.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:03.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:03.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:03.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:03.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:03.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:03.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:04.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:04.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:04.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:04.140 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:04.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:04.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:04.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 104147 104131 80.326 57.5625 0.00729397 0.0124442 2024-09-14T04:14:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:04.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:04.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:04.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:04.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:04.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:04.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:04.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:04.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:04.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:04.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:04.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:04.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:04.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:04.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:04.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:04 smithi114 ceph-mon[103029]: pgmap v938: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 42 MiB/s wr, 671 op/s; 12531/298959 objects degraded (4.192%); 8750/298959 objects misplaced (2.927%); 9.4 MiB/s, 151 objects/s recovering 2024-09-14T04:14:04.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 12531/298959 objects degraded (4.192%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:04 smithi114 ceph-mon[106969]: pgmap v938: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 42 MiB/s wr, 671 op/s; 12531/298959 objects degraded (4.192%); 8750/298959 objects misplaced (2.927%); 9.4 MiB/s, 151 objects/s recovering 2024-09-14T04:14:04.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 12531/298959 objects degraded (4.192%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:04 smithi160 ceph-mon[88407]: pgmap v938: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 42 MiB/s wr, 671 op/s; 12531/298959 objects degraded (4.192%); 8750/298959 objects misplaced (2.927%); 9.4 MiB/s, 151 objects/s recovering 2024-09-14T04:14:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 12531/298959 objects degraded (4.192%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:04.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:04.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:04.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:05.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:05.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:05.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:05.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:05.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 105099 105083 80.072 59.5 0.0128935 0.0124834 2024-09-14T04:14:05.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:05.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:05.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:05.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:05.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:05.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:05.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:05.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:05.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:05.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:05.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:05.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:05.647 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:05.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:05.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:05.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:05.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:05.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:05.906 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:05.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:06.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:06.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:06.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:06.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:06.160 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:06.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:06.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 105924 105908 79.7285 51.5625 0.0149575 0.0125372 2024-09-14T04:14:06.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:06.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:06.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:06.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:06.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:06.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:06.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:06.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:06.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:06.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:06.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:06.701 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:06.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:06.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:06.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:06.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:06.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:06 smithi114 ceph-mon[106969]: pgmap v939: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 12531/310218 objects degraded (4.039%); 7455/310218 objects misplaced (2.403%); 12 MiB/s, 188 objects/s recovering 2024-09-14T04:14:06.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:06 smithi114 ceph-mon[103029]: pgmap v939: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 12531/310218 objects degraded (4.039%); 7455/310218 objects misplaced (2.403%); 12 MiB/s, 188 objects/s recovering 2024-09-14T04:14:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:06 smithi160 ceph-mon[88407]: pgmap v939: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 12531/310218 objects degraded (4.039%); 7455/310218 objects misplaced (2.403%); 12 MiB/s, 188 objects/s recovering 2024-09-14T04:14:06.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:06.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:06.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:07.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:07.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:07.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:07.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:07.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:07.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:07.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:07.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 106909 106893 79.5122 61.5625 0.0142203 0.0125716 2024-09-14T04:14:07.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:07.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:07.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:07.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:07.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:07.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:07.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:07.632 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:07.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:07.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:07.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:07.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:07.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:07.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:07.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:07.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:07.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:08.024 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:08.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:08.106 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:14:08.106 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "" 2024-09-14T04:14:08.106 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:14:08.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:08.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:08.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:08.243 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 107578 107562 79.0687 41.8125 0.00539438 0.0125624 2024-09-14T04:14:08.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:08.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:08.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:08.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:08.374 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:08.396 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:08.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:08.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:08.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:08.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:08.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:08.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:08.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:08.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:08 smithi114 ceph-mon[106969]: pgmap v940: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 57 MiB/s wr, 912 op/s; 12531/311928 objects degraded (4.017%); 7455/311928 objects misplaced (2.390%); 9.8 MiB/s, 157 objects/s recovering 2024-09-14T04:14:08.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:08 smithi114 ceph-mon[106969]: osdmap e257: 8 total, 8 up, 8 in 2024-09-14T04:14:08.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:08.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:08.691 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:08 smithi114 ceph-mon[103029]: pgmap v940: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 57 MiB/s wr, 912 op/s; 12531/311928 objects degraded (4.017%); 7455/311928 objects misplaced (2.390%); 9.8 MiB/s, 157 objects/s recovering 2024-09-14T04:14:08.691 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:08 smithi114 ceph-mon[103029]: osdmap e257: 8 total, 8 up, 8 in 2024-09-14T04:14:08.691 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:08.691 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:08.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:08.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:08.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:08.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:08 smithi160 ceph-mon[88407]: pgmap v940: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 57 MiB/s wr, 912 op/s; 12531/311928 objects degraded (4.017%); 7455/311928 objects misplaced (2.390%); 9.8 MiB/s, 157 objects/s recovering 2024-09-14T04:14:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:08 smithi160 ceph-mon[88407]: osdmap e257: 8 total, 8 up, 8 in 2024-09-14T04:14:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:08.936 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:08.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:09.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:09.071 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:09.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:09.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:09.210 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:09.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:09.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:09.244 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 108335 108319 78.6984 47.3125 0.00465734 0.012702 2024-09-14T04:14:09.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:09.348 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:09.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:09.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:09.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:09.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:09.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:09.608 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:09 smithi114 ceph-mon[103029]: osdmap e258: 8 total, 8 up, 8 in 2024-09-14T04:14:09.608 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:14:09.608 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 12531/311928 objects degraded (4.017%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:09.611 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:09 smithi114 ceph-mon[106969]: osdmap e258: 8 total, 8 up, 8 in 2024-09-14T04:14:09.612 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:14:09.612 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 12531/311928 objects degraded (4.017%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:09.615 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:14:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:09.642 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:09.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:09.659 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:14:09.659 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:14:09.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:09.783 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:09.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:09.805 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:09 smithi160 ceph-mon[88407]: osdmap e258: 8 total, 8 up, 8 in 2024-09-14T04:14:09.806 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:14:09.806 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 12531/311928 objects degraded (4.017%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:09.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:09.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:09.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:10.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:10.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:10.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:10.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:10.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:10.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:10.246 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 109260 109244 78.4573 57.8125 0.00405177 0.0127383 2024-09-14T04:14:10.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:10.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:10.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:10.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:10.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:10.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:10.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:10.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:10.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:14:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:14:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:14:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:10 smithi160 ceph-mon[88407]: pgmap v943: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 850 op/s; 12531/317661 objects degraded (3.945%); 7455/317661 objects misplaced (2.347%); 4.0 MiB/s, 63 objects/s recovering 2024-09-14T04:14:10.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:10.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:10.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:10.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:10.821 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:10 smithi114 ceph-mon[106969]: pgmap v943: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 850 op/s; 12531/317661 objects degraded (3.945%); 7455/317661 objects misplaced (2.347%); 4.0 MiB/s, 63 objects/s recovering 2024-09-14T04:14:10.828 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:10 smithi114 ceph-mon[103029]: pgmap v943: 99 pgs: 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 93 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 850 op/s; 12531/317661 objects degraded (3.945%); 7455/317661 objects misplaced (2.347%); 4.0 MiB/s, 63 objects/s recovering 2024-09-14T04:14:10.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-14T04:14:10.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:10.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:10.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:11.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:11.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:11.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:11.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:11.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:14:11.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:11.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:11.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:11.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 110240 110224 78.2607 61.25 0.0193724 0.0127721 2024-09-14T04:14:11.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:11.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:11.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:11.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:11.502 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:11.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:11.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:14:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:14:11] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-14T04:14:11.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:11.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:11.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:11.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:11.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:11.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:11.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:11.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:12.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:12.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:12.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:12.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:12.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:12.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:12.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 111036 111020 77.9404 49.75 0.0179625 0.0128253 2024-09-14T04:14:12.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:12.305 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:12.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:12.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:12.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:12.439 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:12.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:12.541 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:12.587 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:12.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:12.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:12.709 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:12.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:12.758 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:12 smithi160 ceph-mon[88407]: pgmap v944: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 12531/323208 objects degraded (3.877%); 6040/323208 objects misplaced (1.869%); 8.7 MiB/s, 138 objects/s recovering 2024-09-14T04:14:12.812 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:12.841 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:12.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:12.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:12.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:12 smithi114 ceph-mon[106969]: pgmap v944: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 12531/323208 objects degraded (3.877%); 6040/323208 objects misplaced (1.869%); 8.7 MiB/s, 138 objects/s recovering 2024-09-14T04:14:12.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:12 smithi114 ceph-mon[103029]: pgmap v944: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 12531/323208 objects degraded (3.877%); 6040/323208 objects misplaced (1.869%); 8.7 MiB/s, 138 objects/s recovering 2024-09-14T04:14:12.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:12.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:12.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:12.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:13.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:13.105 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:13.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:13.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:13.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:13.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:13.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:13.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0173 2024-09-14T04:14:13.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 111934 2024-09-14T04:14:13.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-14T04:14:13.247 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 77.717 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 41.328 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 209.562 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 0 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1243 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 661.256 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3353 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 0 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0128619 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0465079 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 2.20623 2024-09-14T04:14:13.248 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00112332 2024-09-14T04:14:13.265 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-14T04:14:13.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:13.360 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:13.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:13.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:13.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:13.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:13.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:13.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:13.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:13.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:13.755 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:13.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:13.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:13.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:13.916 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:13.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:14.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:14.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:14.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:14.139 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:14:14.140 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:14:14.140 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-14T04:14:14.140 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.188629 2024-09-14T04:14:14.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 0.18862908838254944 2024-09-14T04:14:14.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:14.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:14.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:14.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:14.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:14.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:14.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:14.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:14.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:14.595 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:14.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:14.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:14.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:14.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:14.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:14.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:14.813 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:14 smithi160 ceph-mon[88407]: pgmap v945: 99 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 38 MiB/s wr, 603 op/s; 7505/324699 objects degraded (2.311%); 10965/324699 objects misplaced (3.377%); 5.5 MiB/s, 87 objects/s recovering 2024-09-14T04:14:14.813 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7505/324699 objects degraded (2.311%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:14.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:14.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:14 smithi114 ceph-mon[103029]: pgmap v945: 99 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 38 MiB/s wr, 603 op/s; 7505/324699 objects degraded (2.311%); 10965/324699 objects misplaced (3.377%); 5.5 MiB/s, 87 objects/s recovering 2024-09-14T04:14:14.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7505/324699 objects degraded (2.311%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:14.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:14 smithi114 ceph-mon[106969]: pgmap v945: 99 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 38 MiB/s wr, 603 op/s; 7505/324699 objects degraded (2.311%); 10965/324699 objects misplaced (3.377%); 5.5 MiB/s, 87 objects/s recovering 2024-09-14T04:14:14.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7505/324699 objects degraded (2.311%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:14.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:14.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:14.984 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:15.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:15.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:15.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:15.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:15.149 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:15.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:15.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:15.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:15.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:15.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:15.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:15.420 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:15.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:15.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:15.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:15.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:15.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:15.694 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:15.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:15.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:15.826 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:15.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:15.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:15.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:15.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:16.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:16.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:16.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:16.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:16.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:16.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:16.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:16.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:16.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:16.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:16.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:16.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:16.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:16.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:16.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:16.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:16 smithi114 ceph-mon[106969]: pgmap v946: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 1.03k op/s; 7505/333813 objects degraded (2.248%); 10143/333813 objects misplaced (3.039%); 8.7 MiB/s, 138 objects/s recovering 2024-09-14T04:14:16.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:16 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:16.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:16 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:16 smithi114 ceph-mon[106969]: osdmap e259: 8 total, 8 up, 8 in 2024-09-14T04:14:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:16 smithi114 ceph-mon[103029]: pgmap v946: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 1.03k op/s; 7505/333813 objects degraded (2.248%); 10143/333813 objects misplaced (3.039%); 8.7 MiB/s, 138 objects/s recovering 2024-09-14T04:14:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:16 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:16 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:16.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:16 smithi114 ceph-mon[103029]: osdmap e259: 8 total, 8 up, 8 in 2024-09-14T04:14:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:16 smithi160 ceph-mon[88407]: pgmap v946: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 1.03k op/s; 7505/333813 objects degraded (2.248%); 10143/333813 objects misplaced (3.039%); 8.7 MiB/s, 138 objects/s recovering 2024-09-14T04:14:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:16 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:16 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:16 smithi160 ceph-mon[88407]: osdmap e259: 8 total, 8 up, 8 in 2024-09-14T04:14:16.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:16.719 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:16.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:16.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:16.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:16.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:16.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:16.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:16.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:16.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:17.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:17.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:17.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:17.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:17.262 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:17.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:17.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:17.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:17.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:17.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:17.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:17.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:17.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:17.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:17.648 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:17.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:17.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:17.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:17.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:17.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:17.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:17 smithi114 ceph-mon[106969]: osdmap e260: 8 total, 8 up, 8 in 2024-09-14T04:14:17.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:17 smithi114 ceph-mon[103029]: osdmap e260: 8 total, 8 up, 8 in 2024-09-14T04:14:17.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:17.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:17 smithi160 ceph-mon[88407]: osdmap e260: 8 total, 8 up, 8 in 2024-09-14T04:14:17.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:17.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:17.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:18.021 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:18.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:18.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:18.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:18.173 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:18.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:18.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:18.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:18.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:18.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:18.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:18.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:18.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:18.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:18.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:18.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:18.723 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:18.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:18.807 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:18 smithi114 ceph-mon[106969]: pgmap v949: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 127 B/s rd, 48 MiB/s wr, 863 op/s; 7505/333522 objects degraded (2.250%); 9843/333522 objects misplaced (2.951%); 10 MiB/s, 164 objects/s recovering 2024-09-14T04:14:18.807 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:18 smithi114 ceph-mon[106969]: 11.11 scrub starts 2024-09-14T04:14:18.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:18 smithi114 ceph-mon[103029]: pgmap v949: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 127 B/s rd, 48 MiB/s wr, 863 op/s; 7505/333522 objects degraded (2.250%); 9843/333522 objects misplaced (2.951%); 10 MiB/s, 164 objects/s recovering 2024-09-14T04:14:18.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:18 smithi114 ceph-mon[103029]: 11.11 scrub starts 2024-09-14T04:14:18.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:18.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:18.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:18 smithi160 ceph-mon[88407]: pgmap v949: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 127 B/s rd, 48 MiB/s wr, 863 op/s; 7505/333522 objects degraded (2.250%); 9843/333522 objects misplaced (2.951%); 10 MiB/s, 164 objects/s recovering 2024-09-14T04:14:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:18 smithi160 ceph-mon[88407]: 11.11 scrub starts 2024-09-14T04:14:18.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:18.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:18.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:19.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:19.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:19.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:19.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:19.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:19.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:19.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:19.404 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:19.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:19.434 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:14:19.434 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:14:19.468 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:14:19.469 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:14:19.469 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:14:19.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:19 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4237529424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:14:19.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7505/333522 objects degraded (2.250%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:19.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:19.521 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:14:19.522 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:14:19.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:19.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:19.559 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:14:19.570 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:14:19.570 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:14:19.598 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:14:19.599 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:14:19.629 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:14:19.630 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:14:19.630 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:19.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:19.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:19.666 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:14:19.666 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:14:19.693 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:14:19.693 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:14:19.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:19.799 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4237529424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:14:19.799 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7505/333522 objects degraded (2.250%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:19.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:19.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:19 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4237529424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:14:19.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7505/333522 objects degraded (2.250%), 2 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:19.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:19.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:19.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:20.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:20.079 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:20.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:20.181 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:20.210 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:20.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:20.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:20.352 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:20.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:20.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:20.490 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:14:20.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:20.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:20.500 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:14:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:14:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:14:20.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:20.634 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:20.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:20.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:20.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:20.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:20.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:20.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:20 smithi114 ceph-mon[103029]: pgmap v950: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 33 MiB/s wr, 869 op/s; 7505/327897 objects degraded (2.289%); 9843/327897 objects misplaced (3.002%); 5.6 MiB/s, 89 objects/s recovering 2024-09-14T04:14:20.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:20 smithi114 ceph-mon[106969]: pgmap v950: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 33 MiB/s wr, 869 op/s; 7505/327897 objects degraded (2.289%); 9843/327897 objects misplaced (3.002%); 5.6 MiB/s, 89 objects/s recovering 2024-09-14T04:14:20.889 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:20.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:20.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:20 smithi160 ceph-mon[88407]: pgmap v950: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 33 MiB/s wr, 869 op/s; 7505/327897 objects degraded (2.289%); 9843/327897 objects misplaced (3.002%); 5.6 MiB/s, 89 objects/s recovering 2024-09-14T04:14:20.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:21.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:21.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:21.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:21.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:21.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:21.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:21.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:21.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:21.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:21.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:21.401 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:21.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:21.503 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:21.540 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:21.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:21.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:21.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:21.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:21.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:21.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:21.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:14:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:14:21] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-14T04:14:21.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:21.910 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:21.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:22.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:22.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:22.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:22.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:22.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:22.169 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:22.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:22.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:22.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:22.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:22.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:22.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:22.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:22.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:22.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:22.576 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:22.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:22.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:22.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:22 smithi114 ceph-mon[103029]: pgmap v951: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 29 MiB/s wr, 1.03k op/s; 7505/322431 objects degraded (2.328%); 9843/322431 objects misplaced (3.053%); 4.8 MiB/s, 77 objects/s recovering 2024-09-14T04:14:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:22 smithi114 ceph-mon[106969]: pgmap v951: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 29 MiB/s wr, 1.03k op/s; 7505/322431 objects degraded (2.328%); 9843/322431 objects misplaced (3.053%); 4.8 MiB/s, 77 objects/s recovering 2024-09-14T04:14:22.678 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:22.710 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:22.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:22.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:22.812 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:22.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:22.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:22.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:22.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:22 smithi160 ceph-mon[88407]: pgmap v951: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 29 MiB/s wr, 1.03k op/s; 7505/322431 objects degraded (2.328%); 9843/322431 objects misplaced (3.053%); 4.8 MiB/s, 77 objects/s recovering 2024-09-14T04:14:22.941 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:22.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:22.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:22.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:23.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:23.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:23.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:23.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:23.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:23.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:23.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:23.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:23.379 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:23.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:23.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:23.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:23.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:23.639 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:23.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:23.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:23.722 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:23.722 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:23.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:23.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:23.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:23.939 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:23.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:24.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:24.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:24.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:24.169 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:24.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:24.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:24.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:24.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:24.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:24.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:14:24.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:24.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:24.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:24.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:24.611 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:24.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:24.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:24 smithi114 ceph-mon[103029]: pgmap v952: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 2.1 MiB/s wr, 630 op/s; 7628/320271 objects degraded (2.382%); 9600/320271 objects misplaced (2.997%); 2.6 MiB/s, 40 objects/s recovering 2024-09-14T04:14:24.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7628/320271 objects degraded (2.382%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:24.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:24 smithi114 ceph-mon[106969]: pgmap v952: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 2.1 MiB/s wr, 630 op/s; 7628/320271 objects degraded (2.382%); 9600/320271 objects misplaced (2.997%); 2.6 MiB/s, 40 objects/s recovering 2024-09-14T04:14:24.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7628/320271 objects degraded (2.382%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:24.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:24.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:24.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:24.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:24.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:24.887 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:24.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:24 smithi160 ceph-mon[88407]: pgmap v952: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 2.1 MiB/s wr, 630 op/s; 7628/320271 objects degraded (2.382%); 9600/320271 objects misplaced (2.997%); 2.6 MiB/s, 40 objects/s recovering 2024-09-14T04:14:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7628/320271 objects degraded (2.382%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:24.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:25.025 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:25.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:25.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:25.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:25.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:25.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:25.312 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:25.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:25.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:25.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:25.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:25.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:25.536 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:25.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:25.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:25.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:25.689 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:25.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:25.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:25.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:25.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:25.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:25.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:25.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:25.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:26.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:26.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:26.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:26.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:26.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:26.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:26.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:26.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:26.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:26.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:26.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:26.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:26.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:26.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:26.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:26.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:26.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:26.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:26.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:123299 11.b 11:ddf171f5:::benchmark_data_smithi114_223345_object11362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.362147+0000", 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.065518245000000003, 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.065557899000000003, 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:14:26.843 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123299 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362147+0000", 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362147+0000", 2024-09-14T04:14:26.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362149+0000", 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.883e-06 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362155+0000", 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9869999999999996e-06 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362170+0000", 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5221999999999999e-05 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362178+0000", 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2300000000000008e-06 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362292+0000", 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113782 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362418+0000", 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000126143 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362543+0000", 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012447700000000001 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.366977+0000", 2024-09-14T04:14:26.847 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044344850000000002 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.371182+0000", 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042044480000000004 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:123317 11.b 11:d165ac94:::benchmark_data_smithi114_223345_object11380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.385232+0000", 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.042432928000000002, 2024-09-14T04:14:26.848 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.042565356999999998, 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123317 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385232+0000", 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:14:26.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385232+0000", 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385233+0000", 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4299999999999998e-07 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:14:26.850 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385235+0000", 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4930000000000001e-06 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385243+0000", 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4800000000000004e-06 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385248+0000", 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5149999999999997e-06 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.851 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385335+0000", 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6181999999999996e-05 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385459+0000", 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012396600000000001 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385572+0000", 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011335 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.391607+0000", 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0060352610000000001 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.392474+0000", 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00086647000000000004 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:123318 11.b 11:d7575392:::benchmark_data_smithi114_223345_object11381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.385351+0000", 2024-09-14T04:14:26.853 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.042314464000000003, 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.042536167999999999, 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123318 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:14:26.854 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385351+0000", 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385351+0000", 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385352+0000", 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2500000000000005e-07 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:14:26.855 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385355+0000", 2024-09-14T04:14:26.856 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4429999999999999e-06 2024-09-14T04:14:26.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.856 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.856 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:14:26.856 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385362+0000", 2024-09-14T04:14:26.856 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.058e-06 2024-09-14T04:14:26.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.856 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.856 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:14:26.856 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385368+0000", 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9800000000000003e-06 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.388571+0000", 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003203165 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.388662+0000", 2024-09-14T04:14:26.857 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0741000000000004e-05 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.388735+0000", 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2967999999999997e-05 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.391779+0000", 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030441439999999999 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:14:26.858 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.392713+0000", 2024-09-14T04:14:26.859 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00093342000000000002 2024-09-14T04:14:26.859 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:14:26.859 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:14:26.859 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:14:26.859 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:14:26.859 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:14:26.859 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 3 2024-09-14T04:14:26.859 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:14:26.859 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:26 smithi114 ceph-mon[106969]: pgmap v953: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 108 B/s rd, 1.7 MiB/s wr, 973 op/s; 7628/307905 objects degraded (2.477%); 9600/307905 objects misplaced (3.118%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:14:26.860 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:26 smithi114 ceph-mon[103029]: pgmap v953: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 108 B/s rd, 1.7 MiB/s wr, 973 op/s; 7628/307905 objects degraded (2.477%); 9600/307905 objects misplaced (3.118%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:14:26.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:26 smithi160 ceph-mon[88407]: pgmap v953: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 108 B/s rd, 1.7 MiB/s wr, 973 op/s; 7628/307905 objects degraded (2.477%); 9600/307905 objects misplaced (3.118%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:14:26.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:26.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:27.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:27.060 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:27.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:27.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:27.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:27.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:27.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:27.335 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:27.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:27.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:27.468 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:27.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:27.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:27.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:27.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:27.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:27.733 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:27.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:27.834 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:27.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:27.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:27.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:27.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:27.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:27.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:28.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:28.121 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:28.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:28.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:28.268 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:28.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:28.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:28.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:28.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:28.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:28.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:28.557 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:28.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:28.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:28 smithi114 ceph-mon[106969]: pgmap v954: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 KiB/s wr, 888 op/s; 7767/305808 objects degraded (2.540%); 9332/305808 objects misplaced (3.052%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:14:28.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:28 smithi114 ceph-mon[103029]: pgmap v954: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 KiB/s wr, 888 op/s; 7767/305808 objects degraded (2.540%); 9332/305808 objects misplaced (3.052%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:14:28.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:28.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:28.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:28.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:28.817 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:28.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:28 smithi160 ceph-mon[88407]: pgmap v954: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 KiB/s wr, 888 op/s; 7767/305808 objects degraded (2.540%); 9332/305808 objects misplaced (3.052%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:14:28.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:28.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:28.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:29.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:29.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:29.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:29.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:29.218 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:29.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:29.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:29.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:29.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:29.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:29.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:29.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:29.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:29.539 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.7 primary-affinity to 0.188629 (8123612) 2024-09-14T04:14:29.540 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:14:29.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:29.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:29.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:29.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:29.736 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:29.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:29.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:29.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:29.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:29.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:29 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1022649602' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.18862908838254944} : dispatch 2024-09-14T04:14:29.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:29 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.18862908838254944} : dispatch 2024-09-14T04:14:29.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:29 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/658380792' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:14:29.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:29 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:14:29.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7767/305808 objects degraded (2.540%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:29.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:29 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1022649602' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.18862908838254944} : dispatch 2024-09-14T04:14:29.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:29 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.18862908838254944} : dispatch 2024-09-14T04:14:29.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:29 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/658380792' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:14:29.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:29 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:14:29.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7767/305808 objects degraded (2.540%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:29.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:29 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1022649602' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.18862908838254944} : dispatch 2024-09-14T04:14:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:29 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.18862908838254944} : dispatch 2024-09-14T04:14:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:29 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/658380792' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:14:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:29 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:14:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7767/305808 objects degraded (2.540%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:29.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:29.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:29.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:30.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:30.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:30.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:30.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:30.230 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:30.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:30.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:30.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:30.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:30.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:30.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:30.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:30.546 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:14:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:14:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:14:30.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:30.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:30.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:30.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:30.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:30.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:30.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:30.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[106969]: pgmap v955: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 KiB/s wr, 946 op/s; 7767/299457 objects degraded (2.594%); 9332/299457 objects misplaced (3.116%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:14:30.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:14:30.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.18862908838254944}]': finished 2024-09-14T04:14:30.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:14:30.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[106969]: osdmap e261: 8 total, 8 up, 8 in 2024-09-14T04:14:30.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[106969]: osdmap e262: 8 total, 8 up, 8 in 2024-09-14T04:14:30.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[103029]: pgmap v955: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 KiB/s wr, 946 op/s; 7767/299457 objects degraded (2.594%); 9332/299457 objects misplaced (3.116%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:14:30.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:14:30.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.18862908838254944}]': finished 2024-09-14T04:14:30.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:14:30.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[103029]: osdmap e261: 8 total, 8 up, 8 in 2024-09-14T04:14:30.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:30 smithi114 ceph-mon[103029]: osdmap e262: 8 total, 8 up, 8 in 2024-09-14T04:14:30.872 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:30 smithi160 ceph-mon[88407]: pgmap v955: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 KiB/s wr, 946 op/s; 7767/299457 objects degraded (2.594%); 9332/299457 objects misplaced (3.116%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:14:30.872 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:30 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:14:30.873 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:30 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.18862908838254944}]': finished 2024-09-14T04:14:30.873 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:30 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:14:30.873 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:30 smithi160 ceph-mon[88407]: osdmap e261: 8 total, 8 up, 8 in 2024-09-14T04:14:30.873 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:30 smithi160 ceph-mon[88407]: osdmap e262: 8 total, 8 up, 8 in 2024-09-14T04:14:30.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:30.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:30.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:30.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:31.016 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:31.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:31.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:31.154 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:31.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:31.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:31.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:31.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:31.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:31.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:31.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:31.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:31.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:31.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:31.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:31.677 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:31.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:31.725 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:31 smithi160 ceph-mon[88407]: pgmap v958: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 7767/293355 objects degraded (2.648%); 9332/293355 objects misplaced (3.181%); 1.0 MiB/s, 16 objects/s recovering 2024-09-14T04:14:31.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:31.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:31.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:31.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:31 smithi114 ceph-mon[103029]: pgmap v958: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 7767/293355 objects degraded (2.648%); 9332/293355 objects misplaced (3.181%); 1.0 MiB/s, 16 objects/s recovering 2024-09-14T04:14:31.872 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:14:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:14:31] "GET /metrics HTTP/1.1" 200 36595 "" "Prometheus/2.51.0" 2024-09-14T04:14:31.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:31 smithi114 ceph-mon[106969]: pgmap v958: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 6.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 7767/293355 objects degraded (2.648%); 9332/293355 objects misplaced (3.181%); 1.0 MiB/s, 16 objects/s recovering 2024-09-14T04:14:31.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:31.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:31.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:32.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:32.082 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:32.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:32.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:32.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:32.221 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:32.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:32.323 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:32.348 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:32.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:32.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:32.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:32.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:32.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:32.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:32.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:32.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:32.757 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:32.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:32.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:32.905 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:32.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:33.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:33.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:33.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:33.141 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:33.174 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:33.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:33.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:33.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:33.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:33.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:33.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:33.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:33.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:33.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:33.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:33.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:33.671 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:33.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:33.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:33.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:33.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:33.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:33.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:33.985 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:33.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:34.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:34.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:34.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:34.213 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:33 smithi114 ceph-mon[106969]: pgmap v959: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 7898/291540 objects degraded (2.709%); 9024/291540 objects misplaced (3.095%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:14:34.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:33 smithi114 ceph-mon[103029]: pgmap v959: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 7898/291540 objects degraded (2.709%); 9024/291540 objects misplaced (3.095%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:14:34.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:34.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:34.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:34.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:34.392 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:34.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:34.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:33 smithi160 ceph-mon[88407]: pgmap v959: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 7898/291540 objects degraded (2.709%); 9024/291540 objects misplaced (3.095%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:14:34.494 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:34.520 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:34.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:34.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:34.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:34.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:34.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:34.747 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:14:34.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-14T04:14:34.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:34.868 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:34.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:34.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:34.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:34.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:34.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:35.082 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7898/291540 objects degraded (2.709%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:35.082 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7898/291540 objects degraded (2.709%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:35.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:35.157 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:35.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:35.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:35.299 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:35.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:35.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:35.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7898/291540 objects degraded (2.709%), 3 pgs degraded (PG_DEGRADED) 2024-09-14T04:14:35.429 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:35.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:35.531 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:35.563 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:35.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:35.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:35.696 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:35.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:35.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:35.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:35.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:35.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:35.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:35.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:36.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:36.097 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:36.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:36.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:36.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:36.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:36.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:36.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:36.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:36.365 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:35 smithi114 ceph-mon[106969]: pgmap v960: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 980 op/s; 7898/282252 objects degraded (2.798%); 9024/282252 objects misplaced (3.197%); 1.4 MiB/s, 21 objects/s recovering 2024-09-14T04:14:36.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:35 smithi114 ceph-mon[103029]: pgmap v960: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 980 op/s; 7898/282252 objects degraded (2.798%); 9024/282252 objects misplaced (3.197%); 1.4 MiB/s, 21 objects/s recovering 2024-09-14T04:14:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:35 smithi160 ceph-mon[88407]: pgmap v960: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 980 op/s; 7898/282252 objects degraded (2.798%); 9024/282252 objects misplaced (3.197%); 1.4 MiB/s, 21 objects/s recovering 2024-09-14T04:14:36.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:36.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:36.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:36.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:36.633 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:36.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:36.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:36.769 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:36.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:36.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:36.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:36.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:37.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:37.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:37.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:37.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:37.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:37.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:37.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:37.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:37.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:37.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:37.460 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:37.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:37.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:37.600 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:37.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:37.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:37.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:37.742 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:37.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:37.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:37.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:37.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:37.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:38.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:38.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:38.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:38.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:37 smithi114 ceph-mon[103029]: pgmap v961: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 808 op/s; 8015/280050 objects degraded (2.862%); 8807/280050 objects misplaced (3.145%); 2.2 MiB/s, 33 objects/s recovering 2024-09-14T04:14:38.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:38.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:37 smithi114 ceph-mon[106969]: pgmap v961: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 808 op/s; 8015/280050 objects degraded (2.862%); 8807/280050 objects misplaced (3.145%); 2.2 MiB/s, 33 objects/s recovering 2024-09-14T04:14:38.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:38.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:38.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:38.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:38.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:38.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:38.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:38.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:37 smithi160 ceph-mon[88407]: pgmap v961: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 808 op/s; 8015/280050 objects degraded (2.862%); 8807/280050 objects misplaced (3.145%); 2.2 MiB/s, 33 objects/s recovering 2024-09-14T04:14:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:38.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:38.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:38.512 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:14:38.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:38.553 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:38.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:38.567 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:14:38.568 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:14:38.606 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:14:38.606 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:14:38.639 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:14:38.639 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:14:38.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:38.683 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:14:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:14:38.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:38.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:38.715 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:14:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:14:38.747 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:14:38.747 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:14:38.777 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:14:38.777 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:14:38.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:38.807 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:14:38.807 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:14:38.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:38.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:38.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:38.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:38.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:39.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:39.105 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:39.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:39.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2843412805' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:14:39.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2843412805' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:14:39.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:39.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:39.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:39.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:39.389 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:39.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:39.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:39.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2843412805' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:14:39.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:39.516 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:39.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:39.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:39.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:39.658 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:39.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:39.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:39.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:39.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:39.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:14:39.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:39.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:39.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:39.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:40.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:40.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:40.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:40.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:40.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:40.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:40.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:40.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:40.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:40.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:39 smithi114 ceph-mon[106969]: pgmap v962: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 8015/271914 objects degraded (2.948%); 8807/271914 objects misplaced (3.239%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T04:14:40.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:39 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8015/271914 objects degraded (2.948%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:40.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:39 smithi114 ceph-mon[103029]: pgmap v962: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 8015/271914 objects degraded (2.948%); 8807/271914 objects misplaced (3.239%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T04:14:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:39 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8015/271914 objects degraded (2.948%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:14:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:14:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:14:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:39 smithi160 ceph-mon[88407]: pgmap v962: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 8015/271914 objects degraded (2.948%); 8807/271914 objects misplaced (3.239%); 1.8 MiB/s, 29 objects/s recovering 2024-09-14T04:14:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:39 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8015/271914 objects degraded (2.948%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:40.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:40.492 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:40.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:40.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:40.638 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:40.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:40.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:40.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:40.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:40.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:40.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:40.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:40.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:41.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:41.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:41.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:41.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:41.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:41.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:41.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:41.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:41.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:41.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:41.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:41.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:41.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:41.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:41.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:41.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:14:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:14:41] "GET /metrics HTTP/1.1" 200 36593 "" "Prometheus/2.51.0" 2024-09-14T04:14:41.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:41.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:41.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:41.852 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:14:41.852 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:14:41.852 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-14T04:14:41.852 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:14:41.852 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:14:41.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-14T04:14:41.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:41.873 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:41.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:41.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:42.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:42.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:42.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:42.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:42.180 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:42.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:42.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:42.262 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:41 smithi114 ceph-mon[103029]: pgmap v963: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 8015/267630 objects degraded (2.995%); 8807/267630 objects misplaced (3.291%); 1.7 MiB/s, 26 objects/s recovering 2024-09-14T04:14:42.263 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:41 smithi114 ceph-mon[106969]: pgmap v963: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 8015/267630 objects degraded (2.995%); 8807/267630 objects misplaced (3.291%); 1.7 MiB/s, 26 objects/s recovering 2024-09-14T04:14:42.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:42.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:42.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:41 smithi160 ceph-mon[88407]: pgmap v963: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 8015/267630 objects degraded (2.995%); 8807/267630 objects misplaced (3.291%); 1.7 MiB/s, 26 objects/s recovering 2024-09-14T04:14:42.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:42.456 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:42.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:42.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:42.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:42.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:42.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:42.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:42.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:42.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:42.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:42.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:42.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:42.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:42.991 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:42.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:43.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:43.144 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:43.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:43.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:43.282 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:43.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:43.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:43.428 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:43.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:43.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:43.567 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:43.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:43.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:43.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:43.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:43.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:43.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:43.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:43.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:43.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:43.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:43.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:44.083 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:14:44.083 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:14:44.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:44.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:44.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:44.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:44.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:44.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:43 smithi114 ceph-mon[106969]: pgmap v964: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 8260/265182 objects degraded (3.115%); 8471/265182 objects misplaced (3.194%); 1.9 MiB/s, 30 objects/s recovering 2024-09-14T04:14:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:43 smithi114 ceph-mon[103029]: pgmap v964: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 8260/265182 objects degraded (3.115%); 8471/265182 objects misplaced (3.194%); 1.9 MiB/s, 30 objects/s recovering 2024-09-14T04:14:44.374 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:44.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:44.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:44.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:43 smithi160 ceph-mon[88407]: pgmap v964: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 8260/265182 objects degraded (3.115%); 8471/265182 objects misplaced (3.194%); 1.9 MiB/s, 30 objects/s recovering 2024-09-14T04:14:44.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:44.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:44.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:44.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:44.685 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:44.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:44.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:44.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:44.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:44.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:44.900 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:14:44.904 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:14:44.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:44.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:44.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:45.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:45.105 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:45 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8260/265182 objects degraded (3.115%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:45.106 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:45 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8260/265182 objects degraded (3.115%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:45.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:45.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:45.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:45.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:45.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:45.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:45.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-14T04:14:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:45.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:45.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:45.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:45 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8260/265182 objects degraded (3.115%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:45.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:45.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:45.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:45.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:45.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:45.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:45.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:45.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:45.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:45.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:45.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:45.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:45.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:46.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:46.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:46.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:46.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:46.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:46.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:46.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:46.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:46.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:46.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:46 smithi114 ceph-mon[103029]: pgmap v965: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 8260/252645 objects degraded (3.269%); 8471/252645 objects misplaced (3.353%); 1018 KiB/s, 15 objects/s recovering 2024-09-14T04:14:46.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:46 smithi114 ceph-mon[106969]: pgmap v965: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 8260/252645 objects degraded (3.269%); 8471/252645 objects misplaced (3.353%); 1018 KiB/s, 15 objects/s recovering 2024-09-14T04:14:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:46 smithi160 ceph-mon[88407]: pgmap v965: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 8260/252645 objects degraded (3.269%); 8471/252645 objects misplaced (3.353%); 1018 KiB/s, 15 objects/s recovering 2024-09-14T04:14:46.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:46.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:46.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:46.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:46.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:46.632 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:46.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:46.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:46.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:46.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:46.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:46.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:46.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:14:46.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:46.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:47.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:47.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:47.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:47.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:47.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:47.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:47.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:47.305 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:47.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:47.407 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:47.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:47.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:47.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:47.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:47.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:47.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:47.712 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:47.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:47.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:47.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:47.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:47.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:47.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:47.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:48.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:48.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:48.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:48.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:48.239 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:48.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:48.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:48.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:48 smithi114 ceph-mon[106969]: pgmap v966: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 8487/250263 objects degraded (3.391%); 8151/250263 objects misplaced (3.257%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:14:48.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:48 smithi114 ceph-mon[106969]: 11.11 scrub starts 2024-09-14T04:14:48.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:48 smithi114 ceph-mon[103029]: pgmap v966: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 8487/250263 objects degraded (3.391%); 8151/250263 objects misplaced (3.257%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:14:48.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:48 smithi114 ceph-mon[103029]: 11.11 scrub starts 2024-09-14T04:14:48.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:48.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:48 smithi160 ceph-mon[88407]: pgmap v966: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 8487/250263 objects degraded (3.391%); 8151/250263 objects misplaced (3.257%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:14:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:48 smithi160 ceph-mon[88407]: 11.11 scrub starts 2024-09-14T04:14:48.478 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:48.506 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:48.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:48.608 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:48.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:48.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:48.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:48.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:48.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:48.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:48.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:48.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:49.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:49.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:49.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:49.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:49.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:49.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:49.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:49.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:49.303 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:49.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:49.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:49.426 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:49.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:49.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:49.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:49.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:49.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:49.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:49.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:49.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:49.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:49.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:49.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:49.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:49.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:49.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:49.950 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:49.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:50.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:50.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:50.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:50.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:50.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:50.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:50.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:50.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:50.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:50.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:50.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:50 smithi114 ceph-mon[106969]: pgmap v967: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 8487/242001 objects degraded (3.507%); 8151/242001 objects misplaced (3.368%); 981 KiB/s, 15 objects/s recovering 2024-09-14T04:14:50.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:50 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8487/242001 objects degraded (3.507%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:50.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:50 smithi114 ceph-mon[103029]: pgmap v967: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 8487/242001 objects degraded (3.507%); 8151/242001 objects misplaced (3.368%); 981 KiB/s, 15 objects/s recovering 2024-09-14T04:14:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:50 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8487/242001 objects degraded (3.507%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:50 smithi160 ceph-mon[88407]: pgmap v967: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 8487/242001 objects degraded (3.507%); 8151/242001 objects misplaced (3.368%); 981 KiB/s, 15 objects/s recovering 2024-09-14T04:14:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:50 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8487/242001 objects degraded (3.507%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:14:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:14:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:14:50.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:50.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:50.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:50.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:50.650 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:50.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:50.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:50.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:50.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:50.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:50.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:50.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:51.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:51.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:51.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:51.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:51.194 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:51.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:51.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:51.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:51.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:51.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:51.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:14:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:14:51.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:14:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:14:51.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:14:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:14:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:14:51.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:51.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:51.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:51.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:51.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:51.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:51.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:51.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:51.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:51.756 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:51.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:51.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:51.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:14:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:14:51] "GET /metrics HTTP/1.1" 200 36593 "" "Prometheus/2.51.0" 2024-09-14T04:14:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:51.895 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:51.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:51.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:52.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:52.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:52.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:52.215 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:52.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:52.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:52.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:52.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:52.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:52 smithi114 ceph-mon[103029]: pgmap v968: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 946 op/s; 8487/237831 objects degraded (3.569%); 8151/237831 objects misplaced (3.427%); 981 KiB/s, 15 objects/s recovering 2024-09-14T04:14:52.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:52 smithi114 ceph-mon[106969]: pgmap v968: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 946 op/s; 8487/237831 objects degraded (3.569%); 8151/237831 objects misplaced (3.427%); 981 KiB/s, 15 objects/s recovering 2024-09-14T04:14:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:52 smithi160 ceph-mon[88407]: pgmap v968: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 946 op/s; 8487/237831 objects degraded (3.569%); 8151/237831 objects misplaced (3.427%); 981 KiB/s, 15 objects/s recovering 2024-09-14T04:14:52.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:52.486 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:52.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:52.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:52.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:52.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:52.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:52.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:52.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:52.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:52.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:52.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:52.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:53.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:53.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:53.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:53.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:53.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:53.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:53.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:53.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:53.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:53.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:53.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:53.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:14:53.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:53.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:53.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:53.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:53.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:53.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:53.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:53.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:53.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:53.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:53.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:53.945 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:53.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:54.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:54.082 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:14:54.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:54.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:54.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:54.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:54.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:54.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:54.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:54.342 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:54.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:54.370 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:14:54.370 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "" 2024-09-14T04:14:54.370 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:14:54.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:54 smithi114 ceph-mon[106969]: pgmap v969: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 8684/235584 objects degraded (3.686%); 7850/235584 objects misplaced (3.332%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:14:54.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:54 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4090719305' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:14:54.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:54 smithi114 ceph-mon[103029]: pgmap v969: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 8684/235584 objects degraded (3.686%); 7850/235584 objects misplaced (3.332%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:14:54.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:54 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4090719305' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:14:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:54 smithi160 ceph-mon[88407]: pgmap v969: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 8684/235584 objects degraded (3.686%); 7850/235584 objects misplaced (3.332%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:14:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:54 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4090719305' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:14:54.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:54.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:54.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:54.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:54.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:54.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:54.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:54.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:54.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:54.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:54.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:54.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:55.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:55.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:55.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:55.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:55.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:55.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:55.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:55 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:14:55.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:55 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4090719305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:14:55.262 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:55 smithi114 ceph-mon[103029]: osdmap e263: 8 total, 8 up, 8 in 2024-09-14T04:14:55.262 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:55 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8684/235584 objects degraded (3.686%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:55.262 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:55 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:14:55.262 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:55 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4090719305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:14:55.262 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:55 smithi114 ceph-mon[106969]: osdmap e263: 8 total, 8 up, 8 in 2024-09-14T04:14:55.262 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:55 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8684/235584 objects degraded (3.686%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:55.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:55.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:55.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:55 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:14:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:55 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4090719305' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:14:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:55 smithi160 ceph-mon[88407]: osdmap e263: 8 total, 8 up, 8 in 2024-09-14T04:14:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:55 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8684/235584 objects degraded (3.686%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:14:55.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:55.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:55.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:55.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:55.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:55.590 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:55.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:55.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:55.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:55.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:55.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:55.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:55.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:55.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:55.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:55.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:55.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:56.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:56.129 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:56.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:56.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:56.154 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:14:56.162 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:14:56.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:56.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:56.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:56.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:56 smithi114 ceph-mon[103029]: pgmap v971: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 945 op/s; 8684/224262 objects degraded (3.872%); 7850/224262 objects misplaced (3.500%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T04:14:56.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:56 smithi114 ceph-mon[106969]: pgmap v971: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 945 op/s; 8684/224262 objects degraded (3.872%); 7850/224262 objects misplaced (3.500%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T04:14:56.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:56.403 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:56.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:56.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:56.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:56 smithi160 ceph-mon[88407]: pgmap v971: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 945 op/s; 8684/224262 objects degraded (3.872%); 7850/224262 objects misplaced (3.500%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T04:14:56.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:56.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:56.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:56.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:56.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:56.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:56.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:56.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:56.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:56.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:56.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:56.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:56.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:57.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:57.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:57.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:57.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:57.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:57.221 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:57.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:57.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:57.323 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:14:57.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:57.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:14:57.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:57.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:57.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:57.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:57.627 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:57.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:57.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:57.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:14:57.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:57.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:57.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:57.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:57.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:57.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:58.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:58.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:58.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:58.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:14:58.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:58.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:14:58.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:58.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:14:58 smithi114 ceph-mon[106969]: pgmap v972: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 8893/221904 objects degraded (4.008%); 7536/221904 objects misplaced (3.396%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:14:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:14:58 smithi114 ceph-mon[103029]: pgmap v972: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 8893/221904 objects degraded (4.008%); 7536/221904 objects misplaced (3.396%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:14:58.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:58.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:58.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:14:58 smithi160 ceph-mon[88407]: pgmap v972: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 944 op/s; 8893/221904 objects degraded (4.008%); 7536/221904 objects misplaced (3.396%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:14:58.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:58.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:58.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:58.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:58.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:58.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:58.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:14:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:58.790 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:58.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:14:58.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:14:58.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:58.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:14:59.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:59.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:59.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:59.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:59.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:59.202 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:59.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:59.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:14:59.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:59.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:14:59.396 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:14:59.396 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:14:59.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:59.449 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:14:59.450 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:14:59.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:59.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:59.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:59.484 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:14:59.485 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:14:59.522 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:14:59.522 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:14:59.552 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:14:59.552 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:14:59.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:14:59.584 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:14:59.584 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:14:59.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:59.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:14:59.618 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:14:59.618 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:14:59.651 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:14:59.651 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:14:59.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:14:59.745 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:14:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:59.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:14:59.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:14:59.881 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:14:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:14:59.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:14:59.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:00.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:00.020 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:00.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:00.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:00.154 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:00.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:00.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:00.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:00.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:00.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:00 smithi114 ceph-mon[106969]: pgmap v973: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 8893/214062 objects degraded (4.154%); 7536/214062 objects misplaced (3.520%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:15:00.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2881449239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:00 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8893/214062 objects degraded (4.154%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:00 smithi114 ceph-mon[103029]: pgmap v973: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 8893/214062 objects degraded (4.154%); 7536/214062 objects misplaced (3.520%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:15:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2881449239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:00 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8893/214062 objects degraded (4.154%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:00.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:15:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:15:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:15:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:00 smithi160 ceph-mon[88407]: pgmap v973: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 8893/214062 objects degraded (4.154%); 7536/214062 objects misplaced (3.520%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:15:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2881449239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:00 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8893/214062 objects degraded (4.154%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:00.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:00.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:00.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:00.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:00.564 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:00.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:00.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:00.693 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:00.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:00.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:00.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:00.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:00.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:00.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:00.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:01.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:01.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:01.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:01.156 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:15:01.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-14T04:15:01.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:01.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:01.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:01.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:01.390 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:01.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:01.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:01.512 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:01.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:01.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:01.634 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:01.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:01.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:01.775 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:01.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:01.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:15:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:15:01] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-14T04:15:01.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:01.906 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:01.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:01.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:02.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:02.051 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:02.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:02.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:02.183 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:02.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:02.250 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:02.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:02.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:02.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:02.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:02 smithi114 ceph-mon[103029]: pgmap v974: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 8893/209901 objects degraded (4.237%); 7536/209901 objects misplaced (3.590%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:15:02.346 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:02 smithi114 ceph-mon[106969]: pgmap v974: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 8893/209901 objects degraded (4.237%); 7536/209901 objects misplaced (3.590%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:15:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:02 smithi160 ceph-mon[88407]: pgmap v974: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 8893/209901 objects degraded (4.237%); 7536/209901 objects misplaced (3.590%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:15:02.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:02.463 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:02.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:02.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:02.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:02.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:02.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:02.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:02.727 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:02.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:02.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:02.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:02.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:02.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:03.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:03.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:03.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:03.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:03.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:03.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:03.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:03.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:03.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:03.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:03.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:03.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:03.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:03.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:03.609 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:15:03.609 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:15:03.609 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172828 10.f e216/199 10:fd430cc2:::benchmark_data_smithi114_188657_object52632:head v 216'10769, mlcod=216'10767)", 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.723005+0000", 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 267.88147379499998, 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32168082799999997, 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.610 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723005+0000", 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723005+0000", 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.611 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723005+0000", 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5099999999999999e-07 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723012+0000", 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8079999999999999e-06 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723023+0000", 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0505e-05 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723030+0000", 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2389999999999996e-06 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723091+0000", 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0639e-05 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.613 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723110+0000", 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9511e-05 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043884+0000", 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32077389299999998 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044685+0000", 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00080148199999999995 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.614 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172833 10.d e216/199 10:bca70c7d:::benchmark_data_smithi114_188657_object52637:head v 216'10819, mlcod=216'10817)", 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.723344+0000", 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 267.88113483000001, 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32066517999999999, 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.615 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723344+0000", 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723344+0000", 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723344+0000", 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8299999999999997e-07 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723349+0000", 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3370000000000001e-06 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723357+0000", 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0839999999999992e-06 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.617 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723362+0000", 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5999999999999997e-06 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724401+0000", 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010383079999999999 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724419+0000", 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8598e-05 2024-09-14T04:15:03.618 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.729936+0000", 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0055167020000000001 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044009+0000", 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31407296800000001 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.619 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172830 10.3 e216/199 10:c6093d96:::benchmark_data_smithi114_188657_object52634:head v 216'10738, mlcod=216'10736)", 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.724379+0000", 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 267.880098987, 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32095316000000002, 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724379+0000", 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.620 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724379+0000", 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724380+0000", 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7000000000000003e-07 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724387+0000", 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1360000000000002e-06 2024-09-14T04:15:03.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724397+0000", 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0541e-05 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724403+0000", 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2979999999999997e-06 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724457+0000", 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3940000000000002e-05 2024-09-14T04:15:03.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724475+0000", 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.738e-05 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043920+0000", 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31944481600000002 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.045333+0000", 2024-09-14T04:15:03.623 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001413079 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172832 10.f e216/199 10:f5ac3d97:::benchmark_data_smithi114_188657_object52636:head v 216'10770, mlcod=216'10767)", 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.726063+0000", 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 267.87841524100003, 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32223997999999998, 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726063+0000", 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726063+0000", 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.625 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726064+0000", 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.031e-06 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726070+0000", 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2570000000000004e-06 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726082+0000", 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1039000000000001e-05 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.626 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726088+0000", 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8020000000000003e-06 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.727200+0000", 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001112154 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.727219+0000", 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8822e-05 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.627 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.048262+0000", 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32104314299999998 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.048303+0000", 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0732000000000002e-05 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172838 10.5 e216/199 10:aaa76445:::benchmark_data_smithi114_188657_object52642:head v 216'10852, mlcod=216'10850)", 2024-09-14T04:15:03.628 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.733532+0000", 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 267.87094673799999, 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31986189300000001, 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.733532+0000", 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.733532+0000", 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.733533+0000", 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.003e-06 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.733540+0000", 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3150000000000003e-06 2024-09-14T04:15:03.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043891+0000", 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31035127000000001 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043905+0000", 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4028e-05 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044699+0000", 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00079361500000000001 2024-09-14T04:15:03.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044722+0000", 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2858999999999999e-05 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.053357+0000", 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0086351780000000003 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.632 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.053394+0000", 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6625000000000003e-05 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172841 10.4 e216/199 10:225f533d:::benchmark_data_smithi114_188657_object52645:head v 216'10785, mlcod=216'10783)", 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.738220+0000", 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 267.86625803300001, 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31266361500000001, 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.633 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.738220+0000", 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.738220+0000", 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.738222+0000", 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.235e-06 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.738230+0000", 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8519999999999993e-06 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043948+0000", 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.305717303 2024-09-14T04:15:03.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043957+0000", 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9199999999999993e-06 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044068+0000", 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111172 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.636 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044092+0000", 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3966999999999999e-05 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.050850+0000", 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0067584020000000002 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.050884+0000", 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3763999999999997e-05 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.637 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172845 10.7 e216/199 10:e36bbc21:::benchmark_data_smithi114_188657_object52649:head v 216'10964, mlcod=216'10962)", 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.743159+0000", 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 267.86131906700001, 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30543414600000002, 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.638 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.743159+0000", 2024-09-14T04:15:03.639 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.639 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.639 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.639 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.743159+0000", 2024-09-14T04:15:03.639 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.743161+0000", 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3400000000000001e-06 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.743169+0000", 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1389999999999995e-06 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.640 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044425+0000", 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30125647100000003 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044434+0000", 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9649999999999997e-06 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044713+0000", 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00027863999999999997 2024-09-14T04:15:03.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044733+0000", 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9969000000000001e-05 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.048453+0000", 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003720498 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.048594+0000", 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014012399999999999 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:36239 11.b 11:d0ec9894:::benchmark_data_smithi114_223345_object36238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:12:58.810718+0000", 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 124.793760593, 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.166823939, 2024-09-14T04:15:03.643 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 36239 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.810718+0000", 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.810718+0000", 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.810718+0000", 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.82e-07 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.810764+0000", 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5711000000000003e-05 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.810771+0000", 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8859999999999998e-06 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.810776+0000", 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9819999999999999e-06 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.811394+0000", 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00061839000000000002 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.811499+0000", 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104236 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.811619+0000", 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000120578 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:15:03.647 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.957508+0000", 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.145889144 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.957999+0000", 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00049081099999999996 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.977426+0000", 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.019426901999999999 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.977493+0000", 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6958000000000005e-05 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:12:58.977542+0000", 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8859000000000001e-05 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:76120 11.b 11:da15d4af:::benchmark_data_smithi114_223345_object76119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:15:03.649 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:32.429567+0000", 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 91.174910968000006, 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.224167059, 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 76120 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.650 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.429567+0000", 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.429567+0000", 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.429569+0000", 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.699e-06 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.429593+0000", 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.353e-05 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.429605+0000", 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2447e-05 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.429613+0000", 2024-09-14T04:15:03.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4229999999999999e-06 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.429804+0000", 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019131399999999999 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.429964+0000", 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000159989 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:03.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.430107+0000", 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014331599999999999 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.432880+0000", 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027723309999999998 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.434062+0000", 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011823420000000001 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.654 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.653617+0000", 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.219554896 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.653681+0000", 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4708000000000005e-05 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.653735+0000", 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3063999999999999e-05 2024-09-14T04:15:03.655 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:76134 11.9 11:9003b689:::benchmark_data_smithi114_223345_object76133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:32.444443+0000", 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 91.160035921000002, 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.213074177, 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:15:03.656 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 76134 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444443+0000", 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444443+0000", 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444444+0000", 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7209999999999999e-06 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444467+0000", 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2379e-05 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444484+0000", 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7490999999999999e-05 2024-09-14T04:15:03.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444494+0000", 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5130000000000002e-06 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444557+0000", 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3403000000000003e-05 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444712+0000", 2024-09-14T04:15:03.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015542900000000001 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444820+0000", 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107896 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.446754+0000", 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019335159999999999 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:15:03.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.448433+0000", 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001678676 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.657426+0000", 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.20899303499999999 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.657478+0000", 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2806000000000002e-05 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.661 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.657517+0000", 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8312000000000002e-05 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:76136 11.b 11:d45dbba8:::benchmark_data_smithi114_223345_object76135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:32.444806+0000", 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 91.159672834999995, 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.21279677, 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.662 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 76136 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444806+0000", 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444806+0000", 2024-09-14T04:15:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444807+0000", 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6800000000000009e-07 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444886+0000", 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9600999999999999e-05 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444897+0000", 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1129e-05 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444904+0000", 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6930000000000001e-06 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.444963+0000", 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8984999999999999e-05 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.665 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.445114+0000", 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015134100000000001 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.445282+0000", 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016752499999999999 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.448469+0000", 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031872469999999998 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.448843+0000", 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000374321 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.657502+0000", 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.20865895800000001 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.657566+0000", 2024-09-14T04:15:03.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3892000000000004e-05 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:32.657602+0000", 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6109999999999998e-05 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:119743 11.b 11:d95664ec:::benchmark_data_smithi114_223345_object7806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:21.745782+0000", 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 41.858696473000002, 2024-09-14T04:15:03.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.17826436200000001, 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119743 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.745782+0000", 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.745782+0000", 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.745784+0000", 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.04e-06 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.745790+0000", 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3550000000000001e-06 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.745805+0000", 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4853e-05 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.745820+0000", 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4382000000000001e-05 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.745885+0000", 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5815000000000006e-05 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.746002+0000", 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000116588 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.746111+0000", 2024-09-14T04:15:03.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000108556 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.750167+0000", 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0040562280000000003 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.752049+0000", 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018818509999999999 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923951+0000", 2024-09-14T04:15:03.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.17190228499999999 2024-09-14T04:15:03.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.924003+0000", 2024-09-14T04:15:03.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2275999999999999e-05 2024-09-14T04:15:03.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.924046+0000", 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3133000000000002e-05 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:119744 11.0 e260/243 11:01e15f57:::benchmark_data_smithi114_223345_object7807:head v 260'5559, mlcod=260'5557)", 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:21.747528+0000", 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 41.856950738000002, 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.17642370499999999, 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.675 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.747528+0000", 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.747528+0000", 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.747529+0000", 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9300000000000006e-07 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.747535+0000", 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4679999999999997e-06 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.747547+0000", 2024-09-14T04:15:03.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1708e-05 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.747554+0000", 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7129999999999999e-06 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.748471+0000", 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00091734 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.748490+0000", 2024-09-14T04:15:03.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8702000000000002e-05 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923890+0000", 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.175400216 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923951+0000", 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1564999999999998e-05 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.679 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:119748 11.a 11:5e01be32:::benchmark_data_smithi114_223345_object7811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:21.749661+0000", 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 41.854816993, 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.18126213199999999, 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119748 2024-09-14T04:15:03.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.749661+0000", 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.749661+0000", 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:03.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.749663+0000", 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.46e-06 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.749668+0000", 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7119999999999999e-06 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.749680+0000", 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2532e-05 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.749687+0000", 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2570000000000001e-06 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.749744+0000", 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6110999999999999e-05 2024-09-14T04:15:03.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.749851+0000", 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107939 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.749950+0000", 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8067999999999997e-05 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:15:03.685 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.926593+0000", 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.17664359499999999 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.928928+0000", 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002335312 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.930731+0000", 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018029770000000001 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.930879+0000", 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014771599999999999 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.930924+0000", 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4453000000000003e-05 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.687 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.36273.0:119743 11.b e260/235 ondisk, result = 0)", 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:21.752523+0000", 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 41.851955822000001, 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.171528867, 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.752523+0000", 2024-09-14T04:15:03.688 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.752523+0000", 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.752523+0000", 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1900000000000001e-07 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.752528+0000", 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1160000000000001e-06 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.752538+0000", 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0710000000000001e-05 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923873+0000", 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.17133441899999999 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923934+0000", 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1764e-05 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923944+0000", 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7149999999999993e-06 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.924051+0000", 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107324 2024-09-14T04:15:03.691 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.36273.0:119748 11.a e260/235 ondisk, result = 0)", 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:21.754007+0000", 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 41.850471016, 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.176857187, 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.754007+0000", 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.754007+0000", 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.754008+0000", 2024-09-14T04:15:03.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1100000000000004e-07 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.754014+0000", 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2920000000000003e-06 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923936+0000", 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.16992215999999999 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923943+0000", 2024-09-14T04:15:03.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9829999999999999e-06 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.928915+0000", 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0049719400000000002 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.928922+0000", 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4989999999999997e-06 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.930865+0000", 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001942402 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:119753 11.2 e260/243 11:41fc4d0e:::benchmark_data_smithi114_223345_object7816:head v 260'5643, mlcod=260'5641)", 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:21.755356+0000", 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 41.849122712000003, 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.17352646599999999, 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.696 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.755356+0000", 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.755356+0000", 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.755357+0000", 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0699999999999999e-06 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.755364+0000", 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8229999999999997e-06 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923892+0000", 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.16852832200000001 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923903+0000", 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0808000000000001e-05 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.923975+0000", 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2590999999999997e-05 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.924004+0000", 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8401000000000001e-05 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.928786+0000", 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0047818779999999998 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:21.928882+0000", 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6572999999999998e-05 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.700 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:123299 11.b 11:ddf171f5:::benchmark_data_smithi114_223345_object11362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.362147+0000", 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 38.242331270000001, 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.28379990399999999, 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:15:03.701 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123299 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362147+0000", 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362147+0000", 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362149+0000", 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.883e-06 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362155+0000", 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9869999999999996e-06 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362170+0000", 2024-09-14T04:15:03.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5221999999999999e-05 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362178+0000", 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2300000000000008e-06 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362292+0000", 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113782 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362418+0000", 2024-09-14T04:15:03.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000126143 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.362543+0000", 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012447700000000001 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.366977+0000", 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044344850000000002 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.371182+0000", 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042044480000000004 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.645855+0000", 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.27467320499999998 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.645906+0000", 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1122999999999998e-05 2024-09-14T04:15:03.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.645947+0000", 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0918999999999999e-05 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:123317 11.b 11:d165ac94:::benchmark_data_smithi114_223345_object11380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.385232+0000", 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 38.219245952999998, 2024-09-14T04:15:03.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26265905299999998, 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123317 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385232+0000", 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385232+0000", 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385233+0000", 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4299999999999998e-07 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385235+0000", 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4930000000000001e-06 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385243+0000", 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4800000000000004e-06 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385248+0000", 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5149999999999997e-06 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385335+0000", 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6181999999999996e-05 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385459+0000", 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012396600000000001 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385572+0000", 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011335 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.391607+0000", 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0060352610000000001 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.392474+0000", 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00086647000000000004 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.647797+0000", 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.25532357100000003 2024-09-14T04:15:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.647849+0000", 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1994999999999999e-05 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.647892+0000", 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2327000000000003e-05 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:123318 11.b 11:d7575392:::benchmark_data_smithi114_223345_object11381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.385351+0000", 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 38.219127489000002, 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.262686219, 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123318 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385351+0000", 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385351+0000", 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385352+0000", 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2500000000000005e-07 2024-09-14T04:15:03.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385355+0000", 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4429999999999999e-06 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385362+0000", 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.058e-06 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:15:03.716 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.385368+0000", 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9800000000000003e-06 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.388571+0000", 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003203165 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.388662+0000", 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0741000000000004e-05 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.388735+0000", 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2967999999999997e-05 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.391779+0000", 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030441439999999999 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.392713+0000", 2024-09-14T04:15:03.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00093342000000000002 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.647928+0000", 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.25521566600000001 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.647986+0000", 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8145e-05 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.648037+0000", 2024-09-14T04:15:03.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0763999999999997e-05 2024-09-14T04:15:03.720 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.720 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.720 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.720 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:15:03.720 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:15:03.720 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:15:03.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:03.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:03.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:03.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:03.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:03.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:03.986 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:04.025 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:04.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:04.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:04.161 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:04.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:04.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:04.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:15:04.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:04.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:04.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:04.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:04 smithi114 ceph-mon[103029]: pgmap v975: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 9118/207501 objects degraded (4.394%); 7211/207501 objects misplaced (3.475%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:15:04.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:04 smithi114 ceph-mon[106969]: pgmap v975: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 9118/207501 objects degraded (4.394%); 7211/207501 objects misplaced (3.475%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:15:04.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:04 smithi160 ceph-mon[88407]: pgmap v975: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 9118/207501 objects degraded (4.394%); 7211/207501 objects misplaced (3.475%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:15:04.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:04.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:04.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:04.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:04.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:04.638 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:04.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:04.653 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:04.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:04.767 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:04.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:04.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:04.906 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:04.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:05.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:05.045 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:05.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:05.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:05.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:05.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:05.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:05.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-14T04:15:05.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:05.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:05.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:05 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9118/207501 objects degraded (4.394%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:05.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:05.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:05 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9118/207501 objects degraded (4.394%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:05.432 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:05 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9118/207501 objects degraded (4.394%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:05.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:05.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:05.553 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:05.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:05.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:05.705 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:05.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:05.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:05.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:05.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:05.890 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:05.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:05.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:05.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:06.027 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:06.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:06.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:06.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:06.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:06.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:06.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:06.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:06.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:06 smithi114 ceph-mon[106969]: pgmap v976: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 9118/196581 objects degraded (4.638%); 7211/196581 objects misplaced (3.668%); 1.2 MiB/s, 18 objects/s recovering 2024-09-14T04:15:06.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:06 smithi114 ceph-mon[103029]: pgmap v976: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 9118/196581 objects degraded (4.638%); 7211/196581 objects misplaced (3.668%); 1.2 MiB/s, 18 objects/s recovering 2024-09-14T04:15:06.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:06.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:06 smithi160 ceph-mon[88407]: pgmap v976: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 9118/196581 objects degraded (4.638%); 7211/196581 objects misplaced (3.668%); 1.2 MiB/s, 18 objects/s recovering 2024-09-14T04:15:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:06.436 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:06.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:06.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:06.545 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:06.600 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:06.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:06.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:06.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:06.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:06.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:06.875 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:06.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:06.976 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:07.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:07.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:07.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:07.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:07.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:07.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:07.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:07.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:07.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:07.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:07.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:07.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:07.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:07.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:07.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:07.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:07.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:07.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:07.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:07.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:07.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:07.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:07.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:08.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:08.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:08.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:08.158 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:15:08.163 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:08 smithi114 ceph-mon[103029]: pgmap v977: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 9331/194376 objects degraded (4.800%); 6901/194376 objects misplaced (3.550%); 1.6 MiB/s, 25 objects/s recovering 2024-09-14T04:15:08.163 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:08.163 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1475867039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:15:08.163 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:08 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:15:08.163 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:08 smithi114 ceph-mon[106969]: pgmap v977: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 9331/194376 objects degraded (4.800%); 6901/194376 objects misplaced (3.550%); 1.6 MiB/s, 25 objects/s recovering 2024-09-14T04:15:08.163 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:08.163 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1475867039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:15:08.163 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:08 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:15:08.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:08.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:08.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:08.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:08.369 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:08 smithi160 ceph-mon[88407]: pgmap v977: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 829 op/s; 9331/194376 objects degraded (4.800%); 6901/194376 objects misplaced (3.550%); 1.6 MiB/s, 25 objects/s recovering 2024-09-14T04:15:08.369 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:08.370 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1475867039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:15:08.370 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:08 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:15:08.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:08.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:08.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:08.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:08.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:08.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:08.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:08.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:08.666 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:08.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:08.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:08.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:08.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:08.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:08.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:08.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:08.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:09.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:09.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:09.074 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:09.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:09.114 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:15:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:15:09.114 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-14T04:15:09.114 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:15:09.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:09.210 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:09.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:09.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:09.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:09.352 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:09.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:09.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:09 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:15:09.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:15:09.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:09 smithi114 ceph-mon[106969]: osdmap e264: 8 total, 8 up, 8 in 2024-09-14T04:15:09.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:09 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:15:09.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:15:09.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:09 smithi114 ceph-mon[103029]: osdmap e264: 8 total, 8 up, 8 in 2024-09-14T04:15:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:09 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:15:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:15:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:09 smithi160 ceph-mon[88407]: osdmap e264: 8 total, 8 up, 8 in 2024-09-14T04:15:09.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:09.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:09.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:09.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:09.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:09.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:09.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:09.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:09.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-14T04:15:09.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:09.894 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:09.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:09.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:10.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:10.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:10.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:10.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:10.188 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:10.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:10.207 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:10.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:10.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:10.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:10.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:10.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:10 smithi160 ceph-mon[88407]: pgmap v979: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 9331/186405 objects degraded (5.006%); 6901/186405 objects misplaced (3.702%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T04:15:10.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:10 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9331/186405 objects degraded (5.006%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:10.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:15:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:15:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:15:10.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:10.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:10.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:10.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:10.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:10.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:10.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:10.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:10 smithi114 ceph-mon[106969]: pgmap v979: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 9331/186405 objects degraded (5.006%); 6901/186405 objects misplaced (3.702%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T04:15:10.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:10 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9331/186405 objects degraded (5.006%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:10 smithi114 ceph-mon[103029]: pgmap v979: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 9331/186405 objects degraded (5.006%); 6901/186405 objects misplaced (3.702%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T04:15:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:10 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9331/186405 objects degraded (5.006%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:10.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:10.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:10.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:10.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:10.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:10.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:10.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:10.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:11.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:11.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:11.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:11.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:11.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:11.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:11.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:11.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:11.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:11.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:11.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:11.407 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:11.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:11.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:11.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:11.545 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:11.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:11.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:11.689 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:11.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:11.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:11.816 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:15:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:15:11] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-14T04:15:11.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:11.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:11.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:11.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:11.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:11.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:12.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:12.071 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:12.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:12.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:12.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:12.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:12.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:12.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:12.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:12 smithi160 ceph-mon[88407]: pgmap v980: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 9331/182451 objects degraded (5.114%); 6901/182451 objects misplaced (3.782%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T04:15:12.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:12.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:12.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:12.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:12.604 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:12.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:12.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:12.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:12 smithi114 ceph-mon[103029]: pgmap v980: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 9331/182451 objects degraded (5.114%); 6901/182451 objects misplaced (3.782%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T04:15:12.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:12 smithi114 ceph-mon[106969]: pgmap v980: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 914 op/s; 9331/182451 objects degraded (5.114%); 6901/182451 objects misplaced (3.782%); 1.2 MiB/s, 19 objects/s recovering 2024-09-14T04:15:12.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:12.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:12.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:12.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:12.875 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:12.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:12.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:13.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:13.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:13.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:13.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:13.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:13.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:13.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:13.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:13.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:13.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:13.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:13.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:13.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:13.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:13.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:13.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:13.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:13.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:13.733 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:15:13.733 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:15:13.734 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:15:13.734 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:15:13.734 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:15:13.817 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:13.846 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:13.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:13.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:13.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:13.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:13.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:14.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:14.121 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:14.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:14.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:14.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:14.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:14.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:14.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:14.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:14.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:14 smithi160 ceph-mon[88407]: pgmap v981: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 9487/180270 objects degraded (5.263%); 6609/180270 objects misplaced (3.666%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:15:14.419 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:14 smithi114 ceph-mon[106969]: pgmap v981: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 9487/180270 objects degraded (5.263%); 6609/180270 objects misplaced (3.666%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:15:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:14 smithi114 ceph-mon[103029]: pgmap v981: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 9487/180270 objects degraded (5.263%); 6609/180270 objects misplaced (3.666%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:15:14.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-14T04:15:14.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:14.536 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:14.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:14.638 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:14.668 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:14.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:14.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:14.809 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:14.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:14.873 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:14.909 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:14.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:14.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:15.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:15.086 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:15.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:15.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:15.196 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-14T04:15:15.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:15.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:15.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:15.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:15.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:15.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:15.363 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:15 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9487/180270 objects degraded (5.263%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:15.363 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3301686859' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:15:15.363 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:15 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:15:15.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:15 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9487/180270 objects degraded (5.263%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:15.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3301686859' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:15:15.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:15 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:15:15.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:15 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9487/180270 objects degraded (5.263%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:15.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3301686859' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:15:15.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:15 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:15:15.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:15.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:15.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:15.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:15.632 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:15.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:15.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:15.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:15.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:15.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:15.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:15.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:15.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:16.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:16.056 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:16.057 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:15:16.057 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:15:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:16.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:16.101 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:15:16.101 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:15:16.139 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:15:16.139 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:15:16.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:16.192 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:15:16.192 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:15:16.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:16.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:16.220 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:15:16.221 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:15:16.253 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:15:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:15:16.283 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:15:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:15:16.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:16.317 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:15:16.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:16 smithi114 ceph-mon[103029]: pgmap v982: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 917 op/s; 9487/169041 objects degraded (5.612%); 6609/169041 objects misplaced (3.910%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:15:16.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:16 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:15:16.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:16 smithi114 ceph-mon[103029]: osdmap e265: 8 total, 8 up, 8 in 2024-09-14T04:15:16.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:16 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/500633525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:16.319 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:16 smithi114 ceph-mon[106969]: pgmap v982: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 917 op/s; 9487/169041 objects degraded (5.612%); 6609/169041 objects misplaced (3.910%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:15:16.319 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:16 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:15:16.319 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:16 smithi114 ceph-mon[106969]: osdmap e265: 8 total, 8 up, 8 in 2024-09-14T04:15:16.319 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:16 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/500633525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:16.319 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:15:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:16.337 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:16.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:16.341 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:15:16.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:16.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:16.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:16.568 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:16 smithi160 ceph-mon[88407]: pgmap v982: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 917 op/s; 9487/169041 objects degraded (5.612%); 6609/169041 objects misplaced (3.910%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:15:16.568 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:16 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:15:16.568 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:16 smithi160 ceph-mon[88407]: osdmap e265: 8 total, 8 up, 8 in 2024-09-14T04:15:16.568 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:16 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/500633525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:16.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:16.628 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:16.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:16.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:16.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:16.767 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:16.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:16.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-14T04:15:16.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:16.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:16.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:17.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:17.024 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:17.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:17.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:17.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:17.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:17.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:17.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:17.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:17.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:17.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:17.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:17.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:17.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:17.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:17.562 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:17.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:17.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:17.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:17.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:17.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:17.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:17.829 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:15:17.829 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:15:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:17.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:17.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:17.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:17.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:18.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:18.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:18.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:18.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:18.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:18.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:18.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:18.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:18.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:18.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:18.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:18.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:18.484 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:18 smithi160 ceph-mon[88407]: pgmap v984: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 9586/166869 objects degraded (5.745%); 6360/166869 objects misplaced (3.811%); 2.0 MiB/s, 32 objects/s recovering 2024-09-14T04:15:18.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:18.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:18.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:18.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-14T04:15:18.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:18.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:18 smithi114 ceph-mon[103029]: pgmap v984: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 9586/166869 objects degraded (5.745%); 6360/166869 objects misplaced (3.811%); 2.0 MiB/s, 32 objects/s recovering 2024-09-14T04:15:18.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:18 smithi114 ceph-mon[106969]: pgmap v984: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 9586/166869 objects degraded (5.745%); 6360/166869 objects misplaced (3.811%); 2.0 MiB/s, 32 objects/s recovering 2024-09-14T04:15:18.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:18.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:18.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:18.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:18.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:18.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:18.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:18.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:18.946 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:19.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:19.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:19.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:19.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:19.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:19.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:19.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:19.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:19.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:19 smithi160 ceph-mon[88407]: 11.11 scrub starts 2024-09-14T04:15:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:19 smithi160 ceph-mon[88407]: 11.11 scrub ok 2024-09-14T04:15:19.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:19.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:19.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:19.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:19.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:19.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:19.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:19 smithi114 ceph-mon[103029]: 11.11 scrub starts 2024-09-14T04:15:19.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:19 smithi114 ceph-mon[103029]: 11.11 scrub ok 2024-09-14T04:15:19.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:19 smithi114 ceph-mon[106969]: 11.11 scrub starts 2024-09-14T04:15:19.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:19 smithi114 ceph-mon[106969]: 11.11 scrub ok 2024-09-14T04:15:19.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:19.705 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:19.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:19.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:19.729 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:19.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:19.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:19.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:19.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:19.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:20.000 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:20.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:20.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:20.138 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:20.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:20.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:20.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:20.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:20.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:20 smithi114 ceph-mon[103029]: pgmap v985: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 9586/159663 objects degraded (6.004%); 6360/159663 objects misplaced (3.983%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:15:20.301 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:20 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9586/159663 objects degraded (6.004%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:20.301 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:20 smithi114 ceph-mon[106969]: pgmap v985: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 9586/159663 objects degraded (6.004%); 6360/159663 objects misplaced (3.983%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:15:20.301 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:20 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9586/159663 objects degraded (6.004%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:20.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:20.429 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:20.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:20.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:20.480 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:15:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:15:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:15:20.480 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:20 smithi160 ceph-mon[88407]: pgmap v985: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 9586/159663 objects degraded (6.004%); 6360/159663 objects misplaced (3.983%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:15:20.480 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:20 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9586/159663 objects degraded (6.004%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:20.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:20.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:20.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:20.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:20.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:20.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:20.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:20.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:20.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:20.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:20.976 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:15:20.976 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:15:20.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:20.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:21.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:21.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:21.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:21.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:21.281 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:21.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:21.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:21.432 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:21.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:21.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:21.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:21.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:21.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:15:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:15:21] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-14T04:15:21.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:21.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:21.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:21.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:15:21.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-14T04:15:21.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:21.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:21.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:21.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:22.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:22.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:22.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:22.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:22.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:22.215 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:22.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:22.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:22.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:22.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:22.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:22.445 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:22.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:22.468 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:22 smithi114 ceph-mon[106969]: pgmap v986: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 875 op/s; 9586/156189 objects degraded (6.137%); 6360/156189 objects misplaced (4.072%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:15:22.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:22 smithi114 ceph-mon[103029]: pgmap v986: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 875 op/s; 9586/156189 objects degraded (6.137%); 6360/156189 objects misplaced (4.072%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:15:22.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:22.590 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:22.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:22 smithi160 ceph-mon[88407]: pgmap v986: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 875 op/s; 9586/156189 objects degraded (6.137%); 6360/156189 objects misplaced (4.072%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:15:22.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:22.713 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:22.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:22.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:22.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:22.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:22.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:22.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:22.992 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:22.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:23.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:23.135 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:23.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:23.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:23.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:23.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:23.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:23.400 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:23.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:23.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:23.530 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:23.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:23.609 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:23.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:23.659 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:15:23.660 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:23.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:23.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:23.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:23.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:23.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:23.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:23.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:23 smithi114 ceph-mon[103029]: pgmap v987: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 323 op/s; 9716/154266 objects degraded (6.298%); 6095/154266 objects misplaced (3.951%) 2024-09-14T04:15:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/641726772' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:15:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:23 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:15:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:23 smithi114 ceph-mon[106969]: pgmap v987: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 323 op/s; 9716/154266 objects degraded (6.298%); 6095/154266 objects misplaced (3.951%) 2024-09-14T04:15:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:23 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/641726772' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:15:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:23 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:15:23.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:23 smithi160 ceph-mon[88407]: pgmap v987: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 323 op/s; 9716/154266 objects degraded (6.298%); 6095/154266 objects misplaced (3.951%) 2024-09-14T04:15:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:23 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/641726772' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:15:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:23 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:15:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:23.940 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:23.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:24.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:24.079 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:24.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:24.181 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:24.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:24.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:24.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:24.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:24.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:24.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:24.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:24.488 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:24.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:24.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:24.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:24.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:24.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:24.733 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:15:24.733 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:15:24.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:24.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:24.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:24.900 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:24.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:24 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:15:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:24 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:15:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:24 smithi160 ceph-mon[88407]: osdmap e266: 8 total, 8 up, 8 in 2024-09-14T04:15:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9716/154266 objects degraded (6.298%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:24.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:24 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:15:24.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:24 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:15:24.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:24 smithi114 ceph-mon[103029]: osdmap e266: 8 total, 8 up, 8 in 2024-09-14T04:15:24.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9716/154266 objects degraded (6.298%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:24.944 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:24 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:15:24.944 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:24 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:15:24.944 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:24 smithi114 ceph-mon[106969]: osdmap e266: 8 total, 8 up, 8 in 2024-09-14T04:15:24.945 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9716/154266 objects degraded (6.298%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:25.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:25.041 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:25.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:25.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:25.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:25.184 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:25.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:25.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:25.326 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:25.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:25.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:25.456 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:25.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:25.558 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:25.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-14T04:15:25.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:25.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:25.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:25.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:25.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:25.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:25.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:25 smithi114 ceph-mon[106969]: pgmap v989: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 340 op/s; 9716/142785 objects degraded (6.805%); 6095/142785 objects misplaced (4.269%) 2024-09-14T04:15:25.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:25 smithi114 ceph-mon[103029]: pgmap v989: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 340 op/s; 9716/142785 objects degraded (6.805%); 6095/142785 objects misplaced (4.269%) 2024-09-14T04:15:25.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:25.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:25.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:25 smithi160 ceph-mon[88407]: pgmap v989: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 340 op/s; 9716/142785 objects degraded (6.805%); 6095/142785 objects misplaced (4.269%) 2024-09-14T04:15:25.990 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:25.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:26.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:26.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:26.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:26.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:26.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:26.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:26.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:26.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:26.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:26.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:26.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:26.457 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:26.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:26.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:26.597 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:26.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:26.693 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-14T04:15:26.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:26.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:26.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:26.825 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:26 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/392442087' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:15:26.825 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:26 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:15:26.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:26 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/392442087' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:15:26.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:26 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:15:26.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:26.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:26.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:26 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/392442087' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:15:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:26 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:15:26.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:26.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:15:27.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:27.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:27.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:27.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:27.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:27.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:27.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:27.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:27.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:27.281 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:27.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:27.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:27.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:27.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:27.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:27.559 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:27.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:27.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:27.703 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:27.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:27.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:27.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:27.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:27.938 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:27.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:27.961 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:27.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:28.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:28.101 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:28.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:28.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:27 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:15:28.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:27 smithi114 ceph-mon[103029]: osdmap e267: 8 total, 8 up, 8 in 2024-09-14T04:15:28.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:27 smithi114 ceph-mon[103029]: pgmap v991: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 104 op/s; 9821/140856 objects degraded (6.972%); 5842/140856 objects misplaced (4.147%) 2024-09-14T04:15:28.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:27 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1723649874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:28.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:27 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:15:28.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:27 smithi114 ceph-mon[106969]: osdmap e267: 8 total, 8 up, 8 in 2024-09-14T04:15:28.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:27 smithi114 ceph-mon[106969]: pgmap v991: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 104 op/s; 9821/140856 objects degraded (6.972%); 5842/140856 objects misplaced (4.147%) 2024-09-14T04:15:28.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:27 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1723649874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:27 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:15:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:27 smithi160 ceph-mon[88407]: osdmap e267: 8 total, 8 up, 8 in 2024-09-14T04:15:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:27 smithi160 ceph-mon[88407]: pgmap v991: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 104 op/s; 9821/140856 objects degraded (6.972%); 5842/140856 objects misplaced (4.147%) 2024-09-14T04:15:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:27 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1723649874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:28.183 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:15:28.183 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:15:28.183 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:15:28.184 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:15:28.185 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:15:28.185 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:15:28.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:28.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:28.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:28.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:28.353 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:28.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:28.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:28.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:28.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:28.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:28.586 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:15:28.586 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:36749 10.0 10:08747d5e:::benchmark_data_smithi114_188657_object36748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:08:18.554324+0000", 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 430.026676076, 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.26279089999999999, 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:15:28.587 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:15:28.588 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 36749 2024-09-14T04:15:28.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.588 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.554324+0000", 2024-09-14T04:15:28.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.554324+0000", 2024-09-14T04:15:28.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.554321+0000", 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999962 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.554351+0000", 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0326999999999999e-05 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.554360+0000", 2024-09-14T04:15:28.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4900000000000006e-06 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.554365+0000", 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329999999999997e-06 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.554418+0000", 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3372999999999999e-05 2024-09-14T04:15:28.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.554566+0000", 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014810999999999999 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.554688+0000", 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012160899999999999 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.556372+0000", 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0016841060000000001 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.557464+0000", 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001091854 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.817029+0000", 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.25956468799999999 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.817082+0000", 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3127000000000001e-05 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.817115+0000", 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3676000000000001e-05 2024-09-14T04:15:28.594 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:36774 10.b 10:d68d4afa:::benchmark_data_smithi114_188657_object36773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:08:18.571138+0000", 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 430.00986295899997, 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.245968728, 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:15:28.595 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 36774 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.571138+0000", 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.571138+0000", 2024-09-14T04:15:28.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.571137+0000", 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.571188+0000", 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1493000000000003e-05 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.571197+0000", 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8200000000000003e-06 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.571201+0000", 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7450000000000001e-06 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.571237+0000", 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5867000000000001e-05 2024-09-14T04:15:28.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.571313+0000", 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6718999999999995e-05 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.571394+0000", 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0390999999999997e-05 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.572475+0000", 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010812829999999999 2024-09-14T04:15:28.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.572773+0000", 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029824099999999998 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.817046+0000", 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24427216199999999 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.817084+0000", 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8362999999999998e-05 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:08:18.817106+0000", 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2391e-05 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:172828 10.f 10:fd430cc2:::benchmark_data_smithi114_188657_object52632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:10:35.722475+0000", 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 292.85852504299999, 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32259693299999997, 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172828 2024-09-14T04:15:28.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722475+0000", 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722475+0000", 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722475+0000", 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722480+0000", 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8060000000000002e-06 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722491+0000", 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1316e-05 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722497+0000", 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1319999999999999e-06 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722549+0000", 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2296999999999998e-05 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722685+0000", 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013533800000000001 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,3", 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722792+0000", 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000107123 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.728797+0000", 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0060054330000000001 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.734105+0000", 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0053078530000000004 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.044991+0000", 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31088628200000001 2024-09-14T04:15:28.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.045037+0000", 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5812000000000001e-05 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.045072+0000", 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5227e-05 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.608 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:172832 10.f 10:f5ac3d97:::benchmark_data_smithi114_188657_object52636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:10:35.723078+0000", 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 292.85792277100001, 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32550156499999999, 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:15:28.609 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172832 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723078+0000", 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723078+0000", 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723077+0000", 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723081+0000", 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9740000000000004e-06 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723089+0000", 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3969999999999994e-06 2024-09-14T04:15:28.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723094+0000", 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2780000000000004e-06 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.725834+0000", 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0027405300000000001 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.725909+0000", 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4869000000000004e-05 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,3", 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.725974+0000", 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5067999999999994e-05 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.728905+0000", 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029313120000000002 2024-09-14T04:15:28.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.734226+0000", 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.005320361 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.048503+0000", 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31427680400000002 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.048544+0000", 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1359000000000002e-05 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.048579+0000", 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5317000000000001e-05 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.615 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:172833 10.d 10:bca70c7d:::benchmark_data_smithi114_188657_object52637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:10:35.723100+0000", 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 292.85790069000001, 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.321905844, 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172833 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723100+0000", 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723100+0000", 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723100+0000", 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723101+0000", 2024-09-14T04:15:28.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.64e-06 2024-09-14T04:15:28.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723104+0000", 2024-09-14T04:15:28.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2449999999999998e-06 2024-09-14T04:15:28.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723107+0000", 2024-09-14T04:15:28.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2199999999999999e-06 2024-09-14T04:15:28.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723144+0000", 2024-09-14T04:15:28.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7110000000000002e-05 2024-09-14T04:15:28.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723217+0000", 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3182000000000005e-05 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723278+0000", 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1227000000000002e-05 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.726221+0000", 2024-09-14T04:15:28.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029430139999999999 2024-09-14T04:15:28.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.735307+0000", 2024-09-14T04:15:28.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090862159999999994 2024-09-14T04:15:28.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.044872+0000", 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30956473000000001 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.044953+0000", 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0872999999999996e-05 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.045006+0000", 2024-09-14T04:15:28.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2679000000000002e-05 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:172845 10.7 10:e36bbc21:::benchmark_data_smithi114_188657_object52649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:10:35.742910+0000", 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 292.83809027500001, 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30679366600000002, 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.624 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172845 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.742910+0000", 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.742910+0000", 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.742909+0000", 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.742914+0000", 2024-09-14T04:15:28.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.408e-06 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.742924+0000", 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0059e-05 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.742928+0000", 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0090000000000001e-06 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.742957+0000", 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9447000000000001e-05 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.743031+0000", 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3922000000000001e-05 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,5", 2024-09-14T04:15:28.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.743097+0000", 2024-09-14T04:15:28.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5429e-05 2024-09-14T04:15:28.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.746664+0000", 2024-09-14T04:15:28.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035670469999999998 2024-09-14T04:15:28.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.746810+0000", 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014583099999999999 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.049617+0000", 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30280760499999998 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.049664+0000", 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7191999999999998e-05 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.049704+0000", 2024-09-14T04:15:28.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9567999999999998e-05 2024-09-14T04:15:28.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.631 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.631 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:76131 11.1 11:849f9e98:::benchmark_data_smithi114_223345_object76130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:32.441855+0000", 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.139145531, 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.227301801, 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 76131 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.441855+0000", 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.441855+0000", 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.441854+0000", 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.441923+0000", 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9554999999999994e-05 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.441935+0000", 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1440999999999999e-05 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.441939+0000", 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0810000000000004e-06 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.441984+0000", 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5544999999999998e-05 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.442083+0000", 2024-09-14T04:15:28.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8344000000000006e-05 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.442181+0000", 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8302999999999997e-05 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.443510+0000", 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0013293389999999999 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.444239+0000", 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00072815900000000003 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.669103+0000", 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.224864218 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.669132+0000", 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9501000000000001e-05 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:32.669157+0000", 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4528999999999998e-05 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:119756 11.1 11:808ceec6:::benchmark_data_smithi114_223345_object7819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:21.758314+0000", 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 66.822686118999997, 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17033678599999999, 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119756 2024-09-14T04:15:28.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.758314+0000", 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.758314+0000", 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.758313+0000", 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.758320+0000", 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.528e-06 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.758332+0000", 2024-09-14T04:15:28.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.207e-05 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.758339+0000", 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4270000000000002e-06 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.758384+0000", 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5281000000000002e-05 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.758504+0000", 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119838 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.758604+0000", 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9592999999999993e-05 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.760701+0000", 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020977309999999998 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.760819+0000", 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011733799999999999 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.928561+0000", 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.167742164 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.928608+0000", 2024-09-14T04:15:28.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7227e-05 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:21.928651+0000", 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3013000000000003e-05 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:123305 11.3 11:c081c612:::benchmark_data_smithi114_223345_object11368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:25.368967+0000", 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 63.212033691999999, 2024-09-14T04:15:28.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27023498499999998, 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 123305 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.368967+0000", 2024-09-14T04:15:28.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.368967+0000", 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.368965+0000", 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.368972+0000", 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2870000000000001e-06 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.368985+0000", 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.311e-05 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.368991+0000", 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.976e-06 2024-09-14T04:15:28.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.369049+0000", 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8445e-05 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.369149+0000", 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9563999999999997e-05 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-14T04:15:28.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.369238+0000", 2024-09-14T04:15:28.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9382999999999994e-05 2024-09-14T04:15:28.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.371456+0000", 2024-09-14T04:15:28.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022180289999999998 2024-09-14T04:15:28.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.372053+0000", 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00059691099999999999 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.639139+0000", 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.26708627200000001 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.639174+0000", 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4950000000000002e-05 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.639202+0000", 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7647000000000001e-05 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:123308 11.13 11:cdce9a5b:::benchmark_data_smithi114_223345_object11371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:28.652 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:25.372186+0000", 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 63.208814984, 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27376857900000001, 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 123308 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.372186+0000", 2024-09-14T04:15:28.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.372186+0000", 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.372185+0000", 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.372190+0000", 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0089999999999998e-06 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.372198+0000", 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6449999999999994e-06 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.372203+0000", 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9760000000000003e-06 2024-09-14T04:15:28.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.372236+0000", 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2999e-05 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.372315+0000", 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9091999999999994e-05 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-14T04:15:28.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.372384+0000", 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9237000000000001e-05 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.375491+0000", 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031063229999999998 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.375974+0000", 2024-09-14T04:15:28.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048335300000000001 2024-09-14T04:15:28.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.645871+0000", 2024-09-14T04:15:28.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.26989665899999998 2024-09-14T04:15:28.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.645918+0000", 2024-09-14T04:15:28.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7018999999999998e-05 2024-09-14T04:15:28.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.645954+0000", 2024-09-14T04:15:28.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6281e-05 2024-09-14T04:15:28.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.659 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:123309 11.3 11:c637ab3c:::benchmark_data_smithi114_223345_object11372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:25.373415+0000", 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 63.207585440000003, 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27257550899999999, 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 123309 2024-09-14T04:15:28.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.373415+0000", 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.373415+0000", 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.373414+0000", 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.373420+0000", 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4990000000000002e-06 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.373434+0000", 2024-09-14T04:15:28.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4402e-05 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.373440+0000", 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6609999999999997e-06 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.373480+0000", 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9886999999999999e-05 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.373607+0000", 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000127582 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.373713+0000", 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105938 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.375626+0000", 2024-09-14T04:15:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001912574 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.376133+0000", 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00050784500000000004 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.645926+0000", 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.26979252500000001 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.645961+0000", 2024-09-14T04:15:28.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.455e-05 2024-09-14T04:15:28.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.645991+0000", 2024-09-14T04:15:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:28.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:28.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0059000000000001e-05 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:123314 11.13 11:c83b6f58:::benchmark_data_smithi114_223345_object11377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:25.376404+0000", 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 63.204596768000002, 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27285240599999999, 2024-09-14T04:15:28.669 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 123314 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.376404+0000", 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.376404+0000", 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.376403+0000", 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.376408+0000", 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1649999999999997e-06 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.376419+0000", 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1494999999999999e-05 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.376424+0000", 2024-09-14T04:15:28.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2490000000000003e-06 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.376455+0000", 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1019999999999998e-05 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.376536+0000", 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1434999999999995e-05 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.376641+0000", 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104905 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.378747+0000", 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021061610000000001 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.378962+0000", 2024-09-14T04:15:28.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000214589 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.649182+0000", 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27022018599999997 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.649223+0000", 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0874999999999997e-05 2024-09-14T04:15:28.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.649256+0000", 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3319999999999999e-05 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:123323 11.13 11:cfdd8a39:::benchmark_data_smithi114_223345_object11386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:28.676 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:25.396690+0000", 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 63.184310037000003, 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.25300202599999999, 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 123323 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.396690+0000", 2024-09-14T04:15:28.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.396690+0000", 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.396689+0000", 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.396695+0000", 2024-09-14T04:15:28.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4600000000000002e-06 2024-09-14T04:15:28.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.396706+0000", 2024-09-14T04:15:28.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1372000000000001e-05 2024-09-14T04:15:28.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.396710+0000", 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4569999999999998e-06 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.396779+0000", 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8583000000000004e-05 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.396871+0000", 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2221999999999999e-05 2024-09-14T04:15:28.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.396942+0000", 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0616000000000003e-05 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.398523+0000", 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001581042 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.398675+0000", 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015234600000000001 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.649624+0000", 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.250948326 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.649661+0000", 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6928000000000003e-05 2024-09-14T04:15:28.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.649693+0000", 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.201e-05 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127224 11.4 11:21e6e516:::benchmark_data_smithi114_223345_object15287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:15:28.683 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.551373+0000", 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 59.029627681999997, 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0083590950000001, 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127224 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551373+0000", 2024-09-14T04:15:28.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551373+0000", 2024-09-14T04:15:28.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551371+0000", 2024-09-14T04:15:28.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:28.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551379+0000", 2024-09-14T04:15:28.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4420000000000001e-06 2024-09-14T04:15:28.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551393+0000", 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4610000000000001e-05 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551400+0000", 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4579999999999998e-06 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551456+0000", 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6212000000000003e-05 2024-09-14T04:15:28.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551467+0000", 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.075e-05 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553349+0000", 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.001882191 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553444+0000", 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4993000000000003e-05 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553534+0000", 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0431000000000002e-05 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559606+0000", 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.006072076 2024-09-14T04:15:28.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559662+0000", 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5921999999999998e-05 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559669+0000", 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0160000000000003e-06 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559702+0000", 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2906e-05 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559732+0000", 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9733000000000001e-05 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127241 11.14 11:2e01ae4a:::benchmark_data_smithi114_223345_object15304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.572850+0000", 2024-09-14T04:15:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 59.008150690000001, 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98064877299999997, 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127241 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572850+0000", 2024-09-14T04:15:28.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572850+0000", 2024-09-14T04:15:28.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572849+0000", 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572853+0000", 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8460000000000001e-06 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572864+0000", 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.098e-05 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572868+0000", 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4410000000000003e-06 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572922+0000", 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3471999999999999e-05 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572935+0000", 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2975e-05 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.548843+0000", 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.975908687 2024-09-14T04:15:28.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.548971+0000", 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724200000000001 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.549084+0000", 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113316 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551528+0000", 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00244378 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553317+0000", 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017892120000000001 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553369+0000", 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1722999999999999e-05 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553463+0000", 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4036000000000006e-05 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553499+0000", 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5998999999999999e-05 2024-09-14T04:15:28.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127248 11.4 11:26baefff:::benchmark_data_smithi114_223345_object15311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.587658+0000", 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 58.993342730999998, 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97257697799999998, 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.703 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127248 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587658+0000", 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587658+0000", 2024-09-14T04:15:28.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587656+0000", 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587665+0000", 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3259999999999996e-06 2024-09-14T04:15:28.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587683+0000", 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8300000000000001e-05 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587689+0000", 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7420000000000003e-06 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587749+0000", 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0062000000000002e-05 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587761+0000", 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2285e-05 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555352+0000", 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96759091900000005 2024-09-14T04:15:28.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555578+0000", 2024-09-14T04:15:28.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000226176 2024-09-14T04:15:28.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:15:28.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555671+0000", 2024-09-14T04:15:28.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2947000000000003e-05 2024-09-14T04:15:28.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559633+0000", 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039615220000000003 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559737+0000", 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104633 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560169+0000", 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043182499999999999 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560203+0000", 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3831000000000002e-05 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560235+0000", 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1977000000000002e-05 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127262 11.14 11:28c1fb4b:::benchmark_data_smithi114_223345_object15325:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.611254+0000", 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 58.969746559999997, 2024-09-14T04:15:28.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94832032799999999, 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127262 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611254+0000", 2024-09-14T04:15:28.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611254+0000", 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611252+0000", 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611260+0000", 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9840000000000001e-06 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611276+0000", 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5164000000000001e-05 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611282+0000", 2024-09-14T04:15:28.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4060000000000002e-06 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611339+0000", 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7014e-05 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611351+0000", 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1579999999999999e-05 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551549+0000", 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.940198321 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551673+0000", 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012361499999999999 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551780+0000", 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000107011 2024-09-14T04:15:28.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555307+0000", 2024-09-14T04:15:28.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003527002 2024-09-14T04:15:28.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555312+0000", 2024-09-14T04:15:28.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7950000000000004e-06 2024-09-14T04:15:28.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559499+0000", 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041868319999999997 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559541+0000", 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1721999999999998e-05 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559574+0000", 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3407e-05 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127264 11.4 11:260f74ee:::benchmark_data_smithi114_223345_object15327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.613838+0000", 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 58.967162975000001, 2024-09-14T04:15:28.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94978247400000004, 2024-09-14T04:15:28.720 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.720 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.720 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.720 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.720 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.720 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127264 2024-09-14T04:15:28.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.720 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613838+0000", 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613838+0000", 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613836+0000", 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613843+0000", 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1550000000000004e-06 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613855+0000", 2024-09-14T04:15:28.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2069000000000001e-05 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613859+0000", 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3599999999999998e-06 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613896+0000", 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7217e-05 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613904+0000", 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2099999999999996e-06 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557498+0000", 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94359476099999995 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557706+0000", 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020764700000000001 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557796+0000", 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0552999999999999e-05 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560072+0000", 2024-09-14T04:15:28.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022751780000000001 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.562269+0000", 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021972390000000001 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563542+0000", 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001273167 2024-09-14T04:15:28.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563586+0000", 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4351000000000003e-05 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563620+0000", 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3630999999999997e-05 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:156849 11.4 11:27a60543:::benchmark_data_smithi114_223345_object44912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-14T04:15:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:01.340284+0000", 2024-09-14T04:15:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 27.240716417000002, 2024-09-14T04:15:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.23468106699999999, 2024-09-14T04:15:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.728 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156849 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.340284+0000", 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.340284+0000", 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.340284+0000", 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.340287+0000", 2024-09-14T04:15:28.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3639999999999999e-06 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.340293+0000", 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4990000000000002e-06 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.340295+0000", 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.847e-06 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.340335+0000", 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0030000000000001e-05 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.340442+0000", 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010709000000000001 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:15:28.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.340511+0000", 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8501e-05 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.342592+0000", 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020810939999999999 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.342750+0000", 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015745000000000001 2024-09-14T04:15:28.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.574887+0000", 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.23213715500000001 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.574929+0000", 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2126999999999998e-05 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:01.574965+0000", 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6381000000000003e-05 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:165893 11.1 11:835faac2:::benchmark_data_smithi114_223345_object53956:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:11.283877+0000", 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.297123109000001, 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.233496064, 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:28.735 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 165893 2024-09-14T04:15:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.283877+0000", 2024-09-14T04:15:28.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.283877+0000", 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.283876+0000", 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.283886+0000", 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.006e-05 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.283900+0000", 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3645000000000001e-05 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:28.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.283905+0000", 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5629999999999999e-06 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.283957+0000", 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1397000000000003e-05 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.284084+0000", 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012764699999999999 2024-09-14T04:15:28.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.284199+0000", 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011524 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.286836+0000", 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026362540000000002 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.286971+0000", 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013540000000000001 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.517264+0000", 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.230292682 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:28.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.517335+0000", 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1296000000000003e-05 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:11.517373+0000", 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8404e-05 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:28.742 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:15:28.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:28.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:28.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:28.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:28.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:28.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:28.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:29.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:29.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:29.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:29.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:29.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:29.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:29.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:29.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:29.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:29.339 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:29.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:29.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:29.472 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:29.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:29.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:29.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:29.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:29.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-14T04:15:29.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:29.754 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:29.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:29.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:29.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:29.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:29.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:30.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:30.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:30.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:30.077 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:30.138 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:30.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:30.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:30.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:30.273 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:29 smithi160 ceph-mon[88407]: pgmap v992: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 262 op/s; 9821/133578 objects degraded (7.352%); 5842/133578 objects misplaced (4.373%) 2024-09-14T04:15:30.273 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9821/133578 objects degraded (7.352%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:30.273 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:15:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:15:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:15:30.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:30.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:29 smithi114 ceph-mon[103029]: pgmap v992: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 262 op/s; 9821/133578 objects degraded (7.352%); 5842/133578 objects misplaced (4.373%) 2024-09-14T04:15:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9821/133578 objects degraded (7.352%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:29 smithi114 ceph-mon[106969]: pgmap v992: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 262 op/s; 9821/133578 objects degraded (7.352%); 5842/133578 objects misplaced (4.373%) 2024-09-14T04:15:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9821/133578 objects degraded (7.352%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:30.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:30.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:30.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:30.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:30.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:30.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:30.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:30.704 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:30.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:30.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:30.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:30.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:30.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:30.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:30.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:30.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:31.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:31.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:31.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:31.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:31.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:31.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:31.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:31.365 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:31.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:31.467 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:31.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:31.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:31.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:31.651 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:31.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:31.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:31.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:31.824 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:15:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:15:31] "GET /metrics HTTP/1.1" 200 36593 "" "Prometheus/2.51.0" 2024-09-14T04:15:31.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:31.960 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:31.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:32.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:32.096 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:31 smithi160 ceph-mon[88407]: pgmap v993: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 9821/129912 objects degraded (7.560%); 5842/129912 objects misplaced (4.497%); 1.2 MiB/s, 18 objects/s recovering 2024-09-14T04:15:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:32.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:32.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:32.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:31 smithi114 ceph-mon[103029]: pgmap v993: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 9821/129912 objects degraded (7.560%); 5842/129912 objects misplaced (4.497%); 1.2 MiB/s, 18 objects/s recovering 2024-09-14T04:15:32.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:31 smithi114 ceph-mon[106969]: pgmap v993: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 9821/129912 objects degraded (7.560%); 5842/129912 objects misplaced (4.497%); 1.2 MiB/s, 18 objects/s recovering 2024-09-14T04:15:32.124 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:183777 11.4 e267/261 11:26996ce7:::benchmark_data_smithi114_223345_object71840:head v 267'7753, mlcod=267'7750)", 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:32.122776+0000", 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00017934299999999999, 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210717, 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:32.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:32.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:32.122776+0000", 2024-09-14T04:15:32.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:32.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:32.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:32.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:32.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:32.122776+0000", 2024-09-14T04:15:32.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:32.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:32.122775+0000", 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:32.122784+0000", 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.85e-06 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:32.122798+0000", 2024-09-14T04:15:32.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4287999999999999e-05 2024-09-14T04:15:32.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:32.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:32.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:32.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:32.122804+0000", 2024-09-14T04:15:32.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9769999999999997e-06 2024-09-14T04:15:32.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:32.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:32.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:32.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:32.122854+0000", 2024-09-14T04:15:32.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9270000000000001e-05 2024-09-14T04:15:32.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:32.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:32.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:32.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:32.122874+0000", 2024-09-14T04:15:32.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9848999999999999e-05 2024-09-14T04:15:32.129 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:32.129 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:32.129 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:32.130 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:32.130 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:15:32.130 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 1 2024-09-14T04:15:32.130 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:15:32.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:32.252 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:32.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:32.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:32.393 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:32.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:32.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:32.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:32.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:32.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-14T04:15:32.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:32.703 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:32.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:32.726 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:32.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:32.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:32.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:32.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:32.963 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:32.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:33.010 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-14T04:15:33.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:33.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:33.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:33.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:33.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:33.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:33.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:33.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:33.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:33.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:33.398 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:33.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:33.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:33.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:33.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:33.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:33.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:33.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:33.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:33.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:33.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:15:33.883 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:33.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:33.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:34.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:34.051 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:34.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:34.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:34 smithi114 ceph-mon[103029]: pgmap v994: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 9920/127947 objects degraded (7.753%); 5560/127947 objects misplaced (4.346%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:15:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:34 smithi114 ceph-mon[106969]: pgmap v994: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 9920/127947 objects degraded (7.753%); 5560/127947 objects misplaced (4.346%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:15:34.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:34.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:34 smithi160 ceph-mon[88407]: pgmap v994: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 9920/127947 objects degraded (7.753%); 5560/127947 objects misplaced (4.346%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:15:34.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:34.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:34.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:34.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:34.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:34.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:34.435 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:34.468 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:34.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:34.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:34.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:34.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:34.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:34.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:34.756 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:34.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:34.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:34.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:34.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:34.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:35.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:35.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:35.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:35.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:35.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:35.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:35.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:35.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:35.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:35 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9920/127947 objects degraded (7.753%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:35 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9920/127947 objects degraded (7.753%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:35.375 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:35 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9920/127947 objects degraded (7.753%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:35.419 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:35.432 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:35.433 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:15:35.433 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:15:35.433 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:15:35.433 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:15:35.433 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:15:35.472 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:35.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:35.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:35.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:35.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:35.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:35.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:35.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:35.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:35.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:35.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:35.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:36.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:36.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:36.098 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:36 smithi160 ceph-mon[88407]: pgmap v995: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 9920/117264 objects degraded (8.460%); 5560/117264 objects misplaced (4.741%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:15:36.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:36.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-14T04:15:36.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:36.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:36.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:36.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:36.362 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:36 smithi114 ceph-mon[103029]: pgmap v995: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 9920/117264 objects degraded (8.460%); 5560/117264 objects misplaced (4.741%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:15:36.362 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:36 smithi114 ceph-mon[106969]: pgmap v995: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 9920/117264 objects degraded (8.460%); 5560/117264 objects misplaced (4.741%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:15:36.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:36.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:36.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:36.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:36.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:36.595 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:36.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:36.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:36.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:36.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:36.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:36.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:36.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:36.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:36.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:37.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:37.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:37.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:37.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:37.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:37.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:37.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:37.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:37.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:37.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:37.484 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:37.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:37.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:37.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:37.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:37.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:37.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:37.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:37.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:37.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:37.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:37.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:37.973 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:38.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:38.042 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:38.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:38.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:38 smithi114 ceph-mon[103029]: pgmap v996: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 898 op/s; 10050/115182 objects degraded (8.725%); 5282/115182 objects misplaced (4.586%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:15:38.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:38.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:38.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/942080564' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:15:38.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:38 smithi114 ceph-mon[106969]: pgmap v996: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 898 op/s; 10050/115182 objects degraded (8.725%); 5282/115182 objects misplaced (4.586%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:15:38.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:38.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:38.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/942080564' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:15:38.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:38 smithi160 ceph-mon[88407]: pgmap v996: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 898 op/s; 10050/115182 objects degraded (8.725%); 5282/115182 objects misplaced (4.586%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:15:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/942080564' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:15:38.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:38.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:38.284 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:38.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:38.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:38.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:38.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:38.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:38.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:38.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:38.644 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:15:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:15:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:15:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:15:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:195981 9.e 9:741dedbf:::benchmark_data_smithi114_153893_object66520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-14T04:15:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:06.788523+0000", 2024-09-14T04:15:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 571.85000898800001, 2024-09-14T04:15:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.331278462, 2024-09-14T04:15:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195981 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788523+0000", 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788523+0000", 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788522+0000", 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788528+0000", 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3659999999999997e-06 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788541+0000", 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2755000000000001e-05 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788546+0000", 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0239999999999997e-06 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788597+0000", 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1316000000000001e-05 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788726+0000", 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012843300000000001 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,4", 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.788884+0000", 2024-09-14T04:15:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015847600000000001 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.792615+0000", 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003730424 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.792730+0000", 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011485900000000001 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.805843+0000", 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013113855000000001 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119607+0000", 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31376335799999999 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119669+0000", 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2397000000000006e-05 2024-09-14T04:15:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119802+0000", 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013257899999999999 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:195983 9.e 9:76113637:::benchmark_data_smithi114_153893_object66522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:06.793113+0000", 2024-09-14T04:15:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 571.84541978599998, 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32693720900000001, 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195983 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793113+0000", 2024-09-14T04:15:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793113+0000", 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793111+0000", 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793117+0000", 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4919999999999998e-06 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793130+0000", 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2347999999999999e-05 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793136+0000", 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2310000000000001e-06 2024-09-14T04:15:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793173+0000", 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7468000000000001e-05 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793290+0000", 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116841 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,4", 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.793372+0000", 2024-09-14T04:15:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1866000000000005e-05 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.796363+0000", 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029914579999999998 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.797301+0000", 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000937089 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.805934+0000", 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0086334329999999994 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119867+0000", 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31393343200000001 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.119922+0000", 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4187000000000001e-05 2024-09-14T04:15:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.120050+0000", 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012817899999999999 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:195999 9.e 9:7a76e1c2:::benchmark_data_smithi114_153893_object66538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:06.807061+0000", 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 571.83147121699994, 2024-09-14T04:15:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31979472599999997, 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195999 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807061+0000", 2024-09-14T04:15:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807061+0000", 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807060+0000", 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807064+0000", 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.912e-06 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807075+0000", 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0807e-05 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807081+0000", 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9699999999999996e-06 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807122+0000", 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1507000000000001e-05 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807247+0000", 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012494900000000001 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,4", 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.807381+0000", 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000133621 2024-09-14T04:15:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.810839+0000", 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034575180000000001 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.810906+0000", 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7020999999999994e-05 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:06.811227+0000", 2024-09-14T04:15:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00032114300000000003 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.126723+0000", 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31549654399999999 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.126759+0000", 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5809999999999998e-05 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:07.126856+0000", 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6773000000000003e-05 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221222 9.e 9:71e3d4ca:::benchmark_data_smithi114_153893_object91761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.075396+0000", 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.56313662800005, 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98772043600000003, 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221222 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075396+0000", 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075396+0000", 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075394+0000", 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075399+0000", 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2059999999999998e-06 2024-09-14T04:15:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075414+0000", 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4457999999999999e-05 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075419+0000", 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0590000000000002e-06 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075463+0000", 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4357000000000002e-05 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.075471+0000", 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5900000000000002e-06 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.054310+0000", 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97883944700000003 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.054405+0000", 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4752000000000004e-05 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.054480+0000", 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4501999999999999e-05 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.062905+0000", 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0084256130000000002 2024-09-14T04:15:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.062914+0000", 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8750000000000006e-06 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063021+0000", 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106359 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063077+0000", 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6490999999999998e-05 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063116+0000", 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9150999999999998e-05 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221279 9.e 9:7a0e57f8:::benchmark_data_smithi114_153893_object91818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.162273+0000", 2024-09-14T04:15:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.47625899399998, 2024-09-14T04:15:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:15:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90723786100000003, 2024-09-14T04:15:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221279 2024-09-14T04:15:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162273+0000", 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162273+0000", 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162271+0000", 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162279+0000", 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9810000000000003e-06 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162291+0000", 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1804000000000001e-05 2024-09-14T04:15:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162297+0000", 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2790000000000004e-06 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162350+0000", 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2383999999999999e-05 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.162358+0000", 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.057426+0000", 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89506750499999999 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.057490+0000", 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4579999999999995e-05 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.057567+0000", 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6237999999999998e-05 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.062944+0000", 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0053775539999999997 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063126+0000", 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000181977 2024-09-14T04:15:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069407+0000", 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062807540000000004 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069473+0000", 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6450000000000002e-05 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069511+0000", 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7894e-05 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221287 9.e 9:7c5f7cde:::benchmark_data_smithi114_153893_object91826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.168058+0000", 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.47047482899995, 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90162981099999995, 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221287 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168058+0000", 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168058+0000", 2024-09-14T04:15:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168056+0000", 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168061+0000", 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8089999999999998e-06 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168075+0000", 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3091000000000001e-05 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168080+0000", 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7799999999999997e-06 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168154+0000", 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3906999999999996e-05 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.168162+0000", 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4819999999999997e-06 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.060288+0000", 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89212670500000002 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.060357+0000", 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8893999999999995e-05 2024-09-14T04:15:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.060439+0000", 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.174e-05 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063132+0000", 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002692664 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069374+0000", 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062418459999999997 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069617+0000", 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024384999999999999 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069658+0000", 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0562000000000002e-05 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069687+0000", 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9346999999999999e-05 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221290 9.e 9:7e953b68:::benchmark_data_smithi114_153893_object91829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.169237+0000", 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.46929548000003, 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.90707892899999998, 2024-09-14T04:15:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221290 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169237+0000", 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169237+0000", 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169236+0000", 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169242+0000", 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0349999999999998e-06 2024-09-14T04:15:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169254+0000", 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2427e-05 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169260+0000", 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9259999999999997e-06 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169302+0000", 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2283999999999997e-05 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.169309+0000", 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0589999999999997e-06 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063175+0000", 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89386510500000005 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063285+0000", 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011004 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.063400+0000", 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011540300000000001 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069380+0000", 2024-09-14T04:15:38.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059798780000000001 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069573+0000", 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000193191 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076249+0000", 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0066763270000000001 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076282+0000", 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2206999999999998e-05 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076316+0000", 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4171000000000002e-05 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221308 9.e 9:7b1cb916:::benchmark_data_smithi114_153893_object91847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.695 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.194179+0000", 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.44435303800003, 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88227606199999997, 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221308 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194179+0000", 2024-09-14T04:15:38.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194179+0000", 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194178+0000", 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194184+0000", 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5339999999999999e-06 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194195+0000", 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1090000000000001e-05 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194200+0000", 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4079999999999997e-06 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194253+0000", 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3186999999999997e-05 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.194260+0000", 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8920000000000003e-06 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.065682+0000", 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87142183299999998 2024-09-14T04:15:38.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.065750+0000", 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7461999999999999e-05 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.065824+0000", 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4950999999999994e-05 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069695+0000", 2024-09-14T04:15:38.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0038707199999999998 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069743+0000", 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7444999999999997e-05 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076385+0000", 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0066419080000000002 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076424+0000", 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9212000000000003e-05 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076455+0000", 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1605e-05 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221336 9.e 9:7fbc0f9f:::benchmark_data_smithi114_153893_object91875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.416696161, 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86189694800000005, 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221336 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.703 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221836+0000", 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221839+0000", 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8779999999999998e-06 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221844+0000", 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9030000000000001e-06 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221848+0000", 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.506e-06 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221883+0000", 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5491000000000003e-05 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.221888+0000", 2024-09-14T04:15:38.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7690000000000004e-06 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069784+0000", 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84789569300000001 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.069903+0000", 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119528 2024-09-14T04:15:38.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.070022+0000", 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118662 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076341+0000", 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0063191180000000003 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076464+0000", 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012243599999999999 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083657+0000", 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0071937420000000004 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083700+0000", 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2095000000000003e-05 2024-09-14T04:15:38.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083733+0000", 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3596000000000002e-05 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221353 9.e 9:7a884025:::benchmark_data_smithi114_153893_object91892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.247072+0000", 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.39145993900001, 2024-09-14T04:15:38.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85085830500000004, 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221353 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247072+0000", 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247072+0000", 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247071+0000", 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247078+0000", 2024-09-14T04:15:38.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6290000000000004e-06 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247089+0000", 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1168999999999999e-05 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247093+0000", 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.578e-06 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247140+0000", 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6113999999999999e-05 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.247148+0000", 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1839999999999999e-06 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.073043+0000", 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82589522500000001 2024-09-14T04:15:38.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.073170+0000", 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000127541 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.073311+0000", 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014082 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083510+0000", 2024-09-14T04:15:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010198239 2024-09-14T04:15:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083573+0000", 2024-09-14T04:15:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3004999999999997e-05 2024-09-14T04:15:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097818+0000", 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014245821000000001 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097866+0000", 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8029999999999999e-05 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097931+0000", 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4312000000000004e-05 2024-09-14T04:15:38.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.718 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221359 9.e 9:76f61d7d:::benchmark_data_smithi114_153893_object91898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.252688+0000", 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.38584401100002, 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84535683900000003, 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221359 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.720 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252688+0000", 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252688+0000", 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252687+0000", 2024-09-14T04:15:38.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252696+0000", 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5210000000000008e-06 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252709+0000", 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2852e-05 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252716+0000", 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5749999999999997e-06 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252760+0000", 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.405e-05 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.252768+0000", 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0490000000000004e-06 2024-09-14T04:15:38.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076490+0000", 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82372204199999999 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076560+0000", 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9900999999999994e-05 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.076623+0000", 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3226999999999996e-05 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083562+0000", 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0069392339999999999 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083593+0000", 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0880999999999998e-05 2024-09-14T04:15:38.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097986+0000", 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014392922000000001 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098017+0000", 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1362000000000002e-05 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098045+0000", 2024-09-14T04:15:38.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7875999999999999e-05 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221381 9.e 9:70816107:::benchmark_data_smithi114_153893_object91920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.278485+0000", 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.36004686199999, 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81964557199999999, 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.727 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221381 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278485+0000", 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278485+0000", 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278484+0000", 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278490+0000", 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5910000000000003e-06 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278502+0000", 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2756e-05 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278508+0000", 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3410000000000002e-06 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278558+0000", 2024-09-14T04:15:38.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9959000000000001e-05 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.278564+0000", 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3319999999999997e-06 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.080018+0000", 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80145439200000002 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.080117+0000", 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8821999999999997e-05 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.080228+0000", 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111292 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083771+0000", 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003542842 2024-09-14T04:15:38.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097698+0000", 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013926565 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098075+0000", 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037682900000000002 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098104+0000", 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9519e-05 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098131+0000", 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6872999999999998e-05 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221383 9.e 9:78a714d0:::benchmark_data_smithi114_153893_object91922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.734 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.282793+0000", 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.35573943700001, 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82646461900000001, 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221383 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282793+0000", 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282793+0000", 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282791+0000", 2024-09-14T04:15:38.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:38.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282797+0000", 2024-09-14T04:15:38.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5149999999999997e-06 2024-09-14T04:15:38.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282808+0000", 2024-09-14T04:15:38.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1358e-05 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282820+0000", 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1313e-05 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282857+0000", 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6928000000000003e-05 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.282862+0000", 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.677e-06 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083790+0000", 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.80092809499999995 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083893+0000", 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103038 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.083996+0000", 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010259600000000001 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097702+0000", 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013706476 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097754+0000", 2024-09-14T04:15:38.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1768000000000003e-05 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109094+0000", 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011340195000000001 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109221+0000", 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000127019 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109258+0000", 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6075000000000003e-05 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221421 9.e 9:7e5137ba:::benchmark_data_smithi114_153893_object91960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.359708+0000", 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.27882445700004, 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.74967277499999996, 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.742 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221421 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359708+0000", 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359708+0000", 2024-09-14T04:15:38.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359706+0000", 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359713+0000", 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5989999999999998e-06 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359732+0000", 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9040000000000001e-05 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359738+0000", 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.404e-06 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359780+0000", 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1572000000000001e-05 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.359787+0000", 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4490000000000002e-06 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.087250+0000", 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72746270599999996 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.087357+0000", 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010766 2024-09-14T04:15:38.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.087467+0000", 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109231 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097707+0000", 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010240383 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097786+0000", 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9331000000000003e-05 2024-09-14T04:15:38.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109308+0000", 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011522107 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109345+0000", 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6078999999999998e-05 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109381+0000", 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6149000000000002e-05 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221427 9.e 9:73b3c5ab:::benchmark_data_smithi114_153893_object91966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.371115+0000", 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.26741693700001, 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.73836333300000001, 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.749 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221427 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371115+0000", 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371115+0000", 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371114+0000", 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371120+0000", 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3069999999999999e-06 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371134+0000", 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4318000000000001e-05 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371141+0000", 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9890000000000003e-06 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371193+0000", 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1620999999999999e-05 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.371201+0000", 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2719999999999997e-06 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.090232+0000", 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71903041000000001 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.090303+0000", 2024-09-14T04:15:38.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1452000000000006e-05 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.090372+0000", 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8422000000000003e-05 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097760+0000", 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.007388309 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.097961+0000", 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000200619 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109420+0000", 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011458983000000001 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109451+0000", 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1189999999999998e-05 2024-09-14T04:15:38.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109479+0000", 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8042e-05 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221428 9.e 9:7f963fb1:::benchmark_data_smithi114_153893_object91967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.372209+0000", 2024-09-14T04:15:38.756 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.26632374799999, 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.73735522099999995, 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221428 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372209+0000", 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372209+0000", 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372208+0000", 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372213+0000", 2024-09-14T04:15:38.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4999999999999999e-06 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372227+0000", 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3754000000000001e-05 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372233+0000", 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7749999999999998e-06 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372281+0000", 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8371999999999997e-05 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.372288+0000", 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.306e-06 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.093512+0000", 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72122381700000004 2024-09-14T04:15:38.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.093613+0000", 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100993 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.093719+0000", 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105808 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105360+0000", 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.01164106 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105400+0000", 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0420999999999998e-05 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109507+0000", 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041070669999999998 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109537+0000", 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9247999999999998e-05 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109564+0000", 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7118e-05 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221441 9.e 9:79734224:::benchmark_data_smithi114_153893_object91980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.396588+0000", 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.24194479599998, 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71322003, 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221441 2024-09-14T04:15:38.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396588+0000", 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396588+0000", 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396586+0000", 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396591+0000", 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9289999999999997e-06 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396602+0000", 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0547000000000001e-05 2024-09-14T04:15:38.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396607+0000", 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1000000000000003e-06 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396649+0000", 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1941999999999999e-05 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396656+0000", 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8369999999999999e-06 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098153+0000", 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70149738800000006 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098264+0000", 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011047100000000001 2024-09-14T04:15:38.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.098389+0000", 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000125053 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109038+0000", 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010649614 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109276+0000", 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000237999 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109715+0000", 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043909999999999999 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109776+0000", 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0977999999999998e-05 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109808+0000", 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1319000000000003e-05 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221442 9.e 9:7a9a1506:::benchmark_data_smithi114_153893_object91981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.396654+0000", 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.241878692, 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72359706499999998, 2024-09-14T04:15:38.771 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221442 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396654+0000", 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396654+0000", 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396653+0000", 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396656+0000", 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0419999999999998e-06 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396667+0000", 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1181e-05 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396671+0000", 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9879999999999998e-06 2024-09-14T04:15:38.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396699+0000", 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7458e-05 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.396704+0000", 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0039999999999999e-06 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.101911+0000", 2024-09-14T04:15:38.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70520688399999998 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.102000+0000", 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9349000000000001e-05 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.102100+0000", 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9623999999999993e-05 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109387+0000", 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0072876240000000004 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.117933+0000", 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0085459390000000007 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120171+0000", 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022373919999999999 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120213+0000", 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2882e-05 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120251+0000", 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7323000000000002e-05 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.45572.0:221473 9.e 9:74ac708e:::benchmark_data_smithi114_153893_object92012:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-14T04:15:38.778 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:06:42.530644+0000", 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 536.10788855299995, 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.58970111199999997, 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.45572", 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/4110808035", 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 221473 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530644+0000", 2024-09-14T04:15:38.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530644+0000", 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530643+0000", 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530649+0000", 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1480000000000003e-06 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530660+0000", 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1765e-05 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530666+0000", 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7529999999999998e-06 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530701+0000", 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4487999999999999e-05 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:42.530706+0000", 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4689999999999996e-06 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105426+0000", 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.57472014800000004 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105530+0000", 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103225 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.105641+0000", 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011111399999999999 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.109392+0000", 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0037509470000000001 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.117959+0000", 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.008567346 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120281+0000", 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023216310000000002 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120314+0000", 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3865000000000002e-05 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:06:43.120345+0000", 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0487000000000001e-05 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:123294 11.f 11:f165a627:::benchmark_data_smithi114_223345_object11357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:25.356081+0000", 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 73.282451449999996, 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.286648032, 2024-09-14T04:15:38.785 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 123294 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.356081+0000", 2024-09-14T04:15:38.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.356081+0000", 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.356080+0000", 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.356083+0000", 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.509e-06 2024-09-14T04:15:38.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.356092+0000", 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7220000000000005e-06 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.356095+0000", 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6669999999999998e-06 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.362509+0000", 2024-09-14T04:15:38.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0064135310000000001 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.362621+0000", 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000112048 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.362738+0000", 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117113 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:15:38.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.365751+0000", 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030130930000000001 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.371677+0000", 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059253739999999997 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.642627+0000", 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27095054299999999 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.642681+0000", 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3458000000000002e-05 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:25.642729+0000", 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8381999999999999e-05 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.792 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:15:38.793 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:15:38.793 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:15:38.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:38.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:38.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:38.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:38.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:39.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:39.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:39.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:39.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:39.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:39.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:39.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:39.364 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:39.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:39.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:39 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:15:39.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/942080564' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:15:39.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:39 smithi114 ceph-mon[106969]: osdmap e268: 8 total, 8 up, 8 in 2024-09-14T04:15:39.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:39 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:15:39.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/942080564' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:15:39.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:39 smithi114 ceph-mon[103029]: osdmap e268: 8 total, 8 up, 8 in 2024-09-14T04:15:39.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:39 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:15:39.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/942080564' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:15:39.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:39 smithi160 ceph-mon[88407]: osdmap e268: 8 total, 8 up, 8 in 2024-09-14T04:15:39.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:39.507 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:39.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:39.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:39.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:39.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:39.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:39.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:39.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:39.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:39.927 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:39.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:40.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:40.039 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-14T04:15:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:40.078 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:40.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:40.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:40.217 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:40.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:40.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:40.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:40.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:40.347 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:15:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:15:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:15:40.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:40 smithi160 ceph-mon[88407]: pgmap v998: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 10050/107664 objects degraded (9.335%); 5282/107664 objects misplaced (4.906%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:15:40.347 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2555410622' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:15:40.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:40 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:15:40.348 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:40 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 10050/115182 objects degraded (8.725%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:40.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:40 smithi114 ceph-mon[106969]: pgmap v998: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 10050/107664 objects degraded (9.335%); 5282/107664 objects misplaced (4.906%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:15:40.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2555410622' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:15:40.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:40 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:15:40.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:40 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 10050/115182 objects degraded (8.725%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:40 smithi114 ceph-mon[103029]: pgmap v998: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 10050/107664 objects degraded (9.335%); 5282/107664 objects misplaced (4.906%); 2.1 MiB/s, 33 objects/s recovering 2024-09-14T04:15:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2555410622' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:15:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:40 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:15:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:40 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 10050/115182 objects degraded (8.725%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:40.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:40.475 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:40.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:40.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:40.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:40.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:40.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:40.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:40.764 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:40.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:40.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:40.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:40.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:40.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:15:41.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:41.038 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:41.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:41.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:41 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:15:41.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:41 smithi114 ceph-mon[106969]: osdmap e269: 8 total, 8 up, 8 in 2024-09-14T04:15:41.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:41 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:15:41.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:41 smithi114 ceph-mon[103029]: osdmap e269: 8 total, 8 up, 8 in 2024-09-14T04:15:41.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:41 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:15:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:41 smithi160 ceph-mon[88407]: osdmap e269: 8 total, 8 up, 8 in 2024-09-14T04:15:41.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-14T04:15:41.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:41.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:41.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:41.345 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:41.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:41.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:41.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:41.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:41.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:41.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:41.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:41.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:41.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:41.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:41.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:41.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:41.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:15:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:15:41] "GET /metrics HTTP/1.1" 200 36598 "" "Prometheus/2.51.0" 2024-09-14T04:15:41.889 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:41.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:41.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:42.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:42.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:42.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:42 smithi160 ceph-mon[88407]: pgmap v1000: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 10050/103794 objects degraded (9.683%); 5282/103794 objects misplaced (5.089%); 1.2 MiB/s, 18 objects/s recovering 2024-09-14T04:15:42.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:42.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:42.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:42.184 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:42 smithi114 ceph-mon[106969]: pgmap v1000: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 10050/103794 objects degraded (9.683%); 5282/103794 objects misplaced (5.089%); 1.2 MiB/s, 18 objects/s recovering 2024-09-14T04:15:42.184 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:42 smithi114 ceph-mon[103029]: pgmap v1000: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 10050/103794 objects degraded (9.683%); 5282/103794 objects misplaced (5.089%); 1.2 MiB/s, 18 objects/s recovering 2024-09-14T04:15:42.243 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:15:42.243 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:15:42.250 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:42.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:42.279 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:15:42.279 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:15:42.315 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:15:42.315 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:15:42.316 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:42.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:42.348 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:15:42.349 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:15:42.376 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:15:42.377 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:15:42.406 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:15:42.406 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:15:42.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:42.439 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:15:42.439 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:15:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:42.462 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:42.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:42.473 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:15:42.473 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:15:42.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:42.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:42.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:42.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:42.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:42.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:42.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:42.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:42.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:42.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:42.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:43.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:43.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:43.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:43 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/834440023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:43.161 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:43.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:43.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:43.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:43.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:43.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:43.313 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:15:43.314 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:15:43.365 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:43 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/834440023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:43.365 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:43 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/834440023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:43.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:43.439 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:43.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:43.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:43.582 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:43.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:43.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:43.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:43.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:43.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:43.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:43.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:43.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:15:43.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:44.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-14T04:15:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:44.022 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:44.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:44.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:44.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:44.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:44.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:44.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:44.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:44.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:44 smithi114 ceph-mon[106969]: pgmap v1001: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 10137/101748 objects degraded (9.963%); 5032/101748 objects misplaced (4.946%); 2.3 MiB/s, 38 objects/s recovering 2024-09-14T04:15:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:44 smithi114 ceph-mon[103029]: pgmap v1001: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 10137/101748 objects degraded (9.963%); 5032/101748 objects misplaced (4.946%); 2.3 MiB/s, 38 objects/s recovering 2024-09-14T04:15:44.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:44.404 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:44 smithi160 ceph-mon[88407]: pgmap v1001: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 10137/101748 objects degraded (9.963%); 5032/101748 objects misplaced (4.946%); 2.3 MiB/s, 38 objects/s recovering 2024-09-14T04:15:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:44.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:44.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:44.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:44.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:44.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:44.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:44.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:44.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:44.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:44.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:44.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:44.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:44.979 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:44.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:44.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:45.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:45.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:45.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:45.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:45 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 10137/101748 objects degraded (9.963%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:45.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:45.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:45.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:45.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:45 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 10137/101748 objects degraded (9.963%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:45.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:45 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 10137/101748 objects degraded (9.963%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:45.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:45.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:45.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:45.410 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:45.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:45.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:45.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:45.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:45.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:45.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:45.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:45.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:45.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:45.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:45.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:15:45.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:45.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:45.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:45.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:46.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:46.081 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-14T04:15:46.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:46.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:46.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:46.229 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:46.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:46.290 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:46 smithi114 ceph-mon[103029]: pgmap v1002: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 10137/90060 objects degraded (11.256%); 5032/90060 objects misplaced (5.587%); 1.2 MiB/s, 20 objects/s recovering 2024-09-14T04:15:46.290 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1540092137' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:15:46.291 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:46 smithi114 ceph-mon[106969]: pgmap v1002: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 10137/90060 objects degraded (11.256%); 5032/90060 objects misplaced (5.587%); 1.2 MiB/s, 20 objects/s recovering 2024-09-14T04:15:46.291 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1540092137' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:15:46.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:46.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:46.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:46 smithi160 ceph-mon[88407]: pgmap v1002: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 10137/90060 objects degraded (11.256%); 5032/90060 objects misplaced (5.587%); 1.2 MiB/s, 20 objects/s recovering 2024-09-14T04:15:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1540092137' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:15:46.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:46.490 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:46.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:46.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:46.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:46.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:46.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:46.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:46.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:46.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:46.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:46.908 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:46.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:47.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:47.044 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:47.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:47.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:47.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-14T04:15:47.147 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:47.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:47.205 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:47.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:47.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:47.306 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:47 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1540092137' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:15:47.307 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:47 smithi160 ceph-mon[88407]: osdmap e270: 8 total, 8 up, 8 in 2024-09-14T04:15:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:47.340 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:47.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:47.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1540092137' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:15:47.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:47 smithi114 ceph-mon[103029]: osdmap e270: 8 total, 8 up, 8 in 2024-09-14T04:15:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:47 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1540092137' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:15:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:47 smithi114 ceph-mon[106969]: osdmap e270: 8 total, 8 up, 8 in 2024-09-14T04:15:47.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:47.476 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:47.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:47.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:47.617 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:15:47.617 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:15:47.621 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:47.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:47.725 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:47.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:47.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:47.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:47.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:47.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:47.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:47.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:48.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:48.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:48.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:48.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:48.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:48.195 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:48 smithi114 ceph-mon[103029]: pgmap v1004: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 9948/87957 objects degraded (11.310%); 5032/87957 objects misplaced (5.721%); 1.2 MiB/s, 43 objects/s recovering 2024-09-14T04:15:48.196 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:48 smithi114 ceph-mon[106969]: pgmap v1004: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 9948/87957 objects degraded (11.310%); 5032/87957 objects misplaced (5.721%); 1.2 MiB/s, 43 objects/s recovering 2024-09-14T04:15:48.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:48.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:48.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:48.300 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:48 smithi160 ceph-mon[88407]: pgmap v1004: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 9948/87957 objects degraded (11.310%); 5032/87957 objects misplaced (5.721%); 1.2 MiB/s, 43 objects/s recovering 2024-09-14T04:15:48.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-14T04:15:48.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:48.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:48.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:48.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:48.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:48.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:48.663 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:15:48.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:48.756 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:48.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:48.764 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:15:48.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:48.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:48.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:48.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:49.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:49.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:49.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:49.160 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:49.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:49.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:49.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:49.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:49.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:49.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:49.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:49.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:49.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:49.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:49.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:49.694 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:49.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:49.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:49.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:49.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:49.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:49.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:49.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:50.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:50.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:50.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:50.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:50.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:50.252 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:50.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:50.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:50.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:50.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:50 smithi114 ceph-mon[106969]: pgmap v1005: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 9948/80736 objects degraded (12.322%); 5032/80736 objects misplaced (6.233%); 1.1 MiB/s, 39 objects/s recovering 2024-09-14T04:15:50.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:50 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9948/80736 objects degraded (12.322%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:50 smithi114 ceph-mon[103029]: pgmap v1005: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 9948/80736 objects degraded (12.322%); 5032/80736 objects misplaced (6.233%); 1.1 MiB/s, 39 objects/s recovering 2024-09-14T04:15:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:50 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9948/80736 objects degraded (12.322%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:50.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:50.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:15:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:15:50.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:15:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:50 smithi160 ceph-mon[88407]: pgmap v1005: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 9948/80736 objects degraded (12.322%); 5032/80736 objects misplaced (6.233%); 1.1 MiB/s, 39 objects/s recovering 2024-09-14T04:15:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:50 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9948/80736 objects degraded (12.322%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:50.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:50.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:50.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:50.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:50.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:50.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:50.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:50.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:50.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:50.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:50.824 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:15:50.824 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:15:50.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:50.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:50.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:51.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:51.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:51.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:51.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:51.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:51.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:51.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:51.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:51.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:15:51.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:15:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:51.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:51.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:51.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:15:51.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:51.505 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:51.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:51.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:51.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:51.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:51.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:51.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:51.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:51.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:51.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:51.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:51.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:15:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:15:51] "GET /metrics HTTP/1.1" 200 36598 "" "Prometheus/2.51.0" 2024-09-14T04:15:51.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:51.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:51.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:51.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:52.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:52.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:52.139 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:52.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:52.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:52.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:52.279 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:52.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:52.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-14T04:15:52.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:52 smithi160 ceph-mon[88407]: pgmap v1006: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 9948/76920 objects degraded (12.933%); 5032/76920 objects misplaced (6.542%); 972 KiB/s, 35 objects/s recovering 2024-09-14T04:15:52.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:52.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:52.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:52.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:52.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:52.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:52 smithi114 ceph-mon[103029]: pgmap v1006: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 9948/76920 objects degraded (12.933%); 5032/76920 objects misplaced (6.542%); 972 KiB/s, 35 objects/s recovering 2024-09-14T04:15:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:52 smithi114 ceph-mon[106969]: pgmap v1006: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 9948/76920 objects degraded (12.933%); 5032/76920 objects misplaced (6.542%); 972 KiB/s, 35 objects/s recovering 2024-09-14T04:15:52.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:52.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:52.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:52.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:52.840 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:52.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:52.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:52.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:52.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:53.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:53.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:53.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:53.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:53.252 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:53.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:53.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:53.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:53.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:53.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:53.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:53.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:53.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:53.604 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:53.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:15:53.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:53.671 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:53.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:53.773 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:53.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:53.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:53.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:53.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:53.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:53.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:54.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:54.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:54.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:54.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:54.239 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:54.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:54.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:54.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:54.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:54.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:54.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:54.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:54 smithi160 ceph-mon[88407]: pgmap v1007: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 9755/74883 objects degraded (13.027%); 5032/74883 objects misplaced (6.720%); 0 B/s, 38 objects/s recovering 2024-09-14T04:15:54.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:54.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:54.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:54.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:54 smithi114 ceph-mon[106969]: pgmap v1007: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 9755/74883 objects degraded (13.027%); 5032/74883 objects misplaced (6.720%); 0 B/s, 38 objects/s recovering 2024-09-14T04:15:54.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:54 smithi114 ceph-mon[103029]: pgmap v1007: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 9755/74883 objects degraded (13.027%); 5032/74883 objects misplaced (6.720%); 0 B/s, 38 objects/s recovering 2024-09-14T04:15:54.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:54.647 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:54.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:54.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:54.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:54.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:54.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:54.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:54.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:55.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:55.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:55.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:55.144 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:55.188 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:55.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:55.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:55.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:55.329 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:55.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:55.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:55 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9755/74883 objects degraded (13.027%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:55.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:55.484 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:55.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:55.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:55.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:55 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9755/74883 objects degraded (13.027%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:55.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:55 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9755/74883 objects degraded (13.027%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:15:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:55.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:55.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:55.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:55.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:55.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:55.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:55.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:55.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:55.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:55.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:55.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:56.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:56.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:56.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:56.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:56.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:56.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:56.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:56.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:56.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:56.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:56.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:56.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:56 smithi160 ceph-mon[88407]: pgmap v1008: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 9755/64407 objects degraded (15.146%); 5032/64407 objects misplaced (7.813%); 0 B/s, 38 objects/s recovering 2024-09-14T04:15:56.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:56.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:56.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:56 smithi114 ceph-mon[103029]: pgmap v1008: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 9755/64407 objects degraded (15.146%); 5032/64407 objects misplaced (7.813%); 0 B/s, 38 objects/s recovering 2024-09-14T04:15:56.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:56 smithi114 ceph-mon[106969]: pgmap v1008: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 9755/64407 objects degraded (15.146%); 5032/64407 objects misplaced (7.813%); 0 B/s, 38 objects/s recovering 2024-09-14T04:15:56.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:56.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:56.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:56.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:56.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:56.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:56.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:56.980 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:56.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:57.082 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:57.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:57.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:57.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:57.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:57.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:57.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:57.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:57.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:57.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:57.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:57.550 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:57.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:57.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:57.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:57.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:57.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:15:57.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:57.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:57.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:15:57.861 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:15:57.861 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:15:57.895 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:15:57.895 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:15:57.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:57.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:57.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:57.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:57.949 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:15:57.949 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:15:57.984 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:15:57.984 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:15:58.001 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:15:58.001 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:15:58.016 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:15:58.017 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:15:58.031 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:15:58.032 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:15:58.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:58.047 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:15:58.047 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:15:58.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:58.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:58.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:58.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:58.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:58.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:58.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:58.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:58.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:58.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:58 smithi114 ceph-mon[106969]: pgmap v1009: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 9755/64407 objects degraded (15.146%); 5032/64407 objects misplaced (7.813%); 0 B/s, 35 objects/s recovering 2024-09-14T04:15:58.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:58 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3280648996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:58 smithi114 ceph-mon[103029]: pgmap v1009: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 9755/64407 objects degraded (15.146%); 5032/64407 objects misplaced (7.813%); 0 B/s, 35 objects/s recovering 2024-09-14T04:15:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:58 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3280648996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:58.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:58.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:58.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:58.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:15:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:58.612 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:58.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:15:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:58 smithi160 ceph-mon[88407]: pgmap v1009: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 9755/64407 objects degraded (15.146%); 5032/64407 objects misplaced (7.813%); 0 B/s, 35 objects/s recovering 2024-09-14T04:15:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:58 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3280648996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:15:58.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:58.757 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:58.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:58.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:58.894 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:58.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:58.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:58.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:59.038 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:59.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:59.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:59.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:15:59.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:59.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:15:59.190 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:15:59.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:15:59.308 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:59.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:59.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:15:59.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:15:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:59.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:59.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:15:59.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:15:59.591 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:59.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:15:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:59 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:59 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:15:59 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1308794649' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:15:59.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:59 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:59 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:59.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:15:59 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1308794649' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:15:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:59 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:59 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:15:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:15:59 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1308794649' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:15:59.692 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:15:59.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:15:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:59.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:15:59.828 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:15:59.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:15:59.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:15:59.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:15:59.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:00.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:00.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:00.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:00.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:00.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:00.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:00.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:00.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:00.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:00 smithi114 ceph-mon[106969]: pgmap v1010: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 9505/57327 objects degraded (16.580%); 5032/57327 objects misplaced (8.778%); 0 B/s, 36 objects/s recovering 2024-09-14T04:16:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:00 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:16:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1308794649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:16:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:00 smithi114 ceph-mon[106969]: osdmap e271: 8 total, 8 up, 8 in 2024-09-14T04:16:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:00 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9505/57327 objects degraded (16.580%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:00 smithi114 ceph-mon[103029]: pgmap v1010: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 9505/57327 objects degraded (16.580%); 5032/57327 objects misplaced (8.778%); 0 B/s, 36 objects/s recovering 2024-09-14T04:16:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:00 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:16:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1308794649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:16:00.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:00 smithi114 ceph-mon[103029]: osdmap e271: 8 total, 8 up, 8 in 2024-09-14T04:16:00.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:00 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9505/57327 objects degraded (16.580%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:00.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:00.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:00.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:00.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:00.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:00.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:00.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:00.559 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:16:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:16:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:16:00.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:00 smithi160 ceph-mon[88407]: pgmap v1010: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 9505/57327 objects degraded (16.580%); 5032/57327 objects misplaced (8.778%); 0 B/s, 36 objects/s recovering 2024-09-14T04:16:00.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:00 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:16:00.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1308794649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:16:00.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:00 smithi160 ceph-mon[88407]: osdmap e271: 8 total, 8 up, 8 in 2024-09-14T04:16:00.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:00 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9505/57327 objects degraded (16.580%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:00.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:00.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:00.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:00.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:00.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:00.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:00.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:00.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:00.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:00.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:01.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:01.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:01.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:01.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:01.176 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:01.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:01.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:01.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:16:01.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:01.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:01.465 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:01.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:01.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:01.586 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:16:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:16:01] "GET /metrics HTTP/1.1" 200 36591 "" "Prometheus/2.51.0" 2024-09-14T04:16:01.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:01.645 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:01.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:01.746 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:01.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:01.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:01.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:01.919 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:01.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:02.020 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:02.052 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:02.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:02.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:02.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:02.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:02.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:02.200 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-14T04:16:02.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:02.334 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:02.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:02.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:02 smithi114 ceph-mon[106969]: pgmap v1012: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 827 op/s; 9505/52080 objects degraded (18.251%); 5032/52080 objects misplaced (9.662%); 0 B/s, 44 objects/s recovering 2024-09-14T04:16:02.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:02 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2950878158' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:16:02.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:02 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:16:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:02 smithi114 ceph-mon[103029]: pgmap v1012: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 827 op/s; 9505/52080 objects degraded (18.251%); 5032/52080 objects misplaced (9.662%); 0 B/s, 44 objects/s recovering 2024-09-14T04:16:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:02 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2950878158' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:16:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:02 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:16:02.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:02.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:02.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:02.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:16:02.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:02.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:02.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:02 smithi160 ceph-mon[88407]: pgmap v1012: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 827 op/s; 9505/52080 objects degraded (18.251%); 5032/52080 objects misplaced (9.662%); 0 B/s, 44 objects/s recovering 2024-09-14T04:16:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:02 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2950878158' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:16:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:02 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:16:02.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:02.738 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:02.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:02.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:02.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:02.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:02.976 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:03.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:03.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:03.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:03.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:03.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:03.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:03.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:03.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:03.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:03.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:03.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:03.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:03.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:03.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:03.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:03 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:16:03.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:03 smithi114 ceph-mon[106969]: osdmap e272: 8 total, 8 up, 8 in 2024-09-14T04:16:03.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:03 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:16:03.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:03 smithi114 ceph-mon[103029]: osdmap e272: 8 total, 8 up, 8 in 2024-09-14T04:16:03.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:03 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:16:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:03 smithi160 ceph-mon[88407]: osdmap e272: 8 total, 8 up, 8 in 2024-09-14T04:16:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:03.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:03.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:03.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:03.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:03.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:03.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:03.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:03.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:04.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:04.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:04.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:04.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:04.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:04.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:04.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:04.334 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:04.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:04.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:04.456 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:04.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:04.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:04.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:04.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:04.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:04 smithi114 ceph-mon[103029]: pgmap v1014: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 594 op/s; 9303/50139 objects degraded (18.554%); 5032/50139 objects misplaced (10.036%); 0 B/s, 56 objects/s recovering 2024-09-14T04:16:04.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:04 smithi114 ceph-mon[106969]: pgmap v1014: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 594 op/s; 9303/50139 objects degraded (18.554%); 5032/50139 objects misplaced (10.036%); 0 B/s, 56 objects/s recovering 2024-09-14T04:16:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:04 smithi160 ceph-mon[88407]: pgmap v1014: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 1.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 594 op/s; 9303/50139 objects degraded (18.554%); 5032/50139 objects misplaced (10.036%); 0 B/s, 56 objects/s recovering 2024-09-14T04:16:04.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:04.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:04.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:04.861 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:04.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:04.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:05.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:05.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:05.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:05.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:05.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:05.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:05.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:05.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:05.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:05.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:05.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:05.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:05.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:05.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:05.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:05 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9303/50139 objects degraded (18.554%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:05.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:05 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9303/50139 objects degraded (18.554%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:05.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:05 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9303/50139 objects degraded (18.554%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:05.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:05.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:05.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:05.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:05.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:05.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:05.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:05.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:06.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:06.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:06.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:06.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:06.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:06.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:06.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:06.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:06.387 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:06 smithi114 ceph-mon[103029]: pgmap v1015: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 827 MiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 9303/39750 objects degraded (23.404%); 5032/39750 objects misplaced (12.659%); 0 B/s, 56 objects/s recovering 2024-09-14T04:16:06.388 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:06 smithi114 ceph-mon[106969]: pgmap v1015: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 827 MiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 9303/39750 objects degraded (23.404%); 5032/39750 objects misplaced (12.659%); 0 B/s, 56 objects/s recovering 2024-09-14T04:16:06.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:06.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:06.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:06.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:06.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:06.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:06.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:06 smithi160 ceph-mon[88407]: pgmap v1015: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 827 MiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 9303/39750 objects degraded (23.404%); 5032/39750 objects misplaced (12.659%); 0 B/s, 56 objects/s recovering 2024-09-14T04:16:06.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-14T04:16:06.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:06.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:06.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:06.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:06.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:06.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:06.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:06.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:07.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:07.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:07.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:07.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:07.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:07.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:07.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:07.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:07.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:07.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:07.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:07.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:07.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:07.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:07.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:07.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:07.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:213733 11.16 11:6ecf8f74:::benchmark_data_smithi114_223345_object101796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:07.305014+0000", 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0061282020000000001, 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0061571780000000001, 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:07.816 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 213733 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.305014+0000", 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.305014+0000", 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:07.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.305016+0000", 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8819999999999999e-06 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.305023+0000", 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8199999999999999e-06 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:07.818 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.305037+0000", 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4514e-05 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.305046+0000", 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8289999999999997e-06 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.306796+0000", 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001750073 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.819 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.306918+0000", 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012181200000000001 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.307492+0000", 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00057388800000000005 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.820 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:213730 11.2 e272/243 11:41c43a7b:::benchmark_data_smithi114_223345_object101793:head v 272'8633, mlcod=272'8631)", 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:07.308614+0000", 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0025278869999999999, 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026247589999999999, 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.308614+0000", 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:07.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.308614+0000", 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.308615+0000", 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2160000000000001e-06 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.308621+0000", 2024-09-14T04:16:07.822 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3480000000000003e-06 2024-09-14T04:16:07.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.823 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:07.823 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.308631+0000", 2024-09-14T04:16:07.823 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0288e-05 2024-09-14T04:16:07.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.308638+0000", 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6030000000000001e-06 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.308703+0000", 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5383999999999996e-05 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.824 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.308722+0000", 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8712e-05 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.36273.0:213709 11.e e272/252)", 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:07.309148+0000", 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0019940829999999998, 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021347810000000001, 2024-09-14T04:16:07.825 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.309148+0000", 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.309148+0000", 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:07.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.309149+0000", 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7999999999999995e-07 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.309153+0000", 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0509999999999998e-06 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.309162+0000", 2024-09-14T04:16:07.827 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4509999999999998e-06 2024-09-14T04:16:07.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.828 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:07.828 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.309169+0000", 2024-09-14T04:16:07.828 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5699999999999998e-06 2024-09-14T04:16:07.828 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.828 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:07.828 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:213709 11.e 11:78ef4b79:::benchmark_data_smithi114_223345_object101772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:07.277685+0000", 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.033456881000000001, 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.033637533999999997, 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 213709 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.829 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.277685+0000", 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.277685+0000", 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:07.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.277686+0000", 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.221e-06 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.277690+0000", 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1370000000000004e-06 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.277701+0000", 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0983e-05 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.277708+0000", 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6379999999999998e-06 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.288944+0000", 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011235604 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.289080+0000", 2024-09-14T04:16:07.832 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000136694 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4,7", 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.289212+0000", 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013199300000000001 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.295979+0000", 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0067663869999999996 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:07.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.303464+0000", 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0074852510000000001 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:213723 11.6 11:64acbeb3:::benchmark_data_smithi114_223345_object101786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:07.294544+0000", 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.016597541, 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016845935999999999, 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:16:07.834 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 213723 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.294544+0000", 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.294544+0000", 2024-09-14T04:16:07.835 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.294546+0000", 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.299e-06 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.294551+0000", 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.775e-06 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.294563+0000", 2024-09-14T04:16:07.836 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2298e-05 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.294570+0000", 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0770000000000002e-06 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.294809+0000", 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023949099999999999 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.837 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.295083+0000", 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00027375300000000001 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:07.296062+0000", 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00097834099999999998 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:16:07.838 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 5 2024-09-14T04:16:07.839 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:16:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:07.840 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:07.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:07.941 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:07.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:07.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:08.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:08.126 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:08.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:08.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:08.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:08.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:08.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:08.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:08.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:08.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:08.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:08.532 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:08.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:08.632 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:08 smithi114 ceph-mon[103029]: pgmap v1016: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 827 MiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 732 op/s; 9303/39750 objects degraded (23.404%); 5032/39750 objects misplaced (12.659%); 0 B/s, 25 objects/s recovering 2024-09-14T04:16:08.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:08.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:08 smithi114 ceph-mon[106969]: pgmap v1016: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 827 MiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 732 op/s; 9303/39750 objects degraded (23.404%); 5032/39750 objects misplaced (12.659%); 0 B/s, 25 objects/s recovering 2024-09-14T04:16:08.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:08.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:08.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-14T04:16:08.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:08 smithi160 ceph-mon[88407]: pgmap v1016: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 827 MiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 732 op/s; 9303/39750 objects degraded (23.404%); 5032/39750 objects misplaced (12.659%); 0 B/s, 25 objects/s recovering 2024-09-14T04:16:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:08.669 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:08.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:08.729 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:08.739 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:08.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:08.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:08.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:08.828 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:08.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:08.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:08.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:09.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:09.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:09.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:09.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:09.251 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:09.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:09.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:09.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:09.380 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:09.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:09.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:09.511 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:09.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:09.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:09.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:09.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:09.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:09.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:09.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:09.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:09.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:09.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:09.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:09.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:10.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:10.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:10.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:10.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:10.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:10.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:10.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:10.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:10.400 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:10.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:10.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:10.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:10.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:10.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:10 smithi114 ceph-mon[106969]: pgmap v1017: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 679 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 778 op/s; 9026/32667 objects degraded (27.630%); 5032/32667 objects misplaced (15.404%); 0 B/s, 49 objects/s recovering 2024-09-14T04:16:10.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:10 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9026/32667 objects degraded (27.630%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:10 smithi114 ceph-mon[103029]: pgmap v1017: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 679 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 778 op/s; 9026/32667 objects degraded (27.630%); 5032/32667 objects misplaced (15.404%); 0 B/s, 49 objects/s recovering 2024-09-14T04:16:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:10 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9026/32667 objects degraded (27.630%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:10.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:16:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:16:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:16:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:10 smithi160 ceph-mon[88407]: pgmap v1017: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 679 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 778 op/s; 9026/32667 objects degraded (27.630%); 5032/32667 objects misplaced (15.404%); 0 B/s, 49 objects/s recovering 2024-09-14T04:16:10.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:10 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9026/32667 objects degraded (27.630%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:10.666 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:10.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:10.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:10.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:10.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:10.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:10.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:10.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:10.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:11.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:11.079 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:11.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:11.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:11.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:11.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:11.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:11.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:11.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:11.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:11.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:11.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:11.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:11.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:11.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:11.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:16:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:16:11] "GET /metrics HTTP/1.1" 200 36582 "" "Prometheus/2.51.0" 2024-09-14T04:16:11.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:11.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:11.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:11.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:11.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:11.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:12.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:12.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:12.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:12.142 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:12.166 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:12.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:12.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:12.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:12.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:12.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:12.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:12.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:12.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:12.584 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:12 smithi114 ceph-mon[103029]: pgmap v1018: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 568 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 9026/27342 objects degraded (33.011%); 5032/27342 objects misplaced (18.404%); 0 B/s, 47 objects/s recovering 2024-09-14T04:16:12.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:12 smithi114 ceph-mon[106969]: pgmap v1018: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 568 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 9026/27342 objects degraded (33.011%); 5032/27342 objects misplaced (18.404%); 0 B/s, 47 objects/s recovering 2024-09-14T04:16:12.592 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:12.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:12 smithi160 ceph-mon[88407]: pgmap v1018: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 568 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 9026/27342 objects degraded (33.011%); 5032/27342 objects misplaced (18.404%); 0 B/s, 47 objects/s recovering 2024-09-14T04:16:12.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:12.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:12.738 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:12.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:12.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:12.870 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:12.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:12.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:13.010 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:13.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:13.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:13.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:13.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:13.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:13.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:13.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:13.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:13.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:13.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:13.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:13.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:13.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:13.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:13.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:13.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:13.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:13.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:13.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:13.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:13.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:13.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:13.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:13.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:14.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:14.116 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:14.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:14.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:14.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:14.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:14.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:14.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:14.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:14.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:14.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:14.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:14 smithi114 ceph-mon[103029]: pgmap v1019: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 524 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 8789/25218 objects degraded (34.852%); 5032/25218 objects misplaced (19.954%); 0 B/s, 47 objects/s recovering 2024-09-14T04:16:14.550 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:14 smithi114 ceph-mon[106969]: pgmap v1019: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 524 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 8789/25218 objects degraded (34.852%); 5032/25218 objects misplaced (19.954%); 0 B/s, 47 objects/s recovering 2024-09-14T04:16:14.599 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:14.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:14 smithi160 ceph-mon[88407]: pgmap v1019: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 524 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 8789/25218 objects degraded (34.852%); 5032/25218 objects misplaced (19.954%); 0 B/s, 47 objects/s recovering 2024-09-14T04:16:14.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:14.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:14.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:14.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:14.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:14.857 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:14.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:14.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:14.959 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:14.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:14.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:15.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:15.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:15.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:15.138 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:16:15.139 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:16:15.187 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:16:15.187 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:16:15.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:15.223 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:16:15.223 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:16:15.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:15.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:15.254 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:16:15.255 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:16:15.270 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:16:15.293 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:16:15.293 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:16:15.329 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:16:15.330 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:16:15.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:15.365 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:16:15.365 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:16:15.390 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:15.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:15.399 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:16:15.399 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:16:15.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:15.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:15.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:15.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:15 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8789/25218 objects degraded (34.852%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2319334836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:16:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2800965475' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:16:15.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:15 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:16:15.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:15 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8789/25218 objects degraded (34.852%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:15.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2319334836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:16:15.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2800965475' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:16:15.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:15 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:16:15.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:15 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8789/25218 objects degraded (34.852%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2319334836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:16:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2800965475' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:16:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:15 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:16:15.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:15.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:15.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:15.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:15.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:15.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:15.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:15.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:16.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:16.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:16.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:16.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:16.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:16.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:16.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:16.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:16.373 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:16.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:16.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:16.510 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:16.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:16.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[103029]: pgmap v1020: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 292 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 8789/14076 objects degraded (62.440%); 5032/14076 objects misplaced (35.749%); 0 B/s, 42 objects/s recovering 2024-09-14T04:16:16.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:16:16.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:16:16.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[103029]: osdmap e273: 8 total, 8 up, 8 in 2024-09-14T04:16:16.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2484096335' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:16:16.586 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:16:16.586 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[106969]: pgmap v1020: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 292 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 8789/14076 objects degraded (62.440%); 5032/14076 objects misplaced (35.749%); 0 B/s, 42 objects/s recovering 2024-09-14T04:16:16.586 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:16:16.586 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:16:16.586 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[106969]: osdmap e273: 8 total, 8 up, 8 in 2024-09-14T04:16:16.586 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2484096335' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:16:16.586 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:16 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:16:16.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:16.655 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:16.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:16.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:16 smithi160 ceph-mon[88407]: pgmap v1020: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 292 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 8789/14076 objects degraded (62.440%); 5032/14076 objects misplaced (35.749%); 0 B/s, 42 objects/s recovering 2024-09-14T04:16:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:16 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:16:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:16 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:16:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:16 smithi160 ceph-mon[88407]: osdmap e273: 8 total, 8 up, 8 in 2024-09-14T04:16:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:16 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2484096335' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:16:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:16 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:16:16.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:16.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:16.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:16.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:16.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:16.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:16.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:17.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:17.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:17.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:17.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:17.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:17.218 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-14T04:16:17.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:17.350 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:17.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:17.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:17.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:17.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:17.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:17 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:16:17.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:17 smithi114 ceph-mon[106969]: osdmap e274: 8 total, 8 up, 8 in 2024-09-14T04:16:17.515 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:17 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:16:17.515 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:17 smithi114 ceph-mon[103029]: osdmap e274: 8 total, 8 up, 8 in 2024-09-14T04:16:17.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:17.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:17.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:17 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:16:17.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:17 smithi160 ceph-mon[88407]: osdmap e274: 8 total, 8 up, 8 in 2024-09-14T04:16:17.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:17.756 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:17.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:17.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:17.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:17.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:17.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:18.012 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:18.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:18.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:18.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:18.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:18.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:18.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:18.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:18.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:18.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:18.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:18.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:18.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:18.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:18.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:18.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:18.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:18 smithi114 ceph-mon[106969]: pgmap v1023: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 292 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 774 op/s; 8789/14076 objects degraded (62.440%); 5032/14076 objects misplaced (35.749%); 0 B/s, 29 objects/s recovering 2024-09-14T04:16:18.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:18 smithi114 ceph-mon[103029]: pgmap v1023: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 292 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 774 op/s; 8789/14076 objects degraded (62.440%); 5032/14076 objects misplaced (35.749%); 0 B/s, 29 objects/s recovering 2024-09-14T04:16:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:18 smithi160 ceph-mon[88407]: pgmap v1023: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 292 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 774 op/s; 8789/14076 objects degraded (62.440%); 5032/14076 objects misplaced (35.749%); 0 B/s, 29 objects/s recovering 2024-09-14T04:16:18.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:18.692 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:18.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:18.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:18.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:18.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:18.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:18.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:18.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:18.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:19.072 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 111934 objects 2024-09-14T04:16:19.072 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :125.807 2024-09-14T04:16:19.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:19.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:19.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:19.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:19.235 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:19.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:19.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:19.358 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:19.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:19.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:19.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:19.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:19.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:19.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:19.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:19.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:19.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:19.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:19.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:19.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:19.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:19.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:19.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:19.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:19.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:20.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:20.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:20.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:20.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:20.153 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:20.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:20.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:20.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:20.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:20.347 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:20.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:20.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:20.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:20 smithi114 ceph-mon[106969]: pgmap v1024: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 147 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 8509/7089 objects degraded (120.031%); 5032/7089 objects misplaced (70.983%); 0 B/s, 64 objects/s recovering 2024-09-14T04:16:20.417 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:20 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 8509/7089 objects degraded (120.031%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:20 smithi114 ceph-mon[103029]: pgmap v1024: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 147 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 8509/7089 objects degraded (120.031%); 5032/7089 objects misplaced (70.983%); 0 B/s, 64 objects/s recovering 2024-09-14T04:16:20.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:20 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 8509/7089 objects degraded (120.031%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:20.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:20.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:20.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:20.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:20.593 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:20.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:20 smithi160 ceph-mon[88407]: pgmap v1024: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 147 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 8509/7089 objects degraded (120.031%); 5032/7089 objects misplaced (70.983%); 0 B/s, 64 objects/s recovering 2024-09-14T04:16:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:20 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 8509/7089 objects degraded (120.031%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:16:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:16:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:16:20.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:20.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:20.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:20.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:20.839 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:20.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:20.941 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:20.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-14T04:16:20.962 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:20.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:21.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:21.085 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:21.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:21.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:21.208 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:21.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:21.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:21.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:21.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:21.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:21.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:21.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:21.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:21.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:21.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:21.577 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:16:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:16:21] "GET /metrics HTTP/1.1" 200 36582 "" "Prometheus/2.51.0" 2024-09-14T04:16:21.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:21.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:21.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:16:21.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:21.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:21.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:21.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:21.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:21.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:21.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:21.968 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:21.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:22.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:22.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:22.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:22.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:16:22.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:22.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:22.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:22.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:22.367 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:22.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:22.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:22.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:22.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:22.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:22.615 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:22.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:22.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:22.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:22 smithi114 ceph-mon[103029]: pgmap v1025: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 34 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 981 op/s; 8509/1668 objects degraded (510.132%); 5032/1668 objects misplaced (301.679%); 0 B/s, 34 objects/s recovering 2024-09-14T04:16:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:22 smithi114 ceph-mon[106969]: pgmap v1025: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 34 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 981 op/s; 8509/1668 objects degraded (510.132%); 5032/1668 objects misplaced (301.679%); 0 B/s, 34 objects/s recovering 2024-09-14T04:16:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:22 smithi160 ceph-mon[88407]: pgmap v1025: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 34 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 981 op/s; 8509/1668 objects degraded (510.132%); 5032/1668 objects misplaced (301.679%); 0 B/s, 34 objects/s recovering 2024-09-14T04:16:22.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:22.725 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:22.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:22.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:22.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:22.782 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-09-14T04:16:22.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-09-14T04:16:22.847 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:22.868 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:22.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:22.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:22.970 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:22.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:22.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:23.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:23.117 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:23.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:23.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:23.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:23.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:23.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:23.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:23.363 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:23.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:23.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:23.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:23.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:23.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:23.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:23.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:23 smithi114 ceph-mon[106969]: 11.14 scrub starts 2024-09-14T04:16:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:23 smithi114 ceph-mon[106969]: osdmap e275: 8 total, 8 up, 8 in 2024-09-14T04:16:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:23 smithi114 ceph-mon[103029]: 11.14 scrub starts 2024-09-14T04:16:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:23 smithi114 ceph-mon[103029]: osdmap e275: 8 total, 8 up, 8 in 2024-09-14T04:16:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:23 smithi160 ceph-mon[88407]: 11.14 scrub starts 2024-09-14T04:16:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:23 smithi160 ceph-mon[88407]: osdmap e275: 8 total, 8 up, 8 in 2024-09-14T04:16:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:23.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:23.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:23.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:23.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:23.858 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:23.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:23.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:23.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:23.981 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:23.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:24.082 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:24.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:24.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:24.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:24.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:24.228 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:24.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:24.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:24.350 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:24.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:24.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:24.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:24.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:24.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:24.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:24.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:24.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:24.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:24 smithi114 ceph-mon[103029]: pgmap v1027: 99 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 15 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 576 op/s; 7505/780 objects degraded (962.179%); 5032/780 objects misplaced (645.128%); 0 B/s, 167 objects/s recovering 2024-09-14T04:16:24.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:24 smithi114 ceph-mon[103029]: osdmap e276: 8 total, 8 up, 8 in 2024-09-14T04:16:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:24 smithi114 ceph-mon[106969]: pgmap v1027: 99 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 15 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 576 op/s; 7505/780 objects degraded (962.179%); 5032/780 objects misplaced (645.128%); 0 B/s, 167 objects/s recovering 2024-09-14T04:16:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:24 smithi114 ceph-mon[106969]: osdmap e276: 8 total, 8 up, 8 in 2024-09-14T04:16:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:24 smithi160 ceph-mon[88407]: pgmap v1027: 99 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 15 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 576 op/s; 7505/780 objects degraded (962.179%); 5032/780 objects misplaced (645.128%); 0 B/s, 167 objects/s recovering 2024-09-14T04:16:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:24 smithi160 ceph-mon[88407]: osdmap e276: 8 total, 8 up, 8 in 2024-09-14T04:16:24.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:24.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:24.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:24.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:24.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:24.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:24.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:24.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:24.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:25.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:25.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:25.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:25.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:25.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:25.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:25.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:25.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:25.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:25.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:25.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:25.460 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:25.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:25.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:25.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:25.584 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:25.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:25.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:25 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7505/780 objects degraded (962.179%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:25.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:25 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7505/780 objects degraded (962.179%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:25.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:25 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7505/780 objects degraded (962.179%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:25.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:25.707 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:25.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:25.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:25.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:25.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:25.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:25.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:25.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:26.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:26.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:26.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:26.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:26.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:26.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:26.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:26.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:26.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:26.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:26.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:26.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:26.447 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:26.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:26.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:26.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:26.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:26.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:26.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:26.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:26 smithi114 ceph-mon[103029]: pgmap v1029: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 575 op/s; 7505/255 objects degraded (2943.137%); 4737/255 objects misplaced (1857.647%); 0 B/s, 197 objects/s recovering 2024-09-14T04:16:26.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:26 smithi114 ceph-mon[103029]: 11.14 scrub starts 2024-09-14T04:16:26.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:26 smithi114 ceph-mon[106969]: pgmap v1029: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 575 op/s; 7505/255 objects degraded (2943.137%); 4737/255 objects misplaced (1857.647%); 0 B/s, 197 objects/s recovering 2024-09-14T04:16:26.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:26 smithi114 ceph-mon[106969]: 11.14 scrub starts 2024-09-14T04:16:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:26 smithi160 ceph-mon[88407]: pgmap v1029: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 575 op/s; 7505/255 objects degraded (2943.137%); 4737/255 objects misplaced (1857.647%); 0 B/s, 197 objects/s recovering 2024-09-14T04:16:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:26 smithi160 ceph-mon[88407]: 11.14 scrub starts 2024-09-14T04:16:26.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:26.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:26.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:26.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:26.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:26.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:26.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:26.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:26.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:26.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:27.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:27.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:27.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:27.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:27.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:27.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:27.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:27.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:27.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:27.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:27.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:27.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:27.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:27.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:27.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:27.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:27.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:27 smithi114 ceph-mon[103029]: 11.14 deep-scrub starts 2024-09-14T04:16:27.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:27 smithi114 ceph-mon[106969]: 11.14 deep-scrub starts 2024-09-14T04:16:27.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:27 smithi160 ceph-mon[88407]: 11.14 deep-scrub starts 2024-09-14T04:16:27.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:27.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:27.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:27.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:27.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:27.807 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:27.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:27.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:27.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:27.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:28.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:28.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:28.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:28.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:28.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:28.175 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:28.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:28.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:28.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:28.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:28.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:28.420 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:28.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:28.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:28.544 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:28.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:28.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:28 smithi114 ceph-mon[103029]: pgmap v1030: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 284 op/s; 7505/255 objects degraded (2943.137%); 4737/255 objects misplaced (1857.647%); 0 B/s, 162 objects/s recovering 2024-09-14T04:16:28.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:28 smithi114 ceph-mon[106969]: pgmap v1030: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 284 op/s; 7505/255 objects degraded (2943.137%); 4737/255 objects misplaced (1857.647%); 0 B/s, 162 objects/s recovering 2024-09-14T04:16:28.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:28 smithi160 ceph-mon[88407]: pgmap v1030: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 284 op/s; 7505/255 objects degraded (2943.137%); 4737/255 objects misplaced (1857.647%); 0 B/s, 162 objects/s recovering 2024-09-14T04:16:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:28.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:28.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:28.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:28.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:28.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:28.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:28.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:28.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:29.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:29.037 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:29.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:29.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:29.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:29.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:29.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:29.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:29.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:29.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:29.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:29.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:29.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:29.529 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:29.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:29.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:29 smithi114 ceph-mon[103029]: 11.14 deep-scrub starts 2024-09-14T04:16:29.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:29 smithi114 ceph-mon[106969]: 11.14 deep-scrub starts 2024-09-14T04:16:29.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:29.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:29.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:29.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:29 smithi160 ceph-mon[88407]: 11.14 deep-scrub starts 2024-09-14T04:16:29.753 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:29.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:29.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:29.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:29.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:29.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:30.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:30.027 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:30.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:30.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:30.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:30.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:30.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:30.275 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:30.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:30.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:30.399 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:30.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:30.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:30.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:30.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:30.558 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:16:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:16:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:16:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:30 smithi160 ceph-mon[88407]: pgmap v1031: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 58 op/s; 7505/255 objects degraded (2943.137%); 4737/255 objects misplaced (1857.647%); 0 B/s, 162 objects/s recovering 2024-09-14T04:16:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:30 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7505/255 objects degraded (2943.137%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:30.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:30 smithi114 ceph-mon[106969]: pgmap v1031: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 58 op/s; 7505/255 objects degraded (2943.137%); 4737/255 objects misplaced (1857.647%); 0 B/s, 162 objects/s recovering 2024-09-14T04:16:30.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:30 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7505/255 objects degraded (2943.137%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:30 smithi114 ceph-mon[103029]: pgmap v1031: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 58 op/s; 7505/255 objects degraded (2943.137%); 4737/255 objects misplaced (1857.647%); 0 B/s, 162 objects/s recovering 2024-09-14T04:16:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:30 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7505/255 objects degraded (2943.137%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:30.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:30.645 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:30.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:30.699 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:30.746 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:30.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:30.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:30.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:30.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:30.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:30.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:31.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:31.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:31.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:31.137 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:31.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:31.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:31.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:31.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:31.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:31.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:31.384 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:31.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:31.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:31.507 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:31.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:31.609 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:31.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:31.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:31.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:31.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:31.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:31.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:31.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:16:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:16:31] "GET /metrics HTTP/1.1" 200 36576 "" "Prometheus/2.51.0" 2024-09-14T04:16:31.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:31.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:31.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:32.000 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:32.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:32.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:32.123 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:32.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:32.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:32.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:32.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:32.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:32.374 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:32.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:32.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:32.498 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:32.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:32.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:32.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:32 smithi114 ceph-mon[106969]: pgmap v1032: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 20 op/s; 7505/255 objects degraded (2943.137%); 4073/255 objects misplaced (1597.255%); 0 B/s, 110 objects/s recovering 2024-09-14T04:16:32.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:32 smithi114 ceph-mon[103029]: pgmap v1032: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 20 op/s; 7505/255 objects degraded (2943.137%); 4073/255 objects misplaced (1597.255%); 0 B/s, 110 objects/s recovering 2024-09-14T04:16:32.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:32.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:32 smithi160 ceph-mon[88407]: pgmap v1032: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 20 op/s; 7505/255 objects degraded (2943.137%); 4073/255 objects misplaced (1597.255%); 0 B/s, 110 objects/s recovering 2024-09-14T04:16:32.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:32.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:32.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:32.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:32.872 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:32.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:32.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:32.996 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:32.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:32.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:33.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:33.119 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:33.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:33.221 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:33.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:33.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:33.344 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:33.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:33.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:33.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:33.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:33.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:33.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:33.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:33.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:33.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:33.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:33.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:33.740 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:33.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:33.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:33.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:33.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:33 smithi114 ceph-mon[103029]: 11.14 deep-scrub starts 2024-09-14T04:16:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:33 smithi114 ceph-mon[103029]: pgmap v1033: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 17 op/s; 7505/255 objects degraded (2943.137%); 4073/255 objects misplaced (1597.255%); 0 B/s, 95 objects/s recovering 2024-09-14T04:16:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:33 smithi114 ceph-mon[106969]: 11.14 deep-scrub starts 2024-09-14T04:16:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:33 smithi114 ceph-mon[106969]: pgmap v1033: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 17 op/s; 7505/255 objects degraded (2943.137%); 4073/255 objects misplaced (1597.255%); 0 B/s, 95 objects/s recovering 2024-09-14T04:16:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:33 smithi160 ceph-mon[88407]: 11.14 deep-scrub starts 2024-09-14T04:16:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:33.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:33 smithi160 ceph-mon[88407]: pgmap v1033: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 17 op/s; 7505/255 objects degraded (2943.137%); 4073/255 objects misplaced (1597.255%); 0 B/s, 95 objects/s recovering 2024-09-14T04:16:33.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:33.990 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:33.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:33.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:34.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:34.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:34.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:34.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:34.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:34.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:34.339 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:34.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:34.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:34.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:34.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:34.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:34.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:34.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:34.607 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:34.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:34.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:34.731 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:34.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:34.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:34.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:34.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:34.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:34.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:34.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:34.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:34.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:35.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:35.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:35.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:35.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:35.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:35.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:35.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:35.348 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:35.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:35.403 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:16:35.404 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:16:35.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:35.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:35.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:35.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:35.595 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:35.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:35.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:35.720 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:35.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:35.821 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:35.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:35.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:35.967 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:35.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:36.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:36.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:36.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:36.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:36.214 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:36.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:36.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:36.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:36.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:36.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:36.460 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:36.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:36.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:36.572 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:36.596 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:36.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:36.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:36.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:36.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:36.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:36.821 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:36.843 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:36.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:36.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:36 smithi114 ceph-mon[103029]: pgmap v1034: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 15 op/s; 7505/255 objects degraded (2943.137%); 3358/255 objects misplaced (1316.863%); 0 B/s, 143 objects/s recovering 2024-09-14T04:16:36.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:36 smithi114 ceph-mon[103029]: 11.14 scrub starts 2024-09-14T04:16:36.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:36 smithi114 ceph-mon[106969]: pgmap v1034: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 15 op/s; 7505/255 objects degraded (2943.137%); 3358/255 objects misplaced (1316.863%); 0 B/s, 143 objects/s recovering 2024-09-14T04:16:36.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:36 smithi114 ceph-mon[106969]: 11.14 scrub starts 2024-09-14T04:16:36.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:36 smithi160 ceph-mon[88407]: pgmap v1034: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 15 op/s; 7505/255 objects degraded (2943.137%); 3358/255 objects misplaced (1316.863%); 0 B/s, 143 objects/s recovering 2024-09-14T04:16:36.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:36 smithi160 ceph-mon[88407]: 11.14 scrub starts 2024-09-14T04:16:36.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:36.967 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:36.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:36.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:37.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:37.090 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:37.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:37.192 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:37.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:37.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:37.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:37.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:37.337 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:37.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:37.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:37.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:37.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:37.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:37.584 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:37.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:37.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:16:37.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:16:37.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:37.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:16:37.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:16:37.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:37.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:37.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:37.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:37.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:37.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:37.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:16:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:16:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:37.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:37.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:37.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:38.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:38.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:38.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:38.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:38.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:38.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:38.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:38.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:38.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:38.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:38.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:38.446 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:38.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:38.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:38.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:38.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:38.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:38.695 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:38.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:38.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:38 smithi114 ceph-mon[103029]: pgmap v1035: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 3358/255 objects misplaced (1316.863%); 0 B/s, 114 objects/s recovering 2024-09-14T04:16:38.796 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:38.797 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:38 smithi114 ceph-mon[106969]: pgmap v1035: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 3358/255 objects misplaced (1316.863%); 0 B/s, 114 objects/s recovering 2024-09-14T04:16:38.797 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:38.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:38.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:38.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:38 smithi160 ceph-mon[88407]: pgmap v1035: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 3358/255 objects misplaced (1316.863%); 0 B/s, 114 objects/s recovering 2024-09-14T04:16:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:38.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:38.943 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:38.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:39.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:39.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:39.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:39.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:39.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:39.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:39.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:39.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:39.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:39.336 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:39.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:39.438 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:39.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:39.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:39.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:39.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:39.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:39.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:39.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:39.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:39.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:39.808 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:39.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:39.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-14T04:16:39.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:39.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:39.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:40.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:40.086 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:40.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:40.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:40.209 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:40.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:40.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:40.332 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:40.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:40.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:40.456 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:40.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:40.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:40.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:40.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:40 smithi160 ceph-mon[88407]: pgmap v1036: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 3358/255 objects misplaced (1316.863%); 0 B/s, 114 objects/s recovering 2024-09-14T04:16:40.659 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:16:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:16:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:16:40.684 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:40.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:40.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:40.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:40.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:40.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:40.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:40 smithi114 ceph-mon[103029]: pgmap v1036: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 3358/255 objects misplaced (1316.863%); 0 B/s, 114 objects/s recovering 2024-09-14T04:16:40.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:40 smithi114 ceph-mon[106969]: pgmap v1036: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 3358/255 objects misplaced (1316.863%); 0 B/s, 114 objects/s recovering 2024-09-14T04:16:40.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:40.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:40.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:41.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:41.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:41.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:41.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:41.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:41.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:41.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:41.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:41.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:41.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:41.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:41.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:41.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:41.518 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:41 smithi114 ceph-mon[103029]: 11.14 scrub starts 2024-09-14T04:16:41.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:41 smithi114 ceph-mon[106969]: 11.14 scrub starts 2024-09-14T04:16:41.519 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:16:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:16:41] "GET /metrics HTTP/1.1" 200 36577 "" "Prometheus/2.51.0" 2024-09-14T04:16:41.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:41.573 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:41.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:41.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:41.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:41.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:41.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:41.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:41.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:41 smithi160 ceph-mon[88407]: 11.14 scrub starts 2024-09-14T04:16:41.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:41.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:41.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:42.038 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:16:42.039 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:16:42.039 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:16:42.039 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:16:42.039 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:16:42.039 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:16:42.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:42.050 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:16:42.051 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:16:42.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:42.067 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:16:42.067 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:16:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:42.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:42.086 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:16:42.086 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:16:42.104 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:16:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:16:42.122 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:16:42.122 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:16:42.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:42.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:42.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:42.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:42.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:42.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:42.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:42.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:42.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:42.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:42.458 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-14T04:16:42.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:42.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:42.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:42.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:42 smithi114 ceph-mon[103029]: pgmap v1037: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 2485/255 objects misplaced (974.510%); 0 B/s, 187 objects/s recovering 2024-09-14T04:16:42.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2669488154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:16:42.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/562574830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:16:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:42 smithi114 ceph-mon[106969]: pgmap v1037: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 2485/255 objects misplaced (974.510%); 0 B/s, 187 objects/s recovering 2024-09-14T04:16:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2669488154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:16:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/562574830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:16:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:42 smithi160 ceph-mon[88407]: pgmap v1037: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 2485/255 objects misplaced (974.510%); 0 B/s, 187 objects/s recovering 2024-09-14T04:16:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2669488154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:16:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/562574830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:16:42.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:42.686 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:42.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:42.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:42.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:42.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:42.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:42.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:42.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:42.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:43.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:43.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:43.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:43.160 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:43.185 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:43.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:43.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:43.308 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:43.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:43.409 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:43.431 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:43.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:43.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:43.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:43.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:43.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:43 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2669488154' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:16:43.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:43 smithi114 ceph-mon[106969]: osdmap e277: 8 total, 8 up, 8 in 2024-09-14T04:16:43.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:43 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2669488154' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:16:43.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:43 smithi114 ceph-mon[103029]: osdmap e277: 8 total, 8 up, 8 in 2024-09-14T04:16:43.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:43.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:43.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:43.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:43.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:43.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:43.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:43.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:43.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:43 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2669488154' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:16:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:43 smithi160 ceph-mon[88407]: osdmap e277: 8 total, 8 up, 8 in 2024-09-14T04:16:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:43.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:43.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:44.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:44.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:44.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:44.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:44.171 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:44.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:44.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:44.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:44.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:44.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:44.418 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:44.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:44.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:44.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:44.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:44.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:44.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:44.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:44.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:44.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:44.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:44.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:44 smithi114 ceph-mon[103029]: pgmap v1039: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 2485/255 objects misplaced (974.510%); 0 B/s, 158 objects/s recovering 2024-09-14T04:16:44.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:44 smithi114 ceph-mon[103029]: 11.14 scrub starts 2024-09-14T04:16:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:44 smithi114 ceph-mon[106969]: pgmap v1039: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 2485/255 objects misplaced (974.510%); 0 B/s, 158 objects/s recovering 2024-09-14T04:16:44.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:44 smithi114 ceph-mon[106969]: 11.14 scrub starts 2024-09-14T04:16:44.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:44 smithi160 ceph-mon[88407]: pgmap v1039: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 2485/255 objects misplaced (974.510%); 0 B/s, 158 objects/s recovering 2024-09-14T04:16:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:44 smithi160 ceph-mon[88407]: 11.14 scrub starts 2024-09-14T04:16:44.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:44.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:44.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:45.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:45.035 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:45.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:45.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:45.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:45.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:45.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:45.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:45.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:45.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:45.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:45.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:45.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:45.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:45.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:45.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:45.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:45.650 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:45.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:45.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:45.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:45.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:45.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:45 smithi114 ceph-mon[103029]: 11.14 scrub starts 2024-09-14T04:16:45.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:45 smithi114 ceph-mon[106969]: 11.14 scrub starts 2024-09-14T04:16:45.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:45.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:45.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:45.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:45.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:45 smithi160 ceph-mon[88407]: 11.14 scrub starts 2024-09-14T04:16:46.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:46.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:46.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:46.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:46.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:46.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:46.270 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:46.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:46.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:46.393 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:46.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:46.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:46.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:46.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:46.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:46.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:46.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:46.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:46.763 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:46.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:46.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:46.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:46 smithi114 ceph-mon[103029]: pgmap v1040: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 1065/255 objects misplaced (417.647%); 0 B/s, 229 objects/s recovering 2024-09-14T04:16:46.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:46 smithi114 ceph-mon[106969]: pgmap v1040: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 1065/255 objects misplaced (417.647%); 0 B/s, 229 objects/s recovering 2024-09-14T04:16:46.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:46.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:46.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:46 smithi160 ceph-mon[88407]: pgmap v1040: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 1065/255 objects misplaced (417.647%); 0 B/s, 229 objects/s recovering 2024-09-14T04:16:46.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:47.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:47.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:47.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:47.135 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:47.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:47.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:47.256 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:47.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:47.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:47.380 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:47.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:47.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:47.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-14T04:16:47.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:47.514 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:47.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:47.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:47.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:47.637 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:47.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:47.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:47.760 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:47.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:47.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:47.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:47.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:47.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:48.008 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:48.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:48.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:48.131 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:48.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:48.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:48.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:48.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:48.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:48.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:48.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:48.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:48.506 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:48.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:48.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:48.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:48.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:48.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:48.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:48.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:48.840 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:48 smithi114 ceph-mon[106969]: pgmap v1041: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 1065/255 objects misplaced (417.647%); 0 B/s, 229 objects/s recovering 2024-09-14T04:16:48.840 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:48 smithi114 ceph-mon[106969]: 11.14 scrub starts 2024-09-14T04:16:48.840 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:48 smithi114 ceph-mon[106969]: osdmap e278: 8 total, 8 up, 8 in 2024-09-14T04:16:48.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:48 smithi114 ceph-mon[103029]: pgmap v1041: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 1065/255 objects misplaced (417.647%); 0 B/s, 229 objects/s recovering 2024-09-14T04:16:48.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:48 smithi114 ceph-mon[103029]: 11.14 scrub starts 2024-09-14T04:16:48.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:48 smithi114 ceph-mon[103029]: osdmap e278: 8 total, 8 up, 8 in 2024-09-14T04:16:48.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:48.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:48 smithi160 ceph-mon[88407]: pgmap v1041: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7505/255 objects degraded (2943.137%); 1065/255 objects misplaced (417.647%); 0 B/s, 229 objects/s recovering 2024-09-14T04:16:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:48 smithi160 ceph-mon[88407]: 11.14 scrub starts 2024-09-14T04:16:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:48 smithi160 ceph-mon[88407]: osdmap e278: 8 total, 8 up, 8 in 2024-09-14T04:16:48.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:49.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:49.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:49.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:49.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:49.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:49.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:49.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:49.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:49.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:49.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:49.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:49.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:49.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:49.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:49.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:49.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:49.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:49.518 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:49.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:49.619 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:49.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:49.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:49.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:49.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:49.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:49.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:49.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:49 smithi114 ceph-mon[103029]: osdmap e279: 8 total, 8 up, 8 in 2024-09-14T04:16:49.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:49 smithi114 ceph-mon[106969]: osdmap e279: 8 total, 8 up, 8 in 2024-09-14T04:16:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:49 smithi160 ceph-mon[88407]: osdmap e279: 8 total, 8 up, 8 in 2024-09-14T04:16:49.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:49.990 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:49.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:50.091 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:50.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:50.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:50.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:50.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:50.236 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:50.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:50.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:50.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:50.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:50.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:50.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:50.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:50.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:50.502 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:16:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:16:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:16:50.508 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:16:50.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:50.592 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:50 smithi114 ceph-mon[106969]: pgmap v1044: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7381/255 objects degraded (2894.510%); 1065/255 objects misplaced (417.647%); 0 B/s, 237 objects/s recovering 2024-09-14T04:16:50.592 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:50 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7381/255 objects degraded (2894.510%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:50.592 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:50 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/632124677' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:16:50.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:50 smithi114 ceph-mon[103029]: pgmap v1044: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7381/255 objects degraded (2894.510%); 1065/255 objects misplaced (417.647%); 0 B/s, 237 objects/s recovering 2024-09-14T04:16:50.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:50 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7381/255 objects degraded (2894.510%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:50.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:50 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/632124677' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:16:50.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:50.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:50.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:50.733 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:50.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:50.835 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:50.859 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:50.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:50 smithi160 ceph-mon[88407]: pgmap v1044: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 4.3 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 7381/255 objects degraded (2894.510%); 1065/255 objects misplaced (417.647%); 0 B/s, 237 objects/s recovering 2024-09-14T04:16:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:50 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7381/255 objects degraded (2894.510%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:16:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:50 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/632124677' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:16:50.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:50.982 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:50.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:50.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:51.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:51.105 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:51.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:51.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:51.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:51.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:51.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:51.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:51.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:51.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:51.475 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:51.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:51.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:51.502 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:16:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:16:51] "GET /metrics HTTP/1.1" 200 36577 "" "Prometheus/2.51.0" 2024-09-14T04:16:51.537 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:51.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:51.598 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:51.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:51.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:51.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:51.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:51.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:51.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:51 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:16:51.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/632124677' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:16:51.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:51 smithi114 ceph-mon[103029]: osdmap e280: 8 total, 8 up, 8 in 2024-09-14T04:16:51.809 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:51 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:16:51.809 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:51 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/632124677' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:16:51.809 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:51 smithi114 ceph-mon[106969]: osdmap e280: 8 total, 8 up, 8 in 2024-09-14T04:16:51.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:51.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:51.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:51.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:51 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:16:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:51 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/632124677' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:16:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:51 smithi160 ceph-mon[88407]: osdmap e280: 8 total, 8 up, 8 in 2024-09-14T04:16:51.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:51.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:51.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:52.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:52.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:16:52.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:52.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:52.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:52.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:52.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:52.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:52.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:52.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:52.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:52.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:52.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:52.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:52.515 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_6' removed 2024-09-14T04:16:52.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:52.592 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:52.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:52.693 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:52.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:52.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:52.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:52.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:52.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:52.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:52 smithi114 ceph-mon[103029]: pgmap v1046: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 95 active+clean; 4.3 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 7381/255 objects degraded (2894.510%); 1065/255 objects misplaced (417.647%); 0 B/s, 20 objects/s recovering 2024-09-14T04:16:52.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:52 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/731281065' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:16:52.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:52 smithi114 ceph-mon[106969]: pgmap v1046: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 95 active+clean; 4.3 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 7381/255 objects degraded (2894.510%); 1065/255 objects misplaced (417.647%); 0 B/s, 20 objects/s recovering 2024-09-14T04:16:52.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:52 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/731281065' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:16:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:52 smithi160 ceph-mon[88407]: pgmap v1046: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 95 active+clean; 4.3 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 7381/255 objects degraded (2894.510%); 1065/255 objects misplaced (417.647%); 0 B/s, 20 objects/s recovering 2024-09-14T04:16:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:52 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/731281065' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:16:52.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:52.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:52.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:53.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:53.085 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:53.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:53.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:53.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:53.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:53.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:53.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:53.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:53.435 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:53.457 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:53.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:53.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:53.580 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:53.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:53.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:53.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:53.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:53.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:53.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:53.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:53.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/731281065' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:16:53.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:53 smithi114 ceph-mon[103029]: osdmap e281: 8 total, 8 up, 8 in 2024-09-14T04:16:53.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/731281065' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:16:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:53 smithi114 ceph-mon[106969]: osdmap e281: 8 total, 8 up, 8 in 2024-09-14T04:16:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/731281065' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:16:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:53 smithi160 ceph-mon[88407]: osdmap e281: 8 total, 8 up, 8 in 2024-09-14T04:16:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:16:53.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:53.952 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:53.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:54.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:54.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:54.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:54.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:54.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:54.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:54.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:54.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:54.328 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:54.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:54.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:54.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:54.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:54.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:54.580 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:54.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:54.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:54.704 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:54.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:54.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:54.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:54.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:54.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:54 smithi114 ceph-mon[106969]: pgmap v1048: 73 pgs: 73 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:16:54.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:54 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7381/255 objects degraded (2894.510%), 2 pgs degraded, 2 pgs undersized) 2024-09-14T04:16:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:54 smithi114 ceph-mon[103029]: pgmap v1048: 73 pgs: 73 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:16:54.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:54 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7381/255 objects degraded (2894.510%), 2 pgs degraded, 2 pgs undersized) 2024-09-14T04:16:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:54 smithi160 ceph-mon[88407]: pgmap v1048: 73 pgs: 73 active+clean; 582 KiB data, 6.9 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:16:54.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:54 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7381/255 objects degraded (2894.510%), 2 pgs degraded, 2 pgs undersized) 2024-09-14T04:16:54.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:54.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:54.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:54.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:55.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:55.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:55.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:55.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:55.214 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:55.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:55.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:55.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:55.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:55.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:55.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:55.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:55.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:55.596 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:55.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:55.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:55.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:55.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:55.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:55.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:55.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:55.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:55.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:55.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:56.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:56.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:56.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:56.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:56.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:56.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:56.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:56.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:56.353 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:56.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:56.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:56.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:56.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:56.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:56.602 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:56.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:56.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:56.730 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:56.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:56.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:56.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:56.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:56.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:56 smithi114 ceph-mon[103029]: pgmap v1049: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:16:56.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:56 smithi114 ceph-mon[106969]: pgmap v1049: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:16:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:56 smithi160 ceph-mon[88407]: pgmap v1049: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:16:56.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:56.979 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:56.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:57.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:57.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:57.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:57.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:57.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:57.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:57.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:57.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:57.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:57.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:57.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:57.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:57.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:57.608 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:57.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:57.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:57.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:57.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:57.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:57.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:57.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:16:57.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:57.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:57.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:57.925 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-14T04:16:57.926 INFO:tasks.radosbench:Beginning radosbench... 2024-09-14T04:16:57.926 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-09-14T04:16:57.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-09-14T04:16:58.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:58.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:58.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:58.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:58.159 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:58.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:58.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:58.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:58.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:58.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:58.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:58.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:58.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:16:58.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:58.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:16:58.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:58.659 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:58.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:58.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:16:58.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:58.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:16:58.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:58 smithi114 ceph-mon[103029]: pgmap v1050: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:16:58.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:58 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:58.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:58 smithi114 ceph-mon[106969]: pgmap v1050: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:16:58.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:58 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:58.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:58 smithi160 ceph-mon[88407]: pgmap v1050: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:16:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:58 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:16:58.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:58.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:58.964 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:16:58.971 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:16:58.971 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:16:58.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:16:58.971 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:16:58.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.971 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80126 11.e 11:731aef16:::benchmark_data_smithi114_223345_object80125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-09-14T04:16:58.971 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:37.150850+0000", 2024-09-14T04:16:58.971 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 201.817658959, 2024-09-14T04:16:58.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1726886460000001, 2024-09-14T04:16:58.972 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:58.972 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:58.972 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:58.972 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:58.973 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:58.973 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:58.973 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80126 2024-09-14T04:16:58.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.973 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:58.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:58.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150850+0000", 2024-09-14T04:16:58.973 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:58.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150850+0000", 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150851+0000", 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7370000000000001e-06 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:58.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150878+0000", 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6801e-05 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150897+0000", 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8513e-05 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150907+0000", 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0735e-05 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151518+0000", 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00061008199999999996 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op must wait for map", 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151525+0000", 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.712e-06 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154788+0000", 2024-09-14T04:16:58.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003262881 2024-09-14T04:16:58.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.977 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.977 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:16:58.977 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154798+0000", 2024-09-14T04:16:58.979 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4730000000000006e-06 2024-09-14T04:16:58.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.322040+0000", 2024-09-14T04:16:58.980 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.167242731 2024-09-14T04:16:58.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:58.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.323538+0000", 2024-09-14T04:16:58.980 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001497981 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80128 11.e 11:755cc570:::benchmark_data_smithi114_223345_object80127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:37.150947+0000", 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 201.817561372, 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.17548607, 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:58.981 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80128 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150947+0000", 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150947+0000", 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150948+0000", 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0999999999999998e-07 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150970+0000", 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2118000000000001e-05 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:58.983 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150979+0000", 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1949999999999992e-06 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150985+0000", 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0950000000000001e-06 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151577+0000", 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00059211099999999998 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.984 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:16:58.985 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151582+0000", 2024-09-14T04:16:58.985 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8930000000000004e-06 2024-09-14T04:16:58.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.985 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.985 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.985 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154836+0000", 2024-09-14T04:16:58.985 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003253742 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154844+0000", 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9580000000000007e-06 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.326316+0000", 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1714720439999999 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.986 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.326433+0000", 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000117304 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80131 11.e 11:76a0632e:::benchmark_data_smithi114_223345_object80130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:37.150996+0000", 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 201.817512773, 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.175671168, 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:58.987 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80131 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150996+0000", 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:58.989 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150996+0000", 2024-09-14T04:16:58.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:58.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.989 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:58.989 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.150996+0000", 2024-09-14T04:16:58.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9399999999999995e-07 2024-09-14T04:16:58.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.989 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:58.990 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151019+0000", 2024-09-14T04:16:58.990 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2504e-05 2024-09-14T04:16:58.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.990 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151027+0000", 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.763e-06 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151033+0000", 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.942e-06 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151605+0000", 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00057287600000000005 2024-09-14T04:16:58.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151610+0000", 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4229999999999998e-06 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154866+0000", 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003256508 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.992 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:16:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154872+0000", 2024-09-14T04:16:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1349999999999997e-06 2024-09-14T04:16:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.326464+0000", 2024-09-14T04:16:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1715919260000001 2024-09-14T04:16:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.326667+0000", 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000202597 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80140 11.e 11:768f1fec:::benchmark_data_smithi114_223345_object80139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:37.151041+0000", 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 201.81746759399999, 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.175727258, 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:58.994 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80140 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151041+0000", 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151041+0000", 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151041+0000", 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8299999999999997e-07 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:58.996 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151061+0000", 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9788000000000001e-05 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151069+0000", 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3420000000000002e-06 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151074+0000", 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6049999999999997e-06 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151634+0000", 2024-09-14T04:16:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00055994900000000002 2024-09-14T04:16:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:16:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151638+0000", 2024-09-14T04:16:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8659999999999999e-06 2024-09-14T04:16:58.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154895+0000", 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032565659999999998 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154901+0000", 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5309999999999998e-06 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:58.999 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.326694+0000", 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.171793342 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.326768+0000", 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3786000000000001e-05 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.000 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80152 11.e 11:7967396e:::benchmark_data_smithi114_223345_object80151:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:37.151082+0000", 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 201.81742659400001, 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.205312894, 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80152 2024-09-14T04:16:59.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151082+0000", 2024-09-14T04:16:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151082+0000", 2024-09-14T04:16:59.002 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151082+0000", 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2899999999999999e-07 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151249+0000", 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016685800000000001 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151258+0000", 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2029999999999998e-06 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151265+0000", 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5509999999999996e-06 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.004 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151661+0000", 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00039585899999999998 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151665+0000", 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5290000000000002e-06 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154922+0000", 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032563140000000002 2024-09-14T04:16:59.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154928+0000", 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1199999999999999e-06 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.326795+0000", 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.171867515 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:16:59.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.327623+0000", 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00082731099999999995 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4,7", 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.327778+0000", 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000155167 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.337153+0000", 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0093751640000000001 2024-09-14T04:16:59.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.008 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.337207+0000", 2024-09-14T04:16:59.008 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4438000000000003e-05 2024-09-14T04:16:59.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.008 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.343087+0000", 2024-09-14T04:16:59.008 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0058794349999999997 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.353276+0000", 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010189559000000001 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.353349+0000", 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2280000000000006e-05 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.356395+0000", 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030461619999999998 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80168 11.e 11:7a8839b7:::benchmark_data_smithi114_223345_object80167:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:37.151276+0000", 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 201.81723228000001, 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.207934303, 2024-09-14T04:16:59.010 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.011 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.011 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:16:59.011 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:59.011 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:59.011 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:59.011 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80168 2024-09-14T04:16:59.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.011 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151276+0000", 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151276+0000", 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151277+0000", 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6199999999999998e-07 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151298+0000", 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1030999999999999e-05 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151306+0000", 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8320000000000006e-06 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151312+0000", 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5250000000000002e-06 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151687+0000", 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00037426199999999999 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:16:59.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151690+0000", 2024-09-14T04:16:59.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9500000000000003e-06 2024-09-14T04:16:59.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154949+0000", 2024-09-14T04:16:59.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032580600000000001 2024-09-14T04:16:59.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154956+0000", 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5769999999999996e-06 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.330424+0000", 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1754679320000001 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.330673+0000", 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024906 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4,7", 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.330940+0000", 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00026656999999999999 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.337160+0000", 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062199730000000002 2024-09-14T04:16:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.337270+0000", 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010990800000000001 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.347824+0000", 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010554440999999999 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.356452+0000", 2024-09-14T04:16:59.018 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0086275039999999994 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.356517+0000", 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5457999999999996e-05 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.359211+0000", 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026936579999999998 2024-09-14T04:16:59.019 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80179 11.e 11:7e6d2295:::benchmark_data_smithi114_223345_object80178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:37.151321+0000", 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 201.81718769700001, 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.211150409, 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:59.020 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80179 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151321+0000", 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151321+0000", 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151321+0000", 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0800000000000005e-07 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151381+0000", 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9116000000000001e-05 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151388+0000", 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9559999999999997e-06 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151394+0000", 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.998e-06 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151711+0000", 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00031691100000000002 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151716+0000", 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1629999999999998e-06 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154978+0000", 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032622520000000002 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.154984+0000", 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.224e-06 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.333734+0000", 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1787502780000001 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.333880+0000", 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014581699999999999 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4,7", 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.334038+0000", 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000158157 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.337328+0000", 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003289939 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.337343+0000", 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.436e-05 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.347878+0000", 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010534989999999999 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.359673+0000", 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.01179593 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.359738+0000", 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4034000000000004e-05 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.362471+0000", 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027337759999999998 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.028 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80189 11.e 11:7e8ee3b5:::benchmark_data_smithi114_223345_object80188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e252)", 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:37.151404+0000", 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 201.81710441800001, 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.215781827, 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80189 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151404+0000", 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151404+0000", 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151405+0000", 2024-09-14T04:16:59.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5799999999999999e-07 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151449+0000", 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4079999999999998e-05 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151457+0000", 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9610000000000005e-06 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151463+0000", 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9889999999999997e-06 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151737+0000", 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00027395700000000002 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting_for_map not empty", 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.151741+0000", 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.155e-06 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.155007+0000", 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032660409999999999 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.155013+0000", 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9089999999999997e-06 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.337389+0000", 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1823764699999999 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.337553+0000", 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000164105 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4,7", 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.337732+0000", 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017908600000000001 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.343138+0000", 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054053950000000003 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.343152+0000", 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4345000000000001e-05 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.347927+0000", 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0047744320000000003 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.362533+0000", 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014606397 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.362598+0000", 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5154999999999995e-05 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.367186+0000", 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0045877920000000003 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.036 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.037 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206308 11.6 11:63734b78:::benchmark_data_smithi114_223345_object94371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:16:59.037 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.960890+0000", 2024-09-14T04:16:59.037 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 61.007618602999997, 2024-09-14T04:16:59.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.33796224600000002, 2024-09-14T04:16:59.037 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.037 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.038 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:16:59.038 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:16:59.038 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:16:59.038 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:16:59.038 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206308 2024-09-14T04:16:59.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.038 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960890+0000", 2024-09-14T04:16:59.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960890+0000", 2024-09-14T04:16:59.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960890+0000", 2024-09-14T04:16:59.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8599999999999999e-07 2024-09-14T04:16:59.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960892+0000", 2024-09-14T04:16:59.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.074e-06 2024-09-14T04:16:59.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960898+0000", 2024-09-14T04:16:59.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8900000000000004e-06 2024-09-14T04:16:59.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.042 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960903+0000", 2024-09-14T04:16:59.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6430000000000003e-06 2024-09-14T04:16:59.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.042 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.042 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.964467+0000", 2024-09-14T04:16:59.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035639030000000002 2024-09-14T04:16:59.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:16:59.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.964568+0000", 2024-09-14T04:16:59.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000101437 2024-09-14T04:16:59.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:16:59.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.972215+0000", 2024-09-14T04:16:59.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0076467030000000004 2024-09-14T04:16:59.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:16:59.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.984009+0000", 2024-09-14T04:16:59.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011793483 2024-09-14T04:16:59.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.984065+0000", 2024-09-14T04:16:59.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6131999999999997e-05 2024-09-14T04:16:59.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:16:59.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.298573+0000", 2024-09-14T04:16:59.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31450850299999999 2024-09-14T04:16:59.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:16:59.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.298734+0000", 2024-09-14T04:16:59.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016041199999999999 2024-09-14T04:16:59.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.298852+0000", 2024-09-14T04:16:59.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011868 2024-09-14T04:16:59.047 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.047 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.047 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.047 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e276,252 11:7d6b516c:::benchmark_data_smithi114_223345_object50830:head,263'10291)", 2024-09-14T04:16:59.048 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:42.460879+0000", 2024-09-14T04:16:59.048 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 16.507629301000001, 2024-09-14T04:16:59.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039609729999999996, 2024-09-14T04:16:59.048 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.048 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.048 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.048 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.049 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.049 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.460879+0000", 2024-09-14T04:16:59.049 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.049 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.049 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.460879+0000", 2024-09-14T04:16:59.049 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.460886+0000", 2024-09-14T04:16:59.051 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1330000000000001e-06 2024-09-14T04:16:59.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.460897+0000", 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0784e-05 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.460912+0000", 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4394e-05 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.460923+0000", 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1718e-05 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.463440+0000", 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0025163859999999998 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.464840+0000", 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0014005580000000001 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e277,252 11:79cc6d9d:::benchmark_data_smithi114_223345_object50840:head,263'10293)", 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:42.470665+0000", 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 16.497843719999999, 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001858157, 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.470665+0000", 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.470665+0000", 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.470666+0000", 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.271e-06 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.470673+0000", 2024-09-14T04:16:59.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6900000000000003e-06 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.470685+0000", 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2597e-05 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.470692+0000", 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7730000000000002e-06 2024-09-14T04:16:59.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.471709+0000", 2024-09-14T04:16:59.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00101644 2024-09-14T04:16:59.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.472523+0000", 2024-09-14T04:16:59.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00081438599999999995 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e277,252 11:70585322:::benchmark_data_smithi114_223345_object50859:head,263'10295)", 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:42.476015+0000", 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 16.492493455000002, 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001861066, 2024-09-14T04:16:59.059 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.476015+0000", 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.476015+0000", 2024-09-14T04:16:59.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.476016+0000", 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3200000000000001e-06 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.476026+0000", 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5449999999999992e-06 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.476040+0000", 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3618000000000001e-05 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.476048+0000", 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0260000000000007e-06 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.476961+0000", 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00091290700000000002 2024-09-14T04:16:59.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.477876+0000", 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00091564999999999995 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e277,252 11:7f745a44:::benchmark_data_smithi114_223345_object52279:head,264'10391)", 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:42.679422+0000", 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 16.289086401999999, 2024-09-14T04:16:59.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024660540000000001, 2024-09-14T04:16:59.064 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.064 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.064 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.064 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.679422+0000", 2024-09-14T04:16:59.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.679422+0000", 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.679423+0000", 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9599999999999998e-07 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.065 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.679428+0000", 2024-09-14T04:16:59.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0819999999999998e-06 2024-09-14T04:16:59.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.066 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.066 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.679439+0000", 2024-09-14T04:16:59.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.131e-05 2024-09-14T04:16:59.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.066 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.066 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.679445+0000", 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0310000000000004e-06 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.681161+0000", 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017158130000000001 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:42.681888+0000", 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072702200000000002 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.067 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e277,252 11:7db80ea7:::benchmark_data_smithi114_223345_object54992:head,264'10574)", 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:43.038753+0000", 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 15.929755875, 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021953649999999999, 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.038753+0000", 2024-09-14T04:16:59.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.038753+0000", 2024-09-14T04:16:59.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.038754+0000", 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9199999999999999e-07 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.038761+0000", 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9779999999999999e-06 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.038776+0000", 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4928e-05 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.038784+0000", 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1799999999999996e-06 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.039886+0000", 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011026779999999999 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.040948+0000", 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010617090000000001 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e277,252 11:7f169028:::benchmark_data_smithi114_223345_object56165:head,264'10650)", 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:43.178794+0000", 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 15.78971484, 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001919891, 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.072 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.178794+0000", 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.178794+0000", 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.178794+0000", 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.68e-07 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.178799+0000", 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6500000000000004e-06 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.178823+0000", 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3813999999999999e-05 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.178831+0000", 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.568e-06 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.180135+0000", 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0013036829999999999 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.180714+0000", 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00057850800000000004 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e277,252 11:76426411:::benchmark_data_smithi114_223345_object56516:head,264'10676)", 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:43.226804+0000", 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 15.741704974999999, 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001861564, 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.076 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.226804+0000", 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.226804+0000", 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.226805+0000", 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.686e-06 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.226819+0000", 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3528e-05 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.226842+0000", 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2770000000000001e-05 2024-09-14T04:16:59.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.226853+0000", 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.155e-05 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.227747+0000", 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00089379599999999998 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.228665+0000", 2024-09-14T04:16:59.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00091823399999999998 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e277,252 11:7f193414:::benchmark_data_smithi114_223345_object59935:head,265'10872)", 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:43.572437+0000", 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 15.396071705000001, 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.042608816000000001, 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.080 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.572437+0000", 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.572437+0000", 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.572437+0000", 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9800000000000003e-07 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.572441+0000", 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6040000000000001e-06 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.614442+0000", 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.04200044 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.614453+0000", 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.115e-05 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.614527+0000", 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4158000000000004e-05 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.615046+0000", 2024-09-14T04:16:59.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00051886599999999999 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e277,252 11:757876aa:::benchmark_data_smithi114_223345_object60198:head,265'10885)", 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:43.636401+0000", 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 15.332107853, 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024377804999999999, 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.084 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.636401+0000", 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.636401+0000", 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.636401+0000", 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.46e-07 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.636404+0000", 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.446e-06 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.636410+0000", 2024-09-14T04:16:59.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3389999999999998e-06 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.636414+0000", 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1280000000000001e-06 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.636464+0000", 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0349000000000002e-05 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.660779+0000", 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024314097 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e277,252 11:705cfc12:::benchmark_data_smithi114_223345_object60224:head,265'10886)", 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:43.637850+0000", 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 15.330658301, 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.023787657, 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.088 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.089 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.089 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.637850+0000", 2024-09-14T04:16:59.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.637850+0000", 2024-09-14T04:16:59.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.637851+0000", 2024-09-14T04:16:59.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1699999999999998e-07 2024-09-14T04:16:59.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.637854+0000", 2024-09-14T04:16:59.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5700000000000001e-06 2024-09-14T04:16:59.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.660588+0000", 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.022733920000000001 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.660606+0000", 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8088e-05 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.660802+0000", 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000195055 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:43.661638+0000", 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00083640699999999995 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGRecoveryDeleteReply(11.e e277,252 11:7059f71e:::benchmark_data_smithi114_223345_object62331:head,265'11003)", 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:16:44.076288+0000", 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 14.892220247999999, 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.022528959000000001, 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:16:59.093 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:44.076288+0000", 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:44.076288+0000", 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:16:59.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:44.076289+0000", 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6300000000000005e-07 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:44.076293+0000", 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9380000000000002e-06 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:44.076302+0000", 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9989999999999997e-06 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:16:59.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:44.076307+0000", 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2820000000000004e-06 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:44.076347+0000", 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9876000000000002e-05 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:16:44.098817+0000", 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.022470200999999999 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.097 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:16:59.097 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:16:59.097 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:16:59.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:59.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:16:59.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:59.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:59.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:59.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:59.367 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:59.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:16:59.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:16:59.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:59.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:16:59.527 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.4 to 1 (10000) 2024-09-14T04:16:59.596 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:16:59.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:59.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:59.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:16:59.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:16:59.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:59.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:16:59.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:16:59.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:16:59 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3301856592' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:16:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:16:59 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3301856592' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:16:59.873 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:16:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:59.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:16:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:16:59 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3301856592' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:16:59.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:16:59.996 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:16:59.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:16:59.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:00.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:00.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:00.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:00.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:00.251 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:00.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:00.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:00.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:00.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:00.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:00.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:00.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:00.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:00.529 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:17:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:17:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:17:00.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:00.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:00.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:00.646 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-14T04:17:00.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:17:00.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:00.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:00.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:00.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:00.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:00 smithi114 ceph-mon[106969]: pgmap v1051: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:17:00.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3301856592' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:17:00.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:00 smithi114 ceph-mon[106969]: osdmap e282: 8 total, 8 up, 8 in 2024-09-14T04:17:00.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:00 smithi114 ceph-mon[103029]: pgmap v1051: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:17:00.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3301856592' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:17:00.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:00 smithi114 ceph-mon[103029]: osdmap e282: 8 total, 8 up, 8 in 2024-09-14T04:17:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:00.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:00.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:00 smithi160 ceph-mon[88407]: pgmap v1051: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:17:00.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3301856592' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:17:00.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:00 smithi160 ceph-mon[88407]: osdmap e282: 8 total, 8 up, 8 in 2024-09-14T04:17:00.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:01.005 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:01.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:01.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:01.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:01.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:01.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:01.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:01.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:01.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:01.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-14T04:17:01.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:01.382 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:01.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:01.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:01.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:01.509 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:01.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:01.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:01.637 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:01.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:01.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:01.763 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:01.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:01.770 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:17:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:17:01] "GET /metrics HTTP/1.1" 200 34157 "" "Prometheus/2.51.0" 2024-09-14T04:17:01.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:01.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:01.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:01.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:02.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:02.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:02.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:02.146 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:02.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:02.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:02.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:02.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:02.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:02.343 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:02.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:02.383 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:02.391 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:02.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:02.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:02.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:02.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:02.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:02.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:02.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:02.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:02.740 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:02 smithi114 ceph-mon[103029]: pgmap v1053: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:02.740 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:02 smithi114 ceph-mon[106969]: pgmap v1053: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:02.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:02.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:02.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:02.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:02.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:02.905 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:02.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:02.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:02 smithi160 ceph-mon[88407]: pgmap v1053: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:03.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:03.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:03.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:03.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:03.156 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:03.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:03.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:03.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:03.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:03.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:03.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:03.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:03.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:03.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:03.535 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:03.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:03.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:03.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:03.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:03.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:03.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:03.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:03.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:03.916 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:03.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:03.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:04.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:04.043 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:04.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:04.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:04.162 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:17:04.163 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:17:04.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:04.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:04.179 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:17:04.180 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:17:04.197 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:17:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:17:04.213 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:17:04.213 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:17:04.229 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:17:04.229 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:17:04.245 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:17:04.245 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:17:04.261 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:17:04.261 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:17:04.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:04.277 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:17:04.277 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:17:04.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:04.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:04.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:04.423 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:04.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:04.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:04.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:04.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:04.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:04.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:04.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:04.716 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:04 smithi114 ceph-mon[103029]: pgmap v1054: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:04.716 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:04 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1560396067' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:04.716 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:04 smithi114 ceph-mon[106969]: pgmap v1054: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:04.717 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:04 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1560396067' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:04.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:04.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:04.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:04.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:04 smithi160 ceph-mon[88407]: pgmap v1054: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:04.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:04 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1560396067' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:04.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:04.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:04.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:05.024 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:05.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:05.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:05.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:05.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:05.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:05.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:05.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:05.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:05.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:05.427 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:05.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:05.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:05.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:05.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:05.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:05.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:05.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:05.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:05.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:05.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:05.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:05.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:05.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:05.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:06.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:06.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:06.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:06.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:06.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:06.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:06.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:06.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:06.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:06.400 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:06.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:06.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:06.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:06.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:06.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:06.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:06.627 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:06 smithi114 ceph-mon[103029]: pgmap v1055: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:06.627 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:06 smithi114 ceph-mon[106969]: pgmap v1055: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:06.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:06.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:06.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:06.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:06.824 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:06.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:06.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:06 smithi160 ceph-mon[88407]: pgmap v1055: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:06.925 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:06.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:06.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:06.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:07.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:07.078 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:07.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:07.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:07.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:07.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:07.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:07.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:07.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:07.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:07.449 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:07.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:07.546 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:17:07.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:07.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:07.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:07.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:07 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/389342209' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:17:07.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:07 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:17:07.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:07 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/389342209' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:17:07.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:07 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:17:07.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:07.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:07.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:07.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:07.825 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:07.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:07 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/389342209' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:17:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:07 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:17:07.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:07.952 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:07.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:08.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:08.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:08.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:08.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:08.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:08.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:08.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:08.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:08.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:08.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:08.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:17:08.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:08.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:08.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:08.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:08.585 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:08.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:08.685 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[103029]: pgmap v1056: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:08.685 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:17:08.685 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:17:08.686 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[103029]: osdmap e283: 8 total, 8 up, 8 in 2024-09-14T04:17:08.686 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:08.686 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:08.686 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[106969]: pgmap v1056: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:08.686 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:17:08.686 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:17:08.686 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[106969]: osdmap e283: 8 total, 8 up, 8 in 2024-09-14T04:17:08.686 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:08.686 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:08.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:08.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:08.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:08.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:08.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:08.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:08.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:08 smithi160 ceph-mon[88407]: pgmap v1056: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:08 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:17:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:08 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:17:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:08 smithi160 ceph-mon[88407]: osdmap e283: 8 total, 8 up, 8 in 2024-09-14T04:17:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:08.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:08.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:08.961 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:08.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:08.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:09.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:09.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:09.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:09.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:09.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:09.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:09.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:09.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:09.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:09.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:09.346 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:17:09.347 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:17:09.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:09.464 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:09.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:09.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:09.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:09.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:09.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:09.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:09.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:09.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:09.811 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:09.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:09.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:09.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:09.964 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:09.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:10.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:10.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:10.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:10.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:10.215 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:10.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:10.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:10.342 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:10.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:10.427 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:17:10.428 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:17:10.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:10.464 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:10.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:10.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:10.542 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:17:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:17:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:17:10.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:10.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:10.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:10.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:10.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:10.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:10.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:10.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:10.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:10.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:10.838 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:10 smithi114 ceph-mon[106969]: pgmap v1058: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:10.838 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:10 smithi114 ceph-mon[106969]: from='client.36432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:17:10.838 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:10 smithi114 ceph-mon[103029]: pgmap v1058: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:10.838 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:10 smithi114 ceph-mon[103029]: from='client.36432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:17:10.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:10 smithi160 ceph-mon[88407]: pgmap v1058: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:10.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:10 smithi160 ceph-mon[88407]: from='client.36432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:17:10.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:10.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:10.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:11.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:11.086 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:11.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:11.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:11.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:11.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:11.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:11.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:11.340 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:11.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:11.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:11.468 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:11.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:11.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:11.591 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:11.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:17:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:17:11] "GET /metrics HTTP/1.1" 200 34153 "" "Prometheus/2.51.0" 2024-09-14T04:17:11.693 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:11.719 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:11.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:11.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:11.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:11.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:11.885 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1058,"stamp":"2024-09-14T04:17:08.959292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":595202,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":222,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6991828,"kb_used_data":800764,"kb_used_omap":33,"kb_used_meta":6190942,"kb_avail":742969388,"statfs":{"total":767960285184,"available":760800653312,"internally_reserved":0,"allocated":819982336,"data_stored":809687722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34628,"internal_metadata":6339524796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.429962"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":376,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498767+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:17:08.498767+0000","last_peered":"2024-09-14T04:17:08.498767+0000","last_clean":"2024-09-14T04:17:08.498767+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:17:08.498767+0000","last_undegraded":"2024-09-14T04:17:08.498767+0000","last_fullsized":"2024-09-14T04:17:08.498767+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":281,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:53.216925+0000","last_change":"2024-09-14T04:13:37.176637+0000","last_active":"2024-09-14T04:16:53.216925+0000","last_peered":"2024-09-14T04:16:53.216925+0000","last_clean":"2024-09-14T04:16:53.216925+0000","last_became_active":"2024-09-14T04:13:37.159759+0000","last_became_peered":"2024-09-14T04:13:37.159759+0000","last_unstale":"2024-09-14T04:16:53.216925+0000","last_undegraded":"2024-09-14T04:16:53.216925+0000","last_fullsized":"2024-09-14T04:16:53.216925+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:56.293937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":247,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.525580+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:16:52.525580+0000","last_peered":"2024-09-14T04:16:52.525580+0000","last_clean":"2024-09-14T04:16:52.525580+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:16:52.525580+0000","last_undegraded":"2024-09-14T04:16:52.525580+0000","last_fullsized":"2024-09-14T04:16:52.525580+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":459,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697222+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:16:59.697222+0000","last_peered":"2024-09-14T04:16:59.697222+0000","last_clean":"2024-09-14T04:16:59.697222+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:16:59.697222+0000","last_undegraded":"2024-09-14T04:16:59.697222+0000","last_fullsized":"2024-09-14T04:16:59.697222+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":445,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697539+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:16:59.697539+0000","last_peered":"2024-09-14T04:16:59.697539+0000","last_clean":"2024-09-14T04:16:59.697539+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:16:59.697539+0000","last_undegraded":"2024-09-14T04:16:59.697539+0000","last_fullsized":"2024-09-14T04:16:59.697539+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":244,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516056+0000","last_change":"2024-09-14T04:14:30.541064+0000","last_active":"2024-09-14T04:16:52.516056+0000","last_peered":"2024-09-14T04:16:52.516056+0000","last_clean":"2024-09-14T04:16:52.516056+0000","last_became_active":"2024-09-14T04:14:30.540755+0000","last_became_peered":"2024-09-14T04:14:30.540755+0000","last_unstale":"2024-09-14T04:16:52.516056+0000","last_undegraded":"2024-09-14T04:16:52.516056+0000","last_fullsized":"2024-09-14T04:16:52.516056+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:37.189820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":189,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498795+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:17:08.498795+0000","last_peered":"2024-09-14T04:17:08.498795+0000","last_clean":"2024-09-14T04:17:08.498795+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:17:08.498795+0000","last_undegraded":"2024-09-14T04:17:08.498795+0000","last_fullsized":"2024-09-14T04:17:08.498795+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":256,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.519928+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:16:52.519928+0000","last_peered":"2024-09-14T04:16:52.519928+0000","last_clean":"2024-09-14T04:16:52.519928+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:16:52.519928+0000","last_undegraded":"2024-09-14T04:16:52.519928+0000","last_fullsized":"2024-09-14T04:16:52.519928+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":266,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.518641+0000","last_change":"2024-09-14T04:14:30.543098+0000","last_active":"2024-09-14T04:16:52.518641+0000","last_peered":"2024-09-14T04:16:52.518641+0000","last_clean":"2024-09-14T04:16:52.518641+0000","last_became_active":"2024-09-14T04:14:30.542773+0000","last_became_peered":"2024-09-14T04:14:30.542773+0000","last_unstale":"2024-09-14T04:16:52.518641+0000","last_undegraded":"2024-09-14T04:16:52.518641+0000","last_fullsized":"2024-09-14T04:16:52.518641+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:06:27.423486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":230,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.519722+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:16:52.519722+0000","last_peered":"2024-09-14T04:16:52.519722+0000","last_clean":"2024-09-14T04:16:52.519722+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:16:52.519722+0000","last_undegraded":"2024-09-14T04:16:52.519722+0000","last_fullsized":"2024-09-14T04:16:52.519722+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":267,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.515950+0000","last_change":"2024-09-14T04:14:30.540956+0000","last_active":"2024-09-14T04:16:52.515950+0000","last_peered":"2024-09-14T04:16:52.515950+0000","last_clean":"2024-09-14T04:16:52.515950+0000","last_became_active":"2024-09-14T04:14:30.540644+0000","last_became_peered":"2024-09-14T04:14:30.540644+0000","last_unstale":"2024-09-14T04:16:52.515950+0000","last_undegraded":"2024-09-14T04:16:52.515950+0000","last_fullsized":"2024-09-14T04:16:52.515950+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:26:45.976307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.515905+0000","last_change":"2024-09-14T04:13:37.165507+0000","last_active":"2024-09-14T04:16:52.515905+0000","last_peered":"2024-09-14T04:16:52.515905+0000","last_clean":"2024-09-14T04:16:52.515905+0000","last_became_active":"2024-09-14T04:13:37.165215+0000","last_became_peered":"2024-09-14T04:13:37.165215+0000","last_unstale":"2024-09-14T04:16:52.515905+0000","last_undegraded":"2024-09-14T04:16:52.515905+0000","last_fullsized":"2024-09-14T04:16:52.515905+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:51:26.102907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":43,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.522422+0000","last_change":"2024-09-14T04:14:30.561646+0000","last_active":"2024-09-14T04:16:52.522422+0000","last_peered":"2024-09-14T04:16:52.522422+0000","last_clean":"2024-09-14T04:16:52.522422+0000","last_became_active":"2024-09-14T04:14:30.539174+0000","last_became_peered":"2024-09-14T04:14:30.539174+0000","last_unstale":"2024-09-14T04:16:52.522422+0000","last_undegraded":"2024-09-14T04:16:52.522422+0000","last_fullsized":"2024-09-14T04:16:52.522422+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:02:49.397620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":257,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.518051+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:16:52.518051+0000","last_peered":"2024-09-14T04:16:52.518051+0000","last_clean":"2024-09-14T04:16:52.518051+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:16:52.518051+0000","last_undegraded":"2024-09-14T04:16:52.518051+0000","last_fullsized":"2024-09-14T04:16:52.518051+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":268,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.517435+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:16:52.517435+0000","last_peered":"2024-09-14T04:16:52.517435+0000","last_clean":"2024-09-14T04:16:52.517435+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:16:52.517435+0000","last_undegraded":"2024-09-14T04:16:52.517435+0000","last_fullsized":"2024-09-14T04:16:52.517435+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":480,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697241+0000","last_change":"2024-09-14T04:13:37.165995+0000","last_active":"2024-09-14T04:16:59.697241+0000","last_peered":"2024-09-14T04:16:59.697241+0000","last_clean":"2024-09-14T04:16:59.697241+0000","last_became_active":"2024-09-14T04:13:37.165699+0000","last_became_peered":"2024-09-14T04:13:37.165699+0000","last_unstale":"2024-09-14T04:16:59.697241+0000","last_undegraded":"2024-09-14T04:16:59.697241+0000","last_fullsized":"2024-09-14T04:16:59.697241+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:21.792342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":427,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498519+0000","last_change":"2024-09-14T04:13:37.173224+0000","last_active":"2024-09-14T04:17:08.498519+0000","last_peered":"2024-09-14T04:17:08.498519+0000","last_clean":"2024-09-14T04:17:08.498519+0000","last_became_active":"2024-09-14T04:13:37.158630+0000","last_became_peered":"2024-09-14T04:13:37.158630+0000","last_unstale":"2024-09-14T04:17:08.498519+0000","last_undegraded":"2024-09-14T04:17:08.498519+0000","last_fullsized":"2024-09-14T04:17:08.498519+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:41:25.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516722+0000","last_change":"2024-09-14T04:13:37.163344+0000","last_active":"2024-09-14T04:16:52.516722+0000","last_peered":"2024-09-14T04:16:52.516722+0000","last_clean":"2024-09-14T04:16:52.516722+0000","last_became_active":"2024-09-14T04:13:37.163069+0000","last_became_peered":"2024-09-14T04:13:37.163069+0000","last_unstale":"2024-09-14T04:16:52.516722+0000","last_undegraded":"2024-09-14T04:16:52.516722+0000","last_fullsized":"2024-09-14T04:16:52.516722+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:16:22.162078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":268,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.518153+0000","last_change":"2024-09-14T04:13:37.166773+0000","last_active":"2024-09-14T04:16:52.518153+0000","last_peered":"2024-09-14T04:16:52.518153+0000","last_clean":"2024-09-14T04:16:52.518153+0000","last_became_active":"2024-09-14T04:13:37.166465+0000","last_became_peered":"2024-09-14T04:13:37.166465+0000","last_unstale":"2024-09-14T04:16:52.518153+0000","last_undegraded":"2024-09-14T04:16:52.518153+0000","last_fullsized":"2024-09-14T04:16:52.518153+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:57:18.786263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.517195+0000","last_change":"2024-09-14T04:13:37.166633+0000","last_active":"2024-09-14T04:16:52.517195+0000","last_peered":"2024-09-14T04:16:52.517195+0000","last_clean":"2024-09-14T04:16:52.517195+0000","last_became_active":"2024-09-14T04:13:37.166287+0000","last_became_peered":"2024-09-14T04:13:37.166287+0000","last_unstale":"2024-09-14T04:16:52.517195+0000","last_undegraded":"2024-09-14T04:16:52.517195+0000","last_fullsized":"2024-09-14T04:16:52.517195+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:15:22.237967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.515985+0000","last_change":"2024-09-14T04:13:37.158807+0000","last_active":"2024-09-14T04:16:52.515985+0000","last_peered":"2024-09-14T04:16:52.515985+0000","last_clean":"2024-09-14T04:16:52.515985+0000","last_became_active":"2024-09-14T04:13:37.158475+0000","last_became_peered":"2024-09-14T04:13:37.158475+0000","last_unstale":"2024-09-14T04:16:52.515985+0000","last_undegraded":"2024-09-14T04:16:52.515985+0000","last_fullsized":"2024-09-14T04:16:52.515985+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:09.005093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516023+0000","last_change":"2024-09-14T04:13:37.167580+0000","last_active":"2024-09-14T04:16:52.516023+0000","last_peered":"2024-09-14T04:16:52.516023+0000","last_clean":"2024-09-14T04:16:52.516023+0000","last_became_active":"2024-09-14T04:13:37.167261+0000","last_became_peered":"2024-09-14T04:13:37.167261+0000","last_unstale":"2024-09-14T04:16:52.516023+0000","last_undegraded":"2024-09-14T04:16:52.516023+0000","last_fullsized":"2024-09-14T04:16:52.516023+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:27:52.383112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":190,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498564+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:17:08.498564+0000","last_peered":"2024-09-14T04:17:08.498564+0000","last_clean":"2024-09-14T04:17:08.498564+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:17:08.498564+0000","last_undegraded":"2024-09-14T04:17:08.498564+0000","last_fullsized":"2024-09-14T04:17:08.498564+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":427,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.177591+0000","last_change":"2024-09-14T04:13:37.168484+0000","last_active":"2024-09-14T04:17:08.177591+0000","last_peered":"2024-09-14T04:17:08.177591+0000","last_clean":"2024-09-14T04:17:08.177591+0000","last_became_active":"2024-09-14T04:13:37.168136+0000","last_became_peered":"2024-09-14T04:13:37.168136+0000","last_unstale":"2024-09-14T04:17:08.177591+0000","last_undegraded":"2024-09-14T04:17:08.177591+0000","last_fullsized":"2024-09-14T04:17:08.177591+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:13.681730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":194,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697137+0000","last_change":"2024-09-14T04:14:30.540124+0000","last_active":"2024-09-14T04:16:59.697137+0000","last_peered":"2024-09-14T04:16:59.697137+0000","last_clean":"2024-09-14T04:16:59.697137+0000","last_became_active":"2024-09-14T04:14:30.539704+0000","last_became_peered":"2024-09-14T04:14:30.539704+0000","last_unstale":"2024-09-14T04:16:59.697137+0000","last_undegraded":"2024-09-14T04:16:59.697137+0000","last_fullsized":"2024-09-14T04:16:59.697137+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:07:53.515714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":261,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.517240+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:16:52.517240+0000","last_peered":"2024-09-14T04:16:52.517240+0000","last_clean":"2024-09-14T04:16:52.517240+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:16:52.517240+0000","last_undegraded":"2024-09-14T04:16:52.517240+0000","last_fullsized":"2024-09-14T04:16:52.517240+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":190,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498228+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:17:08.498228+0000","last_peered":"2024-09-14T04:17:08.498228+0000","last_clean":"2024-09-14T04:17:08.498228+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:17:08.498228+0000","last_undegraded":"2024-09-14T04:17:08.498228+0000","last_fullsized":"2024-09-14T04:17:08.498228+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":250,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.519476+0000","last_change":"2024-09-14T04:13:37.166354+0000","last_active":"2024-09-14T04:16:52.519476+0000","last_peered":"2024-09-14T04:16:52.519476+0000","last_clean":"2024-09-14T04:16:52.519476+0000","last_became_active":"2024-09-14T04:13:37.166057+0000","last_became_peered":"2024-09-14T04:13:37.166057+0000","last_unstale":"2024-09-14T04:16:52.519476+0000","last_undegraded":"2024-09-14T04:16:52.519476+0000","last_fullsized":"2024-09-14T04:16:52.519476+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:40.817585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":140,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.177141+0000","last_change":"2024-09-14T04:14:30.539202+0000","last_active":"2024-09-14T04:17:08.177141+0000","last_peered":"2024-09-14T04:17:08.177141+0000","last_clean":"2024-09-14T04:17:08.177141+0000","last_became_active":"2024-09-14T04:14:30.538884+0000","last_became_peered":"2024-09-14T04:14:30.538884+0000","last_unstale":"2024-09-14T04:17:08.177141+0000","last_undegraded":"2024-09-14T04:17:08.177141+0000","last_fullsized":"2024-09-14T04:17:08.177141+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:39.511239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":474,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697085+0000","last_change":"2024-09-14T04:13:37.165079+0000","last_active":"2024-09-14T04:16:59.697085+0000","last_peered":"2024-09-14T04:16:59.697085+0000","last_clean":"2024-09-14T04:16:59.697085+0000","last_became_active":"2024-09-14T04:13:37.164753+0000","last_became_peered":"2024-09-14T04:13:37.164753+0000","last_unstale":"2024-09-14T04:16:59.697085+0000","last_undegraded":"2024-09-14T04:16:59.697085+0000","last_fullsized":"2024-09-14T04:16:59.697085+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:50:32.658239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":457,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697368+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:16:59.697368+0000","last_peered":"2024-09-14T04:16:59.697368+0000","last_clean":"2024-09-14T04:16:59.697368+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:16:59.697368+0000","last_undegraded":"2024-09-14T04:16:59.697368+0000","last_fullsized":"2024-09-14T04:16:59.697368+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":451,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.530625+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:16:59.530625+0000","last_peered":"2024-09-14T04:16:59.530625+0000","last_clean":"2024-09-14T04:16:59.530625+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:16:59.530625+0000","last_undegraded":"2024-09-14T04:16:59.530625+0000","last_fullsized":"2024-09-14T04:16:59.530625+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":414,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.177368+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:17:08.177368+0000","last_peered":"2024-09-14T04:17:08.177368+0000","last_clean":"2024-09-14T04:17:08.177368+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:17:08.177368+0000","last_undegraded":"2024-09-14T04:17:08.177368+0000","last_fullsized":"2024-09-14T04:17:08.177368+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":182,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.530859+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:16:59.530859+0000","last_peered":"2024-09-14T04:16:59.530859+0000","last_clean":"2024-09-14T04:16:59.530859+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:16:59.530859+0000","last_undegraded":"2024-09-14T04:16:59.530859+0000","last_fullsized":"2024-09-14T04:16:59.530859+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":262,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.518178+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:16:52.518178+0000","last_peered":"2024-09-14T04:16:52.518178+0000","last_clean":"2024-09-14T04:16:52.518178+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:16:52.518178+0000","last_undegraded":"2024-09-14T04:16:52.518178+0000","last_fullsized":"2024-09-14T04:16:52.518178+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516827+0000","last_change":"2024-09-14T04:13:37.162303+0000","last_active":"2024-09-14T04:16:52.516827+0000","last_peered":"2024-09-14T04:16:52.516827+0000","last_clean":"2024-09-14T04:16:52.516827+0000","last_became_active":"2024-09-14T04:13:37.162040+0000","last_became_peered":"2024-09-14T04:13:37.162040+0000","last_unstale":"2024-09-14T04:16:52.516827+0000","last_undegraded":"2024-09-14T04:16:52.516827+0000","last_fullsized":"2024-09-14T04:16:52.516827+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:47:16.984589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":439,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.177362+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:17:08.177362+0000","last_peered":"2024-09-14T04:17:08.177362+0000","last_clean":"2024-09-14T04:17:08.177362+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:17:08.177362+0000","last_undegraded":"2024-09-14T04:17:08.177362+0000","last_fullsized":"2024-09-14T04:17:08.177362+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":285,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.525743+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:16:52.525743+0000","last_peered":"2024-09-14T04:16:52.525743+0000","last_clean":"2024-09-14T04:16:52.525743+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:16:52.525743+0000","last_undegraded":"2024-09-14T04:16:52.525743+0000","last_fullsized":"2024-09-14T04:16:52.525743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.177421+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:17:08.177421+0000","last_peered":"2024-09-14T04:17:08.177421+0000","last_clean":"2024-09-14T04:17:08.177421+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:17:08.177421+0000","last_undegraded":"2024-09-14T04:17:08.177421+0000","last_fullsized":"2024-09-14T04:17:08.177421+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":417,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.177323+0000","last_change":"2024-09-14T04:13:37.163670+0000","last_active":"2024-09-14T04:17:08.177323+0000","last_peered":"2024-09-14T04:17:08.177323+0000","last_clean":"2024-09-14T04:17:08.177323+0000","last_became_active":"2024-09-14T04:13:37.163234+0000","last_became_peered":"2024-09-14T04:13:37.163234+0000","last_unstale":"2024-09-14T04:17:08.177323+0000","last_undegraded":"2024-09-14T04:17:08.177323+0000","last_fullsized":"2024-09-14T04:17:08.177323+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:53:34.767208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":414,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.177546+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:17:08.177546+0000","last_peered":"2024-09-14T04:17:08.177546+0000","last_clean":"2024-09-14T04:17:08.177546+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:17:08.177546+0000","last_undegraded":"2024-09-14T04:17:08.177546+0000","last_fullsized":"2024-09-14T04:17:08.177546+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":279,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:53.216501+0000","last_change":"2024-09-14T04:13:37.162059+0000","last_active":"2024-09-14T04:16:53.216501+0000","last_peered":"2024-09-14T04:16:53.216501+0000","last_clean":"2024-09-14T04:16:53.216501+0000","last_became_active":"2024-09-14T04:13:37.161728+0000","last_became_peered":"2024-09-14T04:13:37.161728+0000","last_unstale":"2024-09-14T04:16:53.216501+0000","last_undegraded":"2024-09-14T04:16:53.216501+0000","last_fullsized":"2024-09-14T04:16:53.216501+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:39:59.671661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":435,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697321+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:16:59.697321+0000","last_peered":"2024-09-14T04:16:59.697321+0000","last_clean":"2024-09-14T04:16:59.697321+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:16:59.697321+0000","last_undegraded":"2024-09-14T04:16:59.697321+0000","last_fullsized":"2024-09-14T04:16:59.697321+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":405,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498874+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:17:08.498874+0000","last_peered":"2024-09-14T04:17:08.498874+0000","last_clean":"2024-09-14T04:17:08.498874+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:17:08.498874+0000","last_undegraded":"2024-09-14T04:17:08.498874+0000","last_fullsized":"2024-09-14T04:17:08.498874+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":427,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.177578+0000","last_change":"2024-09-14T04:13:37.169517+0000","last_active":"2024-09-14T04:17:08.177578+0000","last_peered":"2024-09-14T04:17:08.177578+0000","last_clean":"2024-09-14T04:17:08.177578+0000","last_became_active":"2024-09-14T04:13:37.169027+0000","last_became_peered":"2024-09-14T04:13:37.169027+0000","last_unstale":"2024-09-14T04:17:08.177578+0000","last_undegraded":"2024-09-14T04:17:08.177578+0000","last_fullsized":"2024-09-14T04:17:08.177578+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:49:57.369201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":93,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.518997+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:16:52.518997+0000","last_peered":"2024-09-14T04:16:52.518997+0000","last_clean":"2024-09-14T04:16:52.518997+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:16:52.518997+0000","last_undegraded":"2024-09-14T04:16:52.518997+0000","last_fullsized":"2024-09-14T04:16:52.518997+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":471,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697623+0000","last_change":"2024-09-14T04:13:37.178106+0000","last_active":"2024-09-14T04:16:59.697623+0000","last_peered":"2024-09-14T04:16:59.697623+0000","last_clean":"2024-09-14T04:16:59.697623+0000","last_became_active":"2024-09-14T04:13:37.167009+0000","last_became_peered":"2024-09-14T04:13:37.167009+0000","last_unstale":"2024-09-14T04:16:59.697623+0000","last_undegraded":"2024-09-14T04:16:59.697623+0000","last_fullsized":"2024-09-14T04:16:59.697623+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:28:37.700664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":260,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516733+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:16:52.516733+0000","last_peered":"2024-09-14T04:16:52.516733+0000","last_clean":"2024-09-14T04:16:52.516733+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:16:52.516733+0000","last_undegraded":"2024-09-14T04:16:52.516733+0000","last_fullsized":"2024-09-14T04:16:52.516733+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516124+0000","last_change":"2024-09-14T04:13:37.162917+0000","last_active":"2024-09-14T04:16:52.516124+0000","last_peered":"2024-09-14T04:16:52.516124+0000","last_clean":"2024-09-14T04:16:52.516124+0000","last_became_active":"2024-09-14T04:13:37.162614+0000","last_became_peered":"2024-09-14T04:13:37.162614+0000","last_unstale":"2024-09-14T04:16:52.516124+0000","last_undegraded":"2024-09-14T04:16:52.516124+0000","last_fullsized":"2024-09-14T04:16:52.516124+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:19:55.890004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516802+0000","last_change":"2024-09-14T04:13:37.162516+0000","last_active":"2024-09-14T04:16:52.516802+0000","last_peered":"2024-09-14T04:16:52.516802+0000","last_clean":"2024-09-14T04:16:52.516802+0000","last_became_active":"2024-09-14T04:13:37.162248+0000","last_became_peered":"2024-09-14T04:13:37.162248+0000","last_unstale":"2024-09-14T04:16:52.516802+0000","last_undegraded":"2024-09-14T04:16:52.516802+0000","last_fullsized":"2024-09-14T04:16:52.516802+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:41.935908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":50,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516658+0000","last_change":"2024-09-14T04:13:37.175049+0000","last_active":"2024-09-14T04:16:52.516658+0000","last_peered":"2024-09-14T04:16:52.516658+0000","last_clean":"2024-09-14T04:16:52.516658+0000","last_became_active":"2024-09-14T04:13:37.162920+0000","last_became_peered":"2024-09-14T04:13:37.162920+0000","last_unstale":"2024-09-14T04:16:52.516658+0000","last_undegraded":"2024-09-14T04:16:52.516658+0000","last_fullsized":"2024-09-14T04:16:52.516658+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:00:16.816854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":412,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.177209+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:17:08.177209+0000","last_peered":"2024-09-14T04:17:08.177209+0000","last_clean":"2024-09-14T04:17:08.177209+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:17:08.177209+0000","last_undegraded":"2024-09-14T04:17:08.177209+0000","last_fullsized":"2024-09-14T04:17:08.177209+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":66,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498882+0000","last_change":"2024-09-14T04:14:30.539928+0000","last_active":"2024-09-14T04:17:08.498882+0000","last_peered":"2024-09-14T04:17:08.498882+0000","last_clean":"2024-09-14T04:17:08.498882+0000","last_became_active":"2024-09-14T04:14:30.539572+0000","last_became_peered":"2024-09-14T04:14:30.539572+0000","last_unstale":"2024-09-14T04:17:08.498882+0000","last_undegraded":"2024-09-14T04:17:08.498882+0000","last_fullsized":"2024-09-14T04:17:08.498882+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:55:42.675243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":248,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.517044+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:16:52.517044+0000","last_peered":"2024-09-14T04:16:52.517044+0000","last_clean":"2024-09-14T04:16:52.517044+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:16:52.517044+0000","last_undegraded":"2024-09-14T04:16:52.517044+0000","last_fullsized":"2024-09-14T04:16:52.517044+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":364,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498516+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:17:08.498516+0000","last_peered":"2024-09-14T04:17:08.498516+0000","last_clean":"2024-09-14T04:17:08.498516+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:17:08.498516+0000","last_undegraded":"2024-09-14T04:17:08.498516+0000","last_fullsized":"2024-09-14T04:17:08.498516+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":459,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697783+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:16:59.697783+0000","last_peered":"2024-09-14T04:16:59.697783+0000","last_clean":"2024-09-14T04:16:59.697783+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:16:59.697783+0000","last_undegraded":"2024-09-14T04:16:59.697783+0000","last_fullsized":"2024-09-14T04:16:59.697783+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":376,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498652+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:17:08.498652+0000","last_peered":"2024-09-14T04:17:08.498652+0000","last_clean":"2024-09-14T04:17:08.498652+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:17:08.498652+0000","last_undegraded":"2024-09-14T04:17:08.498652+0000","last_fullsized":"2024-09-14T04:17:08.498652+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":257,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516655+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:16:52.516655+0000","last_peered":"2024-09-14T04:16:52.516655+0000","last_clean":"2024-09-14T04:16:52.516655+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:16:52.516655+0000","last_undegraded":"2024-09-14T04:16:52.516655+0000","last_fullsized":"2024-09-14T04:16:52.516655+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":189,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498476+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:17:08.498476+0000","last_peered":"2024-09-14T04:17:08.498476+0000","last_clean":"2024-09-14T04:17:08.498476+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:17:08.498476+0000","last_undegraded":"2024-09-14T04:17:08.498476+0000","last_fullsized":"2024-09-14T04:17:08.498476+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":461,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697446+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:16:59.697446+0000","last_peered":"2024-09-14T04:16:59.697446+0000","last_clean":"2024-09-14T04:16:59.697446+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:16:59.697446+0000","last_undegraded":"2024-09-14T04:16:59.697446+0000","last_fullsized":"2024-09-14T04:16:59.697446+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":426,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.177188+0000","last_change":"2024-09-14T04:13:37.163535+0000","last_active":"2024-09-14T04:17:08.177188+0000","last_peered":"2024-09-14T04:17:08.177188+0000","last_clean":"2024-09-14T04:17:08.177188+0000","last_became_active":"2024-09-14T04:13:37.163164+0000","last_became_peered":"2024-09-14T04:13:37.163164+0000","last_unstale":"2024-09-14T04:17:08.177188+0000","last_undegraded":"2024-09-14T04:17:08.177188+0000","last_fullsized":"2024-09-14T04:17:08.177188+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:13:25.324074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":196,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498786+0000","last_change":"2024-09-14T04:13:37.162587+0000","last_active":"2024-09-14T04:17:08.498786+0000","last_peered":"2024-09-14T04:17:08.498786+0000","last_clean":"2024-09-14T04:17:08.498786+0000","last_became_active":"2024-09-14T04:13:37.162219+0000","last_became_peered":"2024-09-14T04:13:37.162219+0000","last_unstale":"2024-09-14T04:17:08.498786+0000","last_undegraded":"2024-09-14T04:17:08.498786+0000","last_fullsized":"2024-09-14T04:17:08.498786+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:52:47.043189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516407+0000","last_change":"2024-09-14T04:13:37.164992+0000","last_active":"2024-09-14T04:16:52.516407+0000","last_peered":"2024-09-14T04:16:52.516407+0000","last_clean":"2024-09-14T04:16:52.516407+0000","last_became_active":"2024-09-14T04:13:37.164654+0000","last_became_peered":"2024-09-14T04:13:37.164654+0000","last_unstale":"2024-09-14T04:16:52.516407+0000","last_undegraded":"2024-09-14T04:16:52.516407+0000","last_fullsized":"2024-09-14T04:16:52.516407+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:49.248470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":453,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.530786+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:16:59.530786+0000","last_peered":"2024-09-14T04:16:59.530786+0000","last_clean":"2024-09-14T04:16:59.530786+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:16:59.530786+0000","last_undegraded":"2024-09-14T04:16:59.530786+0000","last_fullsized":"2024-09-14T04:16:59.530786+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":450,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697402+0000","last_change":"2024-09-14T04:13:37.166866+0000","last_active":"2024-09-14T04:16:59.697402+0000","last_peered":"2024-09-14T04:16:59.697402+0000","last_clean":"2024-09-14T04:16:59.697402+0000","last_became_active":"2024-09-14T04:13:37.166453+0000","last_became_peered":"2024-09-14T04:13:37.166453+0000","last_unstale":"2024-09-14T04:16:59.697402+0000","last_undegraded":"2024-09-14T04:16:59.697402+0000","last_fullsized":"2024-09-14T04:16:59.697402+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:34:59.229028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516721+0000","last_change":"2024-09-14T04:13:37.166852+0000","last_active":"2024-09-14T04:16:52.516721+0000","last_peered":"2024-09-14T04:16:52.516721+0000","last_clean":"2024-09-14T04:16:52.516721+0000","last_became_active":"2024-09-14T04:13:37.166550+0000","last_became_peered":"2024-09-14T04:13:37.166550+0000","last_unstale":"2024-09-14T04:16:52.516721+0000","last_undegraded":"2024-09-14T04:16:52.516721+0000","last_fullsized":"2024-09-14T04:16:52.516721+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:40:42.399969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":278,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.517371+0000","last_change":"2024-09-14T04:13:37.174369+0000","last_active":"2024-09-14T04:16:52.517371+0000","last_peered":"2024-09-14T04:16:52.517371+0000","last_clean":"2024-09-14T04:16:52.517371+0000","last_became_active":"2024-09-14T04:13:37.165108+0000","last_became_peered":"2024-09-14T04:13:37.165108+0000","last_unstale":"2024-09-14T04:16:52.517371+0000","last_undegraded":"2024-09-14T04:16:52.517371+0000","last_fullsized":"2024-09-14T04:16:52.517371+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:43.400463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.516327+0000","last_change":"2024-09-14T04:13:37.159053+0000","last_active":"2024-09-14T04:16:52.516327+0000","last_peered":"2024-09-14T04:16:52.516327+0000","last_clean":"2024-09-14T04:16:52.516327+0000","last_became_active":"2024-09-14T04:13:37.158702+0000","last_became_peered":"2024-09-14T04:13:37.158702+0000","last_unstale":"2024-09-14T04:16:52.516327+0000","last_undegraded":"2024-09-14T04:16:52.516327+0000","last_fullsized":"2024-09-14T04:16:52.516327+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:36:10.858811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":261,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.525670+0000","last_change":"2024-09-14T04:13:37.168004+0000","last_active":"2024-09-14T04:16:52.525670+0000","last_peered":"2024-09-14T04:16:52.525670+0000","last_clean":"2024-09-14T04:16:52.525670+0000","last_became_active":"2024-09-14T04:13:37.159695+0000","last_became_peered":"2024-09-14T04:13:37.159695+0000","last_unstale":"2024-09-14T04:16:52.525670+0000","last_undegraded":"2024-09-14T04:16:52.525670+0000","last_fullsized":"2024-09-14T04:16:52.525670+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:18:40.779932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":189,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498423+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:17:08.498423+0000","last_peered":"2024-09-14T04:17:08.498423+0000","last_clean":"2024-09-14T04:17:08.498423+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:17:08.498423+0000","last_undegraded":"2024-09-14T04:17:08.498423+0000","last_fullsized":"2024-09-14T04:17:08.498423+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":190,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-09-14T04:17:08.498390+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:17:08.498390+0000","last_peered":"2024-09-14T04:17:08.498390+0000","last_clean":"2024-09-14T04:17:08.498390+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:17:08.498390+0000","last_undegraded":"2024-09-14T04:17:08.498390+0000","last_fullsized":"2024-09-14T04:17:08.498390+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":40,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-14T04:16:52.515802+0000","last_change":"2024-09-14T04:13:37.165840+0000","last_active":"2024-09-14T04:16:52.515802+0000","last_peered":"2024-09-14T04:16:52.515802+0000","last_clean":"2024-09-14T04:16:52.515802+0000","last_became_active":"2024-09-14T04:13:37.165508+0000","last_became_peered":"2024-09-14T04:13:37.165508+0000","last_unstale":"2024-09-14T04:16:52.515802+0000","last_undegraded":"2024-09-14T04:16:52.515802+0000","last_fullsized":"2024-09-14T04:16:52.515802+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:24:02.783651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":449,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-09-14T04:16:59.697242+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:16:59.697242+0000","last_peered":"2024-09-14T04:16:59.697242+0000","last_clean":"2024-09-14T04:16:59.697242+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:16:59.697242+0000","last_undegraded":"2024-09-14T04:16:59.697242+0000","last_fullsized":"2024-09-14T04:16:59.697242+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4834,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430640,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1208476,"kb_used_data":301268,"kb_used_omap":1,"kb_used_meta":907198,"kb_avail":92536676,"statfs":{"total":95995035648,"available":94757556224,"internally_reserved":0,"allocated":308498432,"data_stored":307194200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":928971096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626869,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901268,"kb_used_data":3912,"kb_used_omap":1,"kb_used_meta":897342,"kb_avail":92843884,"statfs":{"total":95995035648,"available":95072137216,"internally_reserved":0,"allocated":4005888,"data_stored":2713502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918878552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823097,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908612,"kb_used_data":3320,"kb_used_omap":1,"kb_used_meta":905278,"kb_avail":92836540,"statfs":{"total":95995035648,"available":95064616960,"internally_reserved":0,"allocated":3399680,"data_stored":2118645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927005016},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917180,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":345812,"kb_used_data":3336,"kb_used_omap":6,"kb_used_meta":342457,"kb_avail":93399340,"statfs":{"total":95995035648,"available":95640924160,"internally_reserved":0,"allocated":3416064,"data_stored":2123191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":350676991},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552479,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692804,"kb_used_data":3320,"kb_used_omap":4,"kb_used_meta":689467,"kb_avail":93052348,"statfs":{"total":95995035648,"available":95285604352,"internally_reserved":0,"allocated":3399680,"data_stored":2118726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":706014653},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435520,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672960,"kb_used_data":3316,"kb_used_omap":6,"kb_used_meta":669625,"kb_avail":93072192,"statfs":{"total":95995035648,"available":95305924608,"internally_reserved":0,"allocated":3395584,"data_stored":2117778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":685696461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827990,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1319552,"kb_used_data":478392,"kb_used_omap":5,"kb_used_meta":841146,"kb_avail":92425600,"statfs":{"total":95995035648,"available":94643814400,"internally_reserved":0,"allocated":489873408,"data_stored":488592383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":861333743},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383977,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942344,"kb_used_data":3900,"kb_used_omap":5,"kb_used_meta":938426,"kb_avail":92802808,"statfs":{"total":95995035648,"available":95030075392,"internally_reserved":0,"allocated":3993600,"data_stored":2709297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":960948284},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:17:11.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:17:11.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:11.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:11.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:12.091 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:12.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:12.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:12.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:12.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:12.240 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:12.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:12.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:12.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:12.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:12.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:12.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:12.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:12.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:12.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-14T04:17:12.544 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:12.550 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_7' created 2024-09-14T04:17:12.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:12.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:12.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:12.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:12.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:12.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:12.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:12.815 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:12.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:12.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:12 smithi114 ceph-mon[103029]: pgmap v1059: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:12.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:12 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2510677806' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-14T04:17:12.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:12 smithi114 ceph-mon[106969]: pgmap v1059: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:12.873 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:12 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2510677806' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-14T04:17:12.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:12.874 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:12.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:12.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:12 smithi160 ceph-mon[88407]: pgmap v1059: 73 pgs: 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:12.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:12 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2510677806' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-14T04:17:12.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:12.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:12.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:13.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:13.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:13.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:13.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:13.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:13.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:13.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:17:13.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:13.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:13.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:13.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:13.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:13.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:13.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:13.647 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:13.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:13.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:13.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:13.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:13.805 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:13.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:13 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2510677806' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-14T04:17:13.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:13 smithi114 ceph-mon[103029]: osdmap e284: 8 total, 8 up, 8 in 2024-09-14T04:17:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:13 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2510677806' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-14T04:17:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:13 smithi114 ceph-mon[106969]: osdmap e284: 8 total, 8 up, 8 in 2024-09-14T04:17:13.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:13.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:13.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:13 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2510677806' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-14T04:17:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:13 smithi160 ceph-mon[88407]: osdmap e284: 8 total, 8 up, 8 in 2024-09-14T04:17:13.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:14.020 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:14.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:14.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:14.144 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:14.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:14.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:14.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:14.268 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:14.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:14.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:14.396 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:14.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:14.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:14.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:14.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:14.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:14.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:14 smithi114 ceph-mon[103029]: pgmap v1061: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:14 smithi114 ceph-mon[103029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:17:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:14 smithi114 ceph-mon[103029]: osdmap e285: 8 total, 8 up, 8 in 2024-09-14T04:17:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:14 smithi114 ceph-mon[106969]: pgmap v1061: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:14 smithi114 ceph-mon[106969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:17:14.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:14 smithi114 ceph-mon[106969]: osdmap e285: 8 total, 8 up, 8 in 2024-09-14T04:17:14.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:14.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:14.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:14.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:14.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:14.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:14.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:14.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:14.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:14 smithi160 ceph-mon[88407]: pgmap v1061: 89 pgs: 16 unknown, 73 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:14 smithi160 ceph-mon[88407]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:17:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:14 smithi160 ceph-mon[88407]: osdmap e285: 8 total, 8 up, 8 in 2024-09-14T04:17:14.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:15.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:15.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:15.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:15.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:15.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:15.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:15.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:15.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:15.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-09-14T04:17:15.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:15.396 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:15.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:15.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:15.498 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:15.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:15.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:15.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:15.652 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:15.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:15.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:15.776 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:15.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:15.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:15 smithi114 ceph-mon[103029]: osdmap e286: 8 total, 8 up, 8 in 2024-09-14T04:17:15.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:15 smithi114 ceph-mon[106969]: osdmap e286: 8 total, 8 up, 8 in 2024-09-14T04:17:15.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:15.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:15.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:15.903 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:17:15.904 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:17:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:15 smithi160 ceph-mon[88407]: osdmap e286: 8 total, 8 up, 8 in 2024-09-14T04:17:15.920 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:17:15.921 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:17:15.941 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:17:15.941 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:17:15.955 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:17:15.955 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:17:15.971 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:17:15.971 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:17:15.986 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:17:15.986 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:17:16.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:16.003 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:17:16.003 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:17:16.019 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:17:16.019 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:17:16.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:16.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:16.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:16.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:16.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:16.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:16.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:16.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:16.273 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:16.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:16.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:16.401 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:16.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:16.503 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:16.525 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:16.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:16.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:16.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:16.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:16.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:16.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:16.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:16.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:16 smithi114 ceph-mon[103029]: pgmap v1064: 89 pgs: 2 creating+peering, 4 unknown, 83 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:16 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2161923546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:16 smithi114 ceph-mon[106969]: pgmap v1064: 89 pgs: 2 creating+peering, 4 unknown, 83 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:16 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2161923546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:16.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:16.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:16.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:16 smithi160 ceph-mon[88407]: pgmap v1064: 89 pgs: 2 creating+peering, 4 unknown, 83 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:16 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2161923546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:16.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:17.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:17.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:17.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:17.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:17.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:17.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:17.269 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:17.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:17.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:17.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:17.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:17.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:17.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:17.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:17.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:17.631 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:17.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:17.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:17.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:17.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:17.776 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:17.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:17.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:17.877 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:17.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:17.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:17.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:18.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:18.027 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:18.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:18.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:18.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:18.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:18.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:18.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:18.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:18.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:18.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:18.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:18.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:18.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:18.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:18.532 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:18.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:18.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:18.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:18.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:18.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:18.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:18.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:18.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:18 smithi114 ceph-mon[103029]: pgmap v1065: 89 pgs: 2 creating+peering, 4 unknown, 83 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:18 smithi114 ceph-mon[106969]: pgmap v1065: 89 pgs: 2 creating+peering, 4 unknown, 83 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:18.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:18.907 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:18.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:18 smithi160 ceph-mon[88407]: pgmap v1065: 89 pgs: 2 creating+peering, 4 unknown, 83 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:19.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:19.033 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:19.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:19.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:19.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:19.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:19.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:19.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:19.284 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:19.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:19.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:19.408 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:19.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:19.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:19.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:19.533 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:19.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:19.568 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:17:19.568 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:17:19.568 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:17:19.568 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:17:19.568 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:17:19.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:17:19.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:19.661 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:19.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:19.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:19.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:19.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:19.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:19.912 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:19.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:20.014 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:20.040 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:20.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:20.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:20.142 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:20.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:20.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:20.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:20.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:20.294 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:20.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:20.312 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:20.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:20.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:20.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:20.457 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:17:20.458 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:17:20.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:20.547 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:20.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:20.573 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:17:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:17:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:17:20.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:20.672 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:20.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:20.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:20.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:20.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:20.810 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:20 smithi114 ceph-mon[103029]: pgmap v1066: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:20.811 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:20 smithi114 ceph-mon[106969]: pgmap v1066: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:20.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-14T04:17:20.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:20.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:20 smithi160 ceph-mon[88407]: pgmap v1066: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:20.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:20.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:21.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:21.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:21.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:21.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:21.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:21.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:21.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:21.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:21.300 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:21.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:21.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:21.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:21.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:21.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:21.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:21.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:21.577 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:17:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:17:21] "GET /metrics HTTP/1.1" 200 34153 "" "Prometheus/2.51.0" 2024-09-14T04:17:21.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:21.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:21.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:21.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:21.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:21.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:21.834 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:21 smithi114 ceph-mon[103029]: from='client.46448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:17:21.835 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:21 smithi114 ceph-mon[106969]: from='client.46448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:17:21.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:21 smithi160 ceph-mon[88407]: from='client.46448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:17:21.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:21.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:21.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:21.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:22.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:22.063 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:22.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:22.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:22.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:22.190 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:22.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:22.292 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:22.313 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:22.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:22.378 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:22.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:22.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:22.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:22.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:22.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:22.568 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:22.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:22.583 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-09-14T04:17:22.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:22.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:22.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:22.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:22.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:22.814 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:22.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:22.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:22 smithi114 ceph-mon[106969]: pgmap v1067: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:22 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2575642428' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:17:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:22 smithi114 ceph-mon[103029]: pgmap v1067: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:22 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2575642428' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:17:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:22 smithi160 ceph-mon[88407]: pgmap v1067: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:22 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2575642428' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:17:22.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:22.928 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1066,"stamp":"2024-09-14T04:17:18.961137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":595202,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":255,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6938724,"kb_used_data":743476,"kb_used_omap":33,"kb_used_meta":6195102,"kb_avail":743022492,"statfs":{"total":767960285184,"available":760855031808,"internally_reserved":0,"allocated":761319424,"data_stored":750922744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34628,"internal_metadata":6343784636},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.408862"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":382,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702890+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:17:14.702890+0000","last_peered":"2024-09-14T04:17:14.702890+0000","last_clean":"2024-09-14T04:17:14.702890+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:17:14.702890+0000","last_undegraded":"2024-09-14T04:17:14.702890+0000","last_fullsized":"2024-09-14T04:17:14.702890+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":284,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554000+0000","last_change":"2024-09-14T04:13:37.176637+0000","last_active":"2024-09-14T04:17:12.554000+0000","last_peered":"2024-09-14T04:17:12.554000+0000","last_clean":"2024-09-14T04:17:12.554000+0000","last_became_active":"2024-09-14T04:13:37.159759+0000","last_became_peered":"2024-09-14T04:13:37.159759+0000","last_unstale":"2024-09-14T04:17:12.554000+0000","last_undegraded":"2024-09-14T04:17:12.554000+0000","last_fullsized":"2024-09-14T04:17:12.554000+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:56.293937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":250,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.555000+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:17:12.555000+0000","last_peered":"2024-09-14T04:17:12.555000+0000","last_clean":"2024-09-14T04:17:12.555000+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:17:12.555000+0000","last_undegraded":"2024-09-14T04:17:12.555000+0000","last_fullsized":"2024-09-14T04:17:12.555000+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":465,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.562940+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:17:13.562940+0000","last_peered":"2024-09-14T04:17:13.562940+0000","last_clean":"2024-09-14T04:17:13.562940+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:17:13.562940+0000","last_undegraded":"2024-09-14T04:17:13.562940+0000","last_fullsized":"2024-09-14T04:17:13.562940+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":451,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.563887+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:17:13.563887+0000","last_peered":"2024-09-14T04:17:13.563887+0000","last_clean":"2024-09-14T04:17:13.563887+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:17:13.563887+0000","last_undegraded":"2024-09-14T04:17:13.563887+0000","last_fullsized":"2024-09-14T04:17:13.563887+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":248,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558266+0000","last_change":"2024-09-14T04:14:30.541064+0000","last_active":"2024-09-14T04:17:13.558266+0000","last_peered":"2024-09-14T04:17:13.558266+0000","last_clean":"2024-09-14T04:17:13.558266+0000","last_became_active":"2024-09-14T04:14:30.540755+0000","last_became_peered":"2024-09-14T04:14:30.540755+0000","last_unstale":"2024-09-14T04:17:13.558266+0000","last_undegraded":"2024-09-14T04:17:13.558266+0000","last_fullsized":"2024-09-14T04:17:13.558266+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:37.189820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":195,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702722+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:17:14.702722+0000","last_peered":"2024-09-14T04:17:14.702722+0000","last_clean":"2024-09-14T04:17:14.702722+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:17:14.702722+0000","last_undegraded":"2024-09-14T04:17:14.702722+0000","last_fullsized":"2024-09-14T04:17:14.702722+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":259,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554211+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:17:12.554211+0000","last_peered":"2024-09-14T04:17:12.554211+0000","last_clean":"2024-09-14T04:17:12.554211+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:17:12.554211+0000","last_undegraded":"2024-09-14T04:17:12.554211+0000","last_fullsized":"2024-09-14T04:17:12.554211+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.570488+0000","last_change":"2024-09-14T04:17:13.570249+0000","last_active":"2024-09-14T04:17:13.570488+0000","last_peered":"2024-09-14T04:17:13.570488+0000","last_clean":"2024-09-14T04:17:13.570488+0000","last_became_active":"2024-09-14T04:17:13.569916+0000","last_became_peered":"2024-09-14T04:17:13.569916+0000","last_unstale":"2024-09-14T04:17:13.570488+0000","last_undegraded":"2024-09-14T04:17:13.570488+0000","last_fullsized":"2024-09-14T04:17:13.570488+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:48:48.871738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":270,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558734+0000","last_change":"2024-09-14T04:14:30.543098+0000","last_active":"2024-09-14T04:17:13.558734+0000","last_peered":"2024-09-14T04:17:13.558734+0000","last_clean":"2024-09-14T04:17:13.558734+0000","last_became_active":"2024-09-14T04:14:30.542773+0000","last_became_peered":"2024-09-14T04:14:30.542773+0000","last_unstale":"2024-09-14T04:17:13.558734+0000","last_undegraded":"2024-09-14T04:17:13.558734+0000","last_fullsized":"2024-09-14T04:17:13.558734+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:06:27.423486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":233,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554767+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:17:12.554767+0000","last_peered":"2024-09-14T04:17:12.554767+0000","last_clean":"2024-09-14T04:17:12.554767+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:17:12.554767+0000","last_undegraded":"2024-09-14T04:17:12.554767+0000","last_fullsized":"2024-09-14T04:17:12.554767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.575564+0000","last_change":"2024-09-14T04:17:13.575564+0000","last_active":"2024-09-14T04:17:13.575564+0000","last_peered":"2024-09-14T04:17:13.575564+0000","last_clean":"2024-09-14T04:17:13.575564+0000","last_became_active":"2024-09-14T04:17:13.575303+0000","last_became_peered":"2024-09-14T04:17:13.575303+0000","last_unstale":"2024-09-14T04:17:13.575564+0000","last_undegraded":"2024-09-14T04:17:13.575564+0000","last_fullsized":"2024-09-14T04:17:13.575564+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:02:07.463876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":271,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558537+0000","last_change":"2024-09-14T04:14:30.540956+0000","last_active":"2024-09-14T04:17:13.558537+0000","last_peered":"2024-09-14T04:17:13.558537+0000","last_clean":"2024-09-14T04:17:13.558537+0000","last_became_active":"2024-09-14T04:14:30.540644+0000","last_became_peered":"2024-09-14T04:14:30.540644+0000","last_unstale":"2024-09-14T04:17:13.558537+0000","last_undegraded":"2024-09-14T04:17:13.558537+0000","last_fullsized":"2024-09-14T04:17:13.558537+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:26:45.976307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558188+0000","last_change":"2024-09-14T04:13:37.165507+0000","last_active":"2024-09-14T04:17:13.558188+0000","last_peered":"2024-09-14T04:17:13.558188+0000","last_clean":"2024-09-14T04:17:13.558188+0000","last_became_active":"2024-09-14T04:13:37.165215+0000","last_became_peered":"2024-09-14T04:13:37.165215+0000","last_unstale":"2024-09-14T04:17:13.558188+0000","last_undegraded":"2024-09-14T04:17:13.558188+0000","last_fullsized":"2024-09-14T04:17:13.558188+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:51:26.102907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.574801+0000","last_change":"2024-09-14T04:17:13.574569+0000","last_active":"2024-09-14T04:17:13.574801+0000","last_peered":"2024-09-14T04:17:13.574801+0000","last_clean":"2024-09-14T04:17:13.574801+0000","last_became_active":"2024-09-14T04:17:13.574286+0000","last_became_peered":"2024-09-14T04:17:13.574286+0000","last_unstale":"2024-09-14T04:17:13.574801+0000","last_undegraded":"2024-09-14T04:17:13.574801+0000","last_fullsized":"2024-09-14T04:17:13.574801+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:19:52.772589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":46,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554916+0000","last_change":"2024-09-14T04:14:30.561646+0000","last_active":"2024-09-14T04:17:12.554916+0000","last_peered":"2024-09-14T04:17:12.554916+0000","last_clean":"2024-09-14T04:17:12.554916+0000","last_became_active":"2024-09-14T04:14:30.539174+0000","last_became_peered":"2024-09-14T04:14:30.539174+0000","last_unstale":"2024-09-14T04:17:12.554916+0000","last_undegraded":"2024-09-14T04:17:12.554916+0000","last_fullsized":"2024-09-14T04:17:12.554916+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:02:49.397620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":260,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554220+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:17:12.554220+0000","last_peered":"2024-09-14T04:17:12.554220+0000","last_clean":"2024-09-14T04:17:12.554220+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:17:12.554220+0000","last_undegraded":"2024-09-14T04:17:12.554220+0000","last_fullsized":"2024-09-14T04:17:12.554220+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.575876+0000","last_change":"2024-09-14T04:17:13.574686+0000","last_active":"2024-09-14T04:17:13.575876+0000","last_peered":"2024-09-14T04:17:13.575876+0000","last_clean":"2024-09-14T04:17:13.575876+0000","last_became_active":"2024-09-14T04:17:13.574396+0000","last_became_peered":"2024-09-14T04:17:13.574396+0000","last_unstale":"2024-09-14T04:17:13.575876+0000","last_undegraded":"2024-09-14T04:17:13.575876+0000","last_fullsized":"2024-09-14T04:17:13.575876+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:07:38.996729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":271,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.555064+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:17:12.555064+0000","last_peered":"2024-09-14T04:17:12.555064+0000","last_clean":"2024-09-14T04:17:12.555064+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:17:12.555064+0000","last_undegraded":"2024-09-14T04:17:12.555064+0000","last_fullsized":"2024-09-14T04:17:12.555064+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":487,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.563296+0000","last_change":"2024-09-14T04:13:37.165995+0000","last_active":"2024-09-14T04:17:13.563296+0000","last_peered":"2024-09-14T04:17:13.563296+0000","last_clean":"2024-09-14T04:17:13.563296+0000","last_became_active":"2024-09-14T04:13:37.165699+0000","last_became_peered":"2024-09-14T04:13:37.165699+0000","last_unstale":"2024-09-14T04:17:13.563296+0000","last_undegraded":"2024-09-14T04:17:13.563296+0000","last_fullsized":"2024-09-14T04:17:13.563296+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:21.792342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.572270+0000","last_change":"2024-09-14T04:17:13.572270+0000","last_active":"2024-09-14T04:17:13.572270+0000","last_peered":"2024-09-14T04:17:13.572270+0000","last_clean":"2024-09-14T04:17:13.572270+0000","last_became_active":"2024-09-14T04:17:13.571986+0000","last_became_peered":"2024-09-14T04:17:13.571986+0000","last_unstale":"2024-09-14T04:17:13.572270+0000","last_undegraded":"2024-09-14T04:17:13.572270+0000","last_fullsized":"2024-09-14T04:17:13.572270+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:58:04.909769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":433,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.703298+0000","last_change":"2024-09-14T04:13:37.173224+0000","last_active":"2024-09-14T04:17:14.703298+0000","last_peered":"2024-09-14T04:17:14.703298+0000","last_clean":"2024-09-14T04:17:14.703298+0000","last_became_active":"2024-09-14T04:13:37.158630+0000","last_became_peered":"2024-09-14T04:13:37.158630+0000","last_unstale":"2024-09-14T04:17:14.703298+0000","last_undegraded":"2024-09-14T04:17:14.703298+0000","last_fullsized":"2024-09-14T04:17:14.703298+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:41:25.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558530+0000","last_change":"2024-09-14T04:13:37.163344+0000","last_active":"2024-09-14T04:17:13.558530+0000","last_peered":"2024-09-14T04:17:13.558530+0000","last_clean":"2024-09-14T04:17:13.558530+0000","last_became_active":"2024-09-14T04:13:37.163069+0000","last_became_peered":"2024-09-14T04:13:37.163069+0000","last_unstale":"2024-09-14T04:17:13.558530+0000","last_undegraded":"2024-09-14T04:17:13.558530+0000","last_fullsized":"2024-09-14T04:17:13.558530+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:16:22.162078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.572972+0000","last_change":"2024-09-14T04:17:13.572972+0000","last_active":"2024-09-14T04:17:13.572972+0000","last_peered":"2024-09-14T04:17:13.572972+0000","last_clean":"2024-09-14T04:17:13.572972+0000","last_became_active":"2024-09-14T04:17:13.572619+0000","last_became_peered":"2024-09-14T04:17:13.572619+0000","last_unstale":"2024-09-14T04:17:13.572972+0000","last_undegraded":"2024-09-14T04:17:13.572972+0000","last_fullsized":"2024-09-14T04:17:13.572972+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:07:11.298450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":271,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554315+0000","last_change":"2024-09-14T04:13:37.166773+0000","last_active":"2024-09-14T04:17:12.554315+0000","last_peered":"2024-09-14T04:17:12.554315+0000","last_clean":"2024-09-14T04:17:12.554315+0000","last_became_active":"2024-09-14T04:13:37.166465+0000","last_became_peered":"2024-09-14T04:13:37.166465+0000","last_unstale":"2024-09-14T04:17:12.554315+0000","last_undegraded":"2024-09-14T04:17:12.554315+0000","last_fullsized":"2024-09-14T04:17:12.554315+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:57:18.786263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558644+0000","last_change":"2024-09-14T04:13:37.166633+0000","last_active":"2024-09-14T04:17:13.558644+0000","last_peered":"2024-09-14T04:17:13.558644+0000","last_clean":"2024-09-14T04:17:13.558644+0000","last_became_active":"2024-09-14T04:13:37.166287+0000","last_became_peered":"2024-09-14T04:13:37.166287+0000","last_unstale":"2024-09-14T04:17:13.558644+0000","last_undegraded":"2024-09-14T04:17:13.558644+0000","last_fullsized":"2024-09-14T04:17:13.558644+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:15:22.237967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.703492+0000","last_change":"2024-09-14T04:17:13.575249+0000","last_active":"2024-09-14T04:17:14.703492+0000","last_peered":"2024-09-14T04:17:14.703492+0000","last_clean":"2024-09-14T04:17:14.703492+0000","last_became_active":"2024-09-14T04:17:13.574960+0000","last_became_peered":"2024-09-14T04:17:13.574960+0000","last_unstale":"2024-09-14T04:17:14.703492+0000","last_undegraded":"2024-09-14T04:17:14.703492+0000","last_fullsized":"2024-09-14T04:17:14.703492+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:57.321355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558246+0000","last_change":"2024-09-14T04:13:37.158807+0000","last_active":"2024-09-14T04:17:13.558246+0000","last_peered":"2024-09-14T04:17:13.558246+0000","last_clean":"2024-09-14T04:17:13.558246+0000","last_became_active":"2024-09-14T04:13:37.158475+0000","last_became_peered":"2024-09-14T04:13:37.158475+0000","last_unstale":"2024-09-14T04:17:13.558246+0000","last_undegraded":"2024-09-14T04:17:13.558246+0000","last_fullsized":"2024-09-14T04:17:13.558246+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:09.005093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558569+0000","last_change":"2024-09-14T04:13:37.167580+0000","last_active":"2024-09-14T04:17:13.558569+0000","last_peered":"2024-09-14T04:17:13.558569+0000","last_clean":"2024-09-14T04:17:13.558569+0000","last_became_active":"2024-09-14T04:13:37.167261+0000","last_became_peered":"2024-09-14T04:13:37.167261+0000","last_unstale":"2024-09-14T04:17:13.558569+0000","last_undegraded":"2024-09-14T04:17:13.558569+0000","last_fullsized":"2024-09-14T04:17:13.558569+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:27:52.383112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.703685+0000","last_change":"2024-09-14T04:17:13.572496+0000","last_active":"2024-09-14T04:17:14.703685+0000","last_peered":"2024-09-14T04:17:14.703685+0000","last_clean":"2024-09-14T04:17:14.703685+0000","last_became_active":"2024-09-14T04:17:13.572182+0000","last_became_peered":"2024-09-14T04:17:13.572182+0000","last_unstale":"2024-09-14T04:17:14.703685+0000","last_undegraded":"2024-09-14T04:17:14.703685+0000","last_fullsized":"2024-09-14T04:17:14.703685+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:00:00.551504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":196,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702516+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:17:14.702516+0000","last_peered":"2024-09-14T04:17:14.702516+0000","last_clean":"2024-09-14T04:17:14.702516+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:17:14.702516+0000","last_undegraded":"2024-09-14T04:17:14.702516+0000","last_fullsized":"2024-09-14T04:17:14.702516+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":433,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.703730+0000","last_change":"2024-09-14T04:13:37.168484+0000","last_active":"2024-09-14T04:17:14.703730+0000","last_peered":"2024-09-14T04:17:14.703730+0000","last_clean":"2024-09-14T04:17:14.703730+0000","last_became_active":"2024-09-14T04:13:37.168136+0000","last_became_peered":"2024-09-14T04:13:37.168136+0000","last_unstale":"2024-09-14T04:17:14.703730+0000","last_undegraded":"2024-09-14T04:17:14.703730+0000","last_fullsized":"2024-09-14T04:17:14.703730+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:13.681730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.572747+0000","last_change":"2024-09-14T04:17:13.572747+0000","last_active":"2024-09-14T04:17:13.572747+0000","last_peered":"2024-09-14T04:17:13.572747+0000","last_clean":"2024-09-14T04:17:13.572747+0000","last_became_active":"2024-09-14T04:17:13.572451+0000","last_became_peered":"2024-09-14T04:17:13.572451+0000","last_unstale":"2024-09-14T04:17:13.572747+0000","last_undegraded":"2024-09-14T04:17:13.572747+0000","last_fullsized":"2024-09-14T04:17:13.572747+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:00:57.163738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":200,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.563159+0000","last_change":"2024-09-14T04:14:30.540124+0000","last_active":"2024-09-14T04:17:13.563159+0000","last_peered":"2024-09-14T04:17:13.563159+0000","last_clean":"2024-09-14T04:17:13.563159+0000","last_became_active":"2024-09-14T04:14:30.539704+0000","last_became_peered":"2024-09-14T04:14:30.539704+0000","last_unstale":"2024-09-14T04:17:13.563159+0000","last_undegraded":"2024-09-14T04:17:13.563159+0000","last_fullsized":"2024-09-14T04:17:13.563159+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:07:53.515714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":264,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554705+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:17:12.554705+0000","last_peered":"2024-09-14T04:17:12.554705+0000","last_clean":"2024-09-14T04:17:12.554705+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:17:12.554705+0000","last_undegraded":"2024-09-14T04:17:12.554705+0000","last_fullsized":"2024-09-14T04:17:12.554705+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":196,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.703391+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:17:14.703391+0000","last_peered":"2024-09-14T04:17:14.703391+0000","last_clean":"2024-09-14T04:17:14.703391+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:17:14.703391+0000","last_undegraded":"2024-09-14T04:17:14.703391+0000","last_fullsized":"2024-09-14T04:17:14.703391+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":253,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554927+0000","last_change":"2024-09-14T04:13:37.166354+0000","last_active":"2024-09-14T04:17:12.554927+0000","last_peered":"2024-09-14T04:17:12.554927+0000","last_clean":"2024-09-14T04:17:12.554927+0000","last_became_active":"2024-09-14T04:13:37.166057+0000","last_became_peered":"2024-09-14T04:13:37.166057+0000","last_unstale":"2024-09-14T04:17:12.554927+0000","last_undegraded":"2024-09-14T04:17:12.554927+0000","last_fullsized":"2024-09-14T04:17:12.554927+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:40.817585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":11,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.574320+0000","last_change":"2024-09-14T04:17:13.570675+0000","last_active":"2024-09-14T04:17:13.574320+0000","last_peered":"2024-09-14T04:17:13.574320+0000","last_clean":"2024-09-14T04:17:13.574320+0000","last_became_active":"2024-09-14T04:17:13.569382+0000","last_became_peered":"2024-09-14T04:17:13.569382+0000","last_unstale":"2024-09-14T04:17:13.574320+0000","last_undegraded":"2024-09-14T04:17:13.574320+0000","last_fullsized":"2024-09-14T04:17:13.574320+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:33:57.054396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":146,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702070+0000","last_change":"2024-09-14T04:14:30.539202+0000","last_active":"2024-09-14T04:17:14.702070+0000","last_peered":"2024-09-14T04:17:14.702070+0000","last_clean":"2024-09-14T04:17:14.702070+0000","last_became_active":"2024-09-14T04:14:30.538884+0000","last_became_peered":"2024-09-14T04:14:30.538884+0000","last_unstale":"2024-09-14T04:17:14.702070+0000","last_undegraded":"2024-09-14T04:17:14.702070+0000","last_fullsized":"2024-09-14T04:17:14.702070+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:39.511239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":480,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.562782+0000","last_change":"2024-09-14T04:13:37.165079+0000","last_active":"2024-09-14T04:17:13.562782+0000","last_peered":"2024-09-14T04:17:13.562782+0000","last_clean":"2024-09-14T04:17:13.562782+0000","last_became_active":"2024-09-14T04:13:37.164753+0000","last_became_peered":"2024-09-14T04:13:37.164753+0000","last_unstale":"2024-09-14T04:17:13.562782+0000","last_undegraded":"2024-09-14T04:17:13.562782+0000","last_fullsized":"2024-09-14T04:17:13.562782+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:50:32.658239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":463,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.563794+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:17:13.563794+0000","last_peered":"2024-09-14T04:17:13.563794+0000","last_clean":"2024-09-14T04:17:13.563794+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:17:13.563794+0000","last_undegraded":"2024-09-14T04:17:13.563794+0000","last_fullsized":"2024-09-14T04:17:13.563794+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.572985+0000","last_change":"2024-09-14T04:17:13.572985+0000","last_active":"2024-09-14T04:17:13.572985+0000","last_peered":"2024-09-14T04:17:13.572985+0000","last_clean":"2024-09-14T04:17:13.572985+0000","last_became_active":"2024-09-14T04:17:13.572399+0000","last_became_peered":"2024-09-14T04:17:13.572399+0000","last_unstale":"2024-09-14T04:17:13.572985+0000","last_undegraded":"2024-09-14T04:17:13.572985+0000","last_fullsized":"2024-09-14T04:17:13.572985+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:58.922138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":459,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.564095+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:17:14.564095+0000","last_peered":"2024-09-14T04:17:14.564095+0000","last_clean":"2024-09-14T04:17:14.564095+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:17:14.564095+0000","last_undegraded":"2024-09-14T04:17:14.564095+0000","last_fullsized":"2024-09-14T04:17:14.564095+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":420,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702790+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:17:14.702790+0000","last_peered":"2024-09-14T04:17:14.702790+0000","last_clean":"2024-09-14T04:17:14.702790+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:17:14.702790+0000","last_undegraded":"2024-09-14T04:17:14.702790+0000","last_fullsized":"2024-09-14T04:17:14.702790+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":190,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.564376+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:17:14.564376+0000","last_peered":"2024-09-14T04:17:14.564376+0000","last_clean":"2024-09-14T04:17:14.564376+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:17:14.564376+0000","last_undegraded":"2024-09-14T04:17:14.564376+0000","last_fullsized":"2024-09-14T04:17:14.564376+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.574259+0000","last_change":"2024-09-14T04:17:13.570232+0000","last_active":"2024-09-14T04:17:13.574259+0000","last_peered":"2024-09-14T04:17:13.574259+0000","last_clean":"2024-09-14T04:17:13.574259+0000","last_became_active":"2024-09-14T04:17:13.569953+0000","last_became_peered":"2024-09-14T04:17:13.569953+0000","last_unstale":"2024-09-14T04:17:13.574259+0000","last_undegraded":"2024-09-14T04:17:13.574259+0000","last_fullsized":"2024-09-14T04:17:13.574259+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:59:19.041805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":265,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554493+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:17:12.554493+0000","last_peered":"2024-09-14T04:17:12.554493+0000","last_clean":"2024-09-14T04:17:12.554493+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:17:12.554493+0000","last_undegraded":"2024-09-14T04:17:12.554493+0000","last_fullsized":"2024-09-14T04:17:12.554493+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558388+0000","last_change":"2024-09-14T04:13:37.162303+0000","last_active":"2024-09-14T04:17:13.558388+0000","last_peered":"2024-09-14T04:17:13.558388+0000","last_clean":"2024-09-14T04:17:13.558388+0000","last_became_active":"2024-09-14T04:13:37.162040+0000","last_became_peered":"2024-09-14T04:13:37.162040+0000","last_unstale":"2024-09-14T04:17:13.558388+0000","last_undegraded":"2024-09-14T04:17:13.558388+0000","last_fullsized":"2024-09-14T04:17:13.558388+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:47:16.984589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":445,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702873+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:17:14.702873+0000","last_peered":"2024-09-14T04:17:14.702873+0000","last_clean":"2024-09-14T04:17:14.702873+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:17:14.702873+0000","last_undegraded":"2024-09-14T04:17:14.702873+0000","last_fullsized":"2024-09-14T04:17:14.702873+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.564137+0000","last_change":"2024-09-14T04:17:13.575911+0000","last_active":"2024-09-14T04:17:14.564137+0000","last_peered":"2024-09-14T04:17:14.564137+0000","last_clean":"2024-09-14T04:17:14.564137+0000","last_became_active":"2024-09-14T04:17:13.575627+0000","last_became_peered":"2024-09-14T04:17:13.575627+0000","last_unstale":"2024-09-14T04:17:14.564137+0000","last_undegraded":"2024-09-14T04:17:14.564137+0000","last_fullsized":"2024-09-14T04:17:14.564137+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:18:44.085967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":288,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.555172+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:17:12.555172+0000","last_peered":"2024-09-14T04:17:12.555172+0000","last_clean":"2024-09-14T04:17:12.555172+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:17:12.555172+0000","last_undegraded":"2024-09-14T04:17:12.555172+0000","last_fullsized":"2024-09-14T04:17:12.555172+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702958+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:17:14.702958+0000","last_peered":"2024-09-14T04:17:14.702958+0000","last_clean":"2024-09-14T04:17:14.702958+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:17:14.702958+0000","last_undegraded":"2024-09-14T04:17:14.702958+0000","last_fullsized":"2024-09-14T04:17:14.702958+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":423,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702543+0000","last_change":"2024-09-14T04:13:37.163670+0000","last_active":"2024-09-14T04:17:14.702543+0000","last_peered":"2024-09-14T04:17:14.702543+0000","last_clean":"2024-09-14T04:17:14.702543+0000","last_became_active":"2024-09-14T04:13:37.163234+0000","last_became_peered":"2024-09-14T04:13:37.163234+0000","last_unstale":"2024-09-14T04:17:14.702543+0000","last_undegraded":"2024-09-14T04:17:14.702543+0000","last_fullsized":"2024-09-14T04:17:14.702543+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:53:34.767208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.703101+0000","last_change":"2024-09-14T04:17:13.573945+0000","last_active":"2024-09-14T04:17:14.703101+0000","last_peered":"2024-09-14T04:17:14.703101+0000","last_clean":"2024-09-14T04:17:14.703101+0000","last_became_active":"2024-09-14T04:17:13.573636+0000","last_became_peered":"2024-09-14T04:17:13.573636+0000","last_unstale":"2024-09-14T04:17:14.703101+0000","last_undegraded":"2024-09-14T04:17:14.703101+0000","last_fullsized":"2024-09-14T04:17:14.703101+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:23:19.932984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":420,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.703092+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:17:14.703092+0000","last_peered":"2024-09-14T04:17:14.703092+0000","last_clean":"2024-09-14T04:17:14.703092+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:17:14.703092+0000","last_undegraded":"2024-09-14T04:17:14.703092+0000","last_fullsized":"2024-09-14T04:17:14.703092+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":282,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554656+0000","last_change":"2024-09-14T04:13:37.162059+0000","last_active":"2024-09-14T04:17:12.554656+0000","last_peered":"2024-09-14T04:17:12.554656+0000","last_clean":"2024-09-14T04:17:12.554656+0000","last_became_active":"2024-09-14T04:13:37.161728+0000","last_became_peered":"2024-09-14T04:13:37.161728+0000","last_unstale":"2024-09-14T04:17:12.554656+0000","last_undegraded":"2024-09-14T04:17:12.554656+0000","last_fullsized":"2024-09-14T04:17:12.554656+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:39:59.671661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":441,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.563342+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:17:13.563342+0000","last_peered":"2024-09-14T04:17:13.563342+0000","last_clean":"2024-09-14T04:17:13.563342+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:17:13.563342+0000","last_undegraded":"2024-09-14T04:17:13.563342+0000","last_fullsized":"2024-09-14T04:17:13.563342+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702268+0000","last_change":"2024-09-14T04:17:13.576627+0000","last_active":"2024-09-14T04:17:14.702268+0000","last_peered":"2024-09-14T04:17:14.702268+0000","last_clean":"2024-09-14T04:17:14.702268+0000","last_became_active":"2024-09-14T04:17:13.576099+0000","last_became_peered":"2024-09-14T04:17:13.576099+0000","last_unstale":"2024-09-14T04:17:14.702268+0000","last_undegraded":"2024-09-14T04:17:14.702268+0000","last_fullsized":"2024-09-14T04:17:14.702268+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:51:13.557881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":411,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702857+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:17:14.702857+0000","last_peered":"2024-09-14T04:17:14.702857+0000","last_clean":"2024-09-14T04:17:14.702857+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:17:14.702857+0000","last_undegraded":"2024-09-14T04:17:14.702857+0000","last_fullsized":"2024-09-14T04:17:14.702857+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":433,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.703615+0000","last_change":"2024-09-14T04:13:37.169517+0000","last_active":"2024-09-14T04:17:14.703615+0000","last_peered":"2024-09-14T04:17:14.703615+0000","last_clean":"2024-09-14T04:17:14.703615+0000","last_became_active":"2024-09-14T04:13:37.169027+0000","last_became_peered":"2024-09-14T04:13:37.169027+0000","last_unstale":"2024-09-14T04:17:14.703615+0000","last_undegraded":"2024-09-14T04:17:14.703615+0000","last_fullsized":"2024-09-14T04:17:14.703615+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:49:57.369201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":96,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554109+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:17:12.554109+0000","last_peered":"2024-09-14T04:17:12.554109+0000","last_clean":"2024-09-14T04:17:12.554109+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:17:12.554109+0000","last_undegraded":"2024-09-14T04:17:12.554109+0000","last_fullsized":"2024-09-14T04:17:12.554109+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.564150+0000","last_change":"2024-09-14T04:17:13.577599+0000","last_active":"2024-09-14T04:17:14.564150+0000","last_peered":"2024-09-14T04:17:14.564150+0000","last_clean":"2024-09-14T04:17:14.564150+0000","last_became_active":"2024-09-14T04:17:13.577330+0000","last_became_peered":"2024-09-14T04:17:13.577330+0000","last_unstale":"2024-09-14T04:17:14.564150+0000","last_undegraded":"2024-09-14T04:17:14.564150+0000","last_fullsized":"2024-09-14T04:17:14.564150+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:27:35.988705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":477,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.563907+0000","last_change":"2024-09-14T04:13:37.178106+0000","last_active":"2024-09-14T04:17:13.563907+0000","last_peered":"2024-09-14T04:17:13.563907+0000","last_clean":"2024-09-14T04:17:13.563907+0000","last_became_active":"2024-09-14T04:13:37.167009+0000","last_became_peered":"2024-09-14T04:13:37.167009+0000","last_unstale":"2024-09-14T04:17:13.563907+0000","last_undegraded":"2024-09-14T04:17:13.563907+0000","last_fullsized":"2024-09-14T04:17:13.563907+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:28:37.700664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":263,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554713+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:17:12.554713+0000","last_peered":"2024-09-14T04:17:12.554713+0000","last_clean":"2024-09-14T04:17:12.554713+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:17:12.554713+0000","last_undegraded":"2024-09-14T04:17:12.554713+0000","last_fullsized":"2024-09-14T04:17:12.554713+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558349+0000","last_change":"2024-09-14T04:13:37.162917+0000","last_active":"2024-09-14T04:17:13.558349+0000","last_peered":"2024-09-14T04:17:13.558349+0000","last_clean":"2024-09-14T04:17:13.558349+0000","last_became_active":"2024-09-14T04:13:37.162614+0000","last_became_peered":"2024-09-14T04:13:37.162614+0000","last_unstale":"2024-09-14T04:17:13.558349+0000","last_undegraded":"2024-09-14T04:17:13.558349+0000","last_fullsized":"2024-09-14T04:17:13.558349+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:19:55.890004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558442+0000","last_change":"2024-09-14T04:13:37.162516+0000","last_active":"2024-09-14T04:17:13.558442+0000","last_peered":"2024-09-14T04:17:13.558442+0000","last_clean":"2024-09-14T04:17:13.558442+0000","last_became_active":"2024-09-14T04:13:37.162248+0000","last_became_peered":"2024-09-14T04:13:37.162248+0000","last_unstale":"2024-09-14T04:17:13.558442+0000","last_undegraded":"2024-09-14T04:17:13.558442+0000","last_fullsized":"2024-09-14T04:17:13.558442+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:41.935908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":54,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558540+0000","last_change":"2024-09-14T04:13:37.175049+0000","last_active":"2024-09-14T04:17:13.558540+0000","last_peered":"2024-09-14T04:17:13.558540+0000","last_clean":"2024-09-14T04:17:13.558540+0000","last_became_active":"2024-09-14T04:13:37.162920+0000","last_became_peered":"2024-09-14T04:13:37.162920+0000","last_unstale":"2024-09-14T04:17:13.558540+0000","last_undegraded":"2024-09-14T04:17:13.558540+0000","last_fullsized":"2024-09-14T04:17:13.558540+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:00:16.816854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":418,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702406+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:17:14.702406+0000","last_peered":"2024-09-14T04:17:14.702406+0000","last_clean":"2024-09-14T04:17:14.702406+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:17:14.702406+0000","last_undegraded":"2024-09-14T04:17:14.702406+0000","last_fullsized":"2024-09-14T04:17:14.702406+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":72,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702560+0000","last_change":"2024-09-14T04:14:30.539928+0000","last_active":"2024-09-14T04:17:14.702560+0000","last_peered":"2024-09-14T04:17:14.702560+0000","last_clean":"2024-09-14T04:17:14.702560+0000","last_became_active":"2024-09-14T04:14:30.539572+0000","last_became_peered":"2024-09-14T04:14:30.539572+0000","last_unstale":"2024-09-14T04:17:14.702560+0000","last_undegraded":"2024-09-14T04:17:14.702560+0000","last_fullsized":"2024-09-14T04:17:14.702560+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:55:42.675243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":251,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554954+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:17:12.554954+0000","last_peered":"2024-09-14T04:17:12.554954+0000","last_clean":"2024-09-14T04:17:12.554954+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:17:12.554954+0000","last_undegraded":"2024-09-14T04:17:12.554954+0000","last_fullsized":"2024-09-14T04:17:12.554954+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":370,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702862+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:17:14.702862+0000","last_peered":"2024-09-14T04:17:14.702862+0000","last_clean":"2024-09-14T04:17:14.702862+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:17:14.702862+0000","last_undegraded":"2024-09-14T04:17:14.702862+0000","last_fullsized":"2024-09-14T04:17:14.702862+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":465,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.563309+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:17:13.563309+0000","last_peered":"2024-09-14T04:17:13.563309+0000","last_clean":"2024-09-14T04:17:13.563309+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:17:13.563309+0000","last_undegraded":"2024-09-14T04:17:13.563309+0000","last_fullsized":"2024-09-14T04:17:13.563309+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":382,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702607+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:17:14.702607+0000","last_peered":"2024-09-14T04:17:14.702607+0000","last_clean":"2024-09-14T04:17:14.702607+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:17:14.702607+0000","last_undegraded":"2024-09-14T04:17:14.702607+0000","last_fullsized":"2024-09-14T04:17:14.702607+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":260,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554681+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:17:12.554681+0000","last_peered":"2024-09-14T04:17:12.554681+0000","last_clean":"2024-09-14T04:17:12.554681+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:17:12.554681+0000","last_undegraded":"2024-09-14T04:17:12.554681+0000","last_fullsized":"2024-09-14T04:17:12.554681+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":195,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702421+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:17:14.702421+0000","last_peered":"2024-09-14T04:17:14.702421+0000","last_clean":"2024-09-14T04:17:14.702421+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:17:14.702421+0000","last_undegraded":"2024-09-14T04:17:14.702421+0000","last_fullsized":"2024-09-14T04:17:14.702421+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":467,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.563803+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:17:13.563803+0000","last_peered":"2024-09-14T04:17:13.563803+0000","last_clean":"2024-09-14T04:17:13.563803+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:17:13.563803+0000","last_undegraded":"2024-09-14T04:17:13.563803+0000","last_fullsized":"2024-09-14T04:17:13.563803+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":432,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702235+0000","last_change":"2024-09-14T04:13:37.163535+0000","last_active":"2024-09-14T04:17:14.702235+0000","last_peered":"2024-09-14T04:17:14.702235+0000","last_clean":"2024-09-14T04:17:14.702235+0000","last_became_active":"2024-09-14T04:13:37.163164+0000","last_became_peered":"2024-09-14T04:13:37.163164+0000","last_unstale":"2024-09-14T04:17:14.702235+0000","last_undegraded":"2024-09-14T04:17:14.702235+0000","last_fullsized":"2024-09-14T04:17:14.702235+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:13:25.324074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":202,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702414+0000","last_change":"2024-09-14T04:13:37.162587+0000","last_active":"2024-09-14T04:17:14.702414+0000","last_peered":"2024-09-14T04:17:14.702414+0000","last_clean":"2024-09-14T04:17:14.702414+0000","last_became_active":"2024-09-14T04:13:37.162219+0000","last_became_peered":"2024-09-14T04:13:37.162219+0000","last_unstale":"2024-09-14T04:17:14.702414+0000","last_undegraded":"2024-09-14T04:17:14.702414+0000","last_fullsized":"2024-09-14T04:17:14.702414+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:52:47.043189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558846+0000","last_change":"2024-09-14T04:13:37.164992+0000","last_active":"2024-09-14T04:17:13.558846+0000","last_peered":"2024-09-14T04:17:13.558846+0000","last_clean":"2024-09-14T04:17:13.558846+0000","last_became_active":"2024-09-14T04:13:37.164654+0000","last_became_peered":"2024-09-14T04:13:37.164654+0000","last_unstale":"2024-09-14T04:17:13.558846+0000","last_undegraded":"2024-09-14T04:17:13.558846+0000","last_fullsized":"2024-09-14T04:17:13.558846+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:49.248470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":461,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.564367+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:17:14.564367+0000","last_peered":"2024-09-14T04:17:14.564367+0000","last_clean":"2024-09-14T04:17:14.564367+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:17:14.564367+0000","last_undegraded":"2024-09-14T04:17:14.564367+0000","last_fullsized":"2024-09-14T04:17:14.564367+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":456,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.563439+0000","last_change":"2024-09-14T04:13:37.166866+0000","last_active":"2024-09-14T04:17:13.563439+0000","last_peered":"2024-09-14T04:17:13.563439+0000","last_clean":"2024-09-14T04:17:13.563439+0000","last_became_active":"2024-09-14T04:13:37.166453+0000","last_became_peered":"2024-09-14T04:13:37.166453+0000","last_unstale":"2024-09-14T04:17:13.563439+0000","last_undegraded":"2024-09-14T04:17:13.563439+0000","last_fullsized":"2024-09-14T04:17:13.563439+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:34:59.229028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558351+0000","last_change":"2024-09-14T04:13:37.166852+0000","last_active":"2024-09-14T04:17:13.558351+0000","last_peered":"2024-09-14T04:17:13.558351+0000","last_clean":"2024-09-14T04:17:13.558351+0000","last_became_active":"2024-09-14T04:13:37.166550+0000","last_became_peered":"2024-09-14T04:13:37.166550+0000","last_unstale":"2024-09-14T04:17:13.558351+0000","last_undegraded":"2024-09-14T04:17:13.558351+0000","last_fullsized":"2024-09-14T04:17:13.558351+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:40:42.399969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":281,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.554877+0000","last_change":"2024-09-14T04:13:37.174369+0000","last_active":"2024-09-14T04:17:12.554877+0000","last_peered":"2024-09-14T04:17:12.554877+0000","last_clean":"2024-09-14T04:17:12.554877+0000","last_became_active":"2024-09-14T04:13:37.165108+0000","last_became_peered":"2024-09-14T04:13:37.165108+0000","last_unstale":"2024-09-14T04:17:12.554877+0000","last_undegraded":"2024-09-14T04:17:12.554877+0000","last_fullsized":"2024-09-14T04:17:12.554877+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:43.400463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558745+0000","last_change":"2024-09-14T04:13:37.159053+0000","last_active":"2024-09-14T04:17:13.558745+0000","last_peered":"2024-09-14T04:17:13.558745+0000","last_clean":"2024-09-14T04:17:13.558745+0000","last_became_active":"2024-09-14T04:13:37.158702+0000","last_became_peered":"2024-09-14T04:13:37.158702+0000","last_unstale":"2024-09-14T04:17:13.558745+0000","last_undegraded":"2024-09-14T04:17:13.558745+0000","last_fullsized":"2024-09-14T04:17:13.558745+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:36:10.858811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":264,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-14T04:17:12.555095+0000","last_change":"2024-09-14T04:13:37.168004+0000","last_active":"2024-09-14T04:17:12.555095+0000","last_peered":"2024-09-14T04:17:12.555095+0000","last_clean":"2024-09-14T04:17:12.555095+0000","last_became_active":"2024-09-14T04:13:37.159695+0000","last_became_peered":"2024-09-14T04:13:37.159695+0000","last_unstale":"2024-09-14T04:17:12.555095+0000","last_undegraded":"2024-09-14T04:17:12.555095+0000","last_fullsized":"2024-09-14T04:17:12.555095+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:18:40.779932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":195,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.702582+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:17:14.702582+0000","last_peered":"2024-09-14T04:17:14.702582+0000","last_clean":"2024-09-14T04:17:14.702582+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:17:14.702582+0000","last_undegraded":"2024-09-14T04:17:14.702582+0000","last_fullsized":"2024-09-14T04:17:14.702582+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":196,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-09-14T04:17:14.703043+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:17:14.703043+0000","last_peered":"2024-09-14T04:17:14.703043+0000","last_clean":"2024-09-14T04:17:14.703043+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:17:14.703043+0000","last_undegraded":"2024-09-14T04:17:14.703043+0000","last_fullsized":"2024-09-14T04:17:14.703043+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":44,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.558731+0000","last_change":"2024-09-14T04:13:37.165840+0000","last_active":"2024-09-14T04:17:13.558731+0000","last_peered":"2024-09-14T04:17:13.558731+0000","last_clean":"2024-09-14T04:17:13.558731+0000","last_became_active":"2024-09-14T04:13:37.165508+0000","last_became_peered":"2024-09-14T04:13:37.165508+0000","last_unstale":"2024-09-14T04:17:13.558731+0000","last_undegraded":"2024-09-14T04:17:13.558731+0000","last_fullsized":"2024-09-14T04:17:13.558731+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:24:02.783651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":455,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-14T04:17:13.563054+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:17:13.563054+0000","last_peered":"2024-09-14T04:17:13.563054+0000","last_clean":"2024-09-14T04:17:13.563054+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:17:13.563054+0000","last_undegraded":"2024-09-14T04:17:13.563054+0000","last_fullsized":"2024-09-14T04:17:13.563054+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4834,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430642,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1189316,"kb_used_data":282104,"kb_used_omap":1,"kb_used_meta":907198,"kb_avail":92555836,"statfs":{"total":95995035648,"available":94777176064,"internally_reserved":0,"allocated":288874496,"data_stored":287558355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":928971096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626871,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901292,"kb_used_data":3936,"kb_used_omap":1,"kb_used_meta":897342,"kb_avail":92843860,"statfs":{"total":95995035648,"available":95072112640,"internally_reserved":0,"allocated":4030464,"data_stored":2730108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918878552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823099,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908652,"kb_used_data":3356,"kb_used_omap":1,"kb_used_meta":905278,"kb_avail":92836500,"statfs":{"total":95995035648,"available":95064576000,"internally_reserved":0,"allocated":3436544,"data_stored":2143600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927005016},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917182,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":350028,"kb_used_data":3384,"kb_used_omap":6,"kb_used_meta":346617,"kb_avail":93395124,"statfs":{"total":95995035648,"available":95636606976,"internally_reserved":0,"allocated":3465216,"data_stored":2156766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":354936831},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552481,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692856,"kb_used_data":3368,"kb_used_omap":4,"kb_used_meta":689467,"kb_avail":93052296,"statfs":{"total":95995035648,"available":95285551104,"internally_reserved":0,"allocated":3448832,"data_stored":2152301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":706014653},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435522,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":673012,"kb_used_data":3364,"kb_used_omap":6,"kb_used_meta":669625,"kb_avail":93072140,"statfs":{"total":95995035648,"available":95305871360,"internally_reserved":0,"allocated":3444736,"data_stored":2151353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":685696461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611827992,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1281200,"kb_used_data":440040,"kb_used_omap":5,"kb_used_meta":841146,"kb_avail":92463952,"statfs":{"total":95995035648,"available":94683086848,"internally_reserved":0,"allocated":450600960,"data_stored":449304358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":861333743},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383979,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942368,"kb_used_data":3924,"kb_used_omap":5,"kb_used_meta":938426,"kb_avail":92802784,"statfs":{"total":95995035648,"available":95030050816,"internally_reserved":0,"allocated":4018176,"data_stored":2725903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":960948284},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:17:22.932 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-14T04:17:22.932 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:17:22.932 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-14T04:17:22.932 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:17:22.932 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:17:22.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-14T04:17:22.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:22.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:23.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:23.060 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:23.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:23.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:23.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:23.184 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:23.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:23.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:23.212 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:17:23.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:23.307 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:23.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:23.409 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:23.431 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:23.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:23.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:23.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:23.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:23.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:23.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:23.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:23.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:23.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:23.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:23.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:23.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2575642428' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:17:23.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:23 smithi114 ceph-mon[103029]: osdmap e287: 8 total, 8 up, 8 in 2024-09-14T04:17:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:23.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:23 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2575642428' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:17:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:23 smithi114 ceph-mon[106969]: osdmap e287: 8 total, 8 up, 8 in 2024-09-14T04:17:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:23.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:23 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2575642428' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:17:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:23 smithi160 ceph-mon[88407]: osdmap e287: 8 total, 8 up, 8 in 2024-09-14T04:17:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:23.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:23.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:23.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:24.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:24.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:24.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:24.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:24.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:24.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:24.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:24.293 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:24.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:24.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:24.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:24.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:24.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:24.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:24.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:24.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:24.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:24.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:24.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:24.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:24.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:24.854 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:24 smithi160 ceph-mon[88407]: pgmap v1069: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:24.855 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:24 smithi160 ceph-mon[88407]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:17:24.863 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:24 smithi114 ceph-mon[106969]: pgmap v1069: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:24.864 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:24 smithi114 ceph-mon[106969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:17:24.864 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:24 smithi114 ceph-mon[103029]: pgmap v1069: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:24.864 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:24 smithi114 ceph-mon[103029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:17:24.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:24.913 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:24.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:25.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:25.036 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:25.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:25.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:25.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:25.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:25.197 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:17:25.197 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-14T04:17:25.197 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:17:25.200 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:25.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:25.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:25.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:25.335 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-09-14T04:17:25.337 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-14T04:17:25.337 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-14T04:17:25.382 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-14T04:17:25.382 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-14T04:17:25.382 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_253692 2024-09-14T04:17:25.382 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:17:25.382 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-14T04:17:25.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:25.431 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:25.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:25.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:25.565 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:25.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:25.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:25.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:25.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:25.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:25.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:25.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:25.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:25.947 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:25.984 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:25.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:26.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:26.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:26.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:26.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:26.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:26.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:26.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:26.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:26.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:26.383 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 3680 3664 228.899 229 0.00454703 0.00435112 2024-09-14T04:17:26.385 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:26.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:26.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:26.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:26.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:26.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:26.645 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:26.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:26.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:26.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:26.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:26.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:26.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:26.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:26 smithi114 ceph-mon[103029]: pgmap v1070: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:26 smithi114 ceph-mon[106969]: pgmap v1070: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:26.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:26 smithi160 ceph-mon[88407]: pgmap v1070: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:26.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:26.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:26.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:27.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:27.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:27.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:27.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:27.095 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:17:27.095 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:17:27.133 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:17:27.133 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:17:27.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:27.175 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:17:27.176 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:17:27.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:27.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:27.220 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:17:27.221 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:17:27.253 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:17:27.253 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:17:27.285 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:17:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:17:27.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:27.319 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:17:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:17:27.324 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:27.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:27.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:27.351 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:17:27.351 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:17:27.383 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 6821 6805 212.6 196.312 0.00494393 0.00469415 2024-09-14T04:17:27.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:27.463 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:27.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:27.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:27.604 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:17:27.605 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:27.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:27.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:27.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:27.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:27.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:27.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:27.861 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:27.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:27.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:27 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/864577871' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:17:27.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:27 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4264767084' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:27.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:27 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/864577871' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:17:27.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:27 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4264767084' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:27.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:27 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/864577871' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:17:27.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:27 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4264767084' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:27.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:28.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:28.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:28.103 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:28.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:28.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:28.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:28.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:28.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:28.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:28.383 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 9924 9908 206.374 193.938 0.00725 0.00483756 2024-09-14T04:17:28.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:28.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:28.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:28.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:28.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:28.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:28.602 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:17:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=cleanup t=2024-09-14T04:17:28.191938688Z level=info msg="Completed cleanup jobs" duration=11.712383ms 2024-09-14T04:17:28.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:28.667 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:28.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:28.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:28.807 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:28.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:28.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:28.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:28 smithi114 ceph-mon[106969]: pgmap v1071: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:28.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:28 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:17:28.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:28 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/864577871' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:17:28.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:28 smithi114 ceph-mon[106969]: osdmap e288: 8 total, 8 up, 8 in 2024-09-14T04:17:28.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:28 smithi114 ceph-mon[103029]: pgmap v1071: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:28.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:28 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:17:28.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:28 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/864577871' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:17:28.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:28 smithi114 ceph-mon[103029]: osdmap e288: 8 total, 8 up, 8 in 2024-09-14T04:17:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:28 smithi160 ceph-mon[88407]: pgmap v1071: 89 pgs: 89 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-09-14T04:17:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:28 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:17:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:28 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/864577871' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:17:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:28 smithi160 ceph-mon[88407]: osdmap e288: 8 total, 8 up, 8 in 2024-09-14T04:17:28.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:28.936 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:28.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:28.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:29.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:29.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:29.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:29.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:29.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:29.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:29.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:29.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:29.329 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:29.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:29.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:29.383 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 12955 12939 202.138 189.438 0.00552854 0.00494059 2024-09-14T04:17:29.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:29.467 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:29.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:29.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:29.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:29.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:29.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:29.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:29.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:29.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:29.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:29.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:29 smithi114 ceph-mon[106969]: pgmap v1073: 89 pgs: 89 active+clean; 256 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 32 MiB/s wr, 509 op/s 2024-09-14T04:17:29.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:29 smithi114 ceph-mon[103029]: pgmap v1073: 89 pgs: 89 active+clean; 256 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 32 MiB/s wr, 509 op/s 2024-09-14T04:17:29.881 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:29.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:29 smithi160 ceph-mon[88407]: pgmap v1073: 89 pgs: 89 active+clean; 256 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 32 MiB/s wr, 509 op/s 2024-09-14T04:17:29.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:30.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:30.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:30.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:30.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:30.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:30.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:30.260 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:30.288 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:30.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:30.383 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 15643 15627 195.308 168 0.00301071 0.00511431 2024-09-14T04:17:30.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:30.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:30.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:30.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:30.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:30.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:17:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:17:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:17:30.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:30.712 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:30.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:30.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:30.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:30.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:30.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:30.981 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:30.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:31.058 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:17:31.058 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:17:31.058 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-14T04:17:31.058 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.436282 2024-09-14T04:17:31.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 0.43628202977914354 2024-09-14T04:17:31.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:31.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:31.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:31.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:31.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:31.263 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:31.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:31.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:31.385 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 18004 17988 187.332 147.562 0.0107883 0.00532885 2024-09-14T04:17:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:31.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:31.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:31.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:31.539 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:31.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:31.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:31.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:31.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:31.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:31.714 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:17:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:17:31] "GET /metrics HTTP/1.1" 200 36503 "" "Prometheus/2.51.0" 2024-09-14T04:17:31.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:31.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:31.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:31.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:31.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:31.970 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:31.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:32.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:32.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:32.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:32 smithi160 ceph-mon[88407]: pgmap v1074: 89 pgs: 89 active+clean; 641 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-14T04:17:32.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:32.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:32.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:32.262 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:17:32.262 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:17:32.275 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:32.279 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:32 smithi114 ceph-mon[106969]: pgmap v1074: 89 pgs: 89 active+clean; 641 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-14T04:17:32.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:32 smithi114 ceph-mon[103029]: pgmap v1074: 89 pgs: 89 active+clean; 641 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-14T04:17:32.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:32.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:32.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:32.385 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 20157 20141 179.755 134.562 0.0042371 0.00555595 2024-09-14T04:17:32.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:17:32.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:32.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:32.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:32.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:32.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:32.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:32.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:32.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:32.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:32.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:32.915 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:32.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:33.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:33.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:33.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:33.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:33.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:33.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:33.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:33.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:33.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:33.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:33.385 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 22242 22226 173.575 130.312 0.00486033 0.00575553 2024-09-14T04:17:33.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:33.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:33.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:33.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:33.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:33.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:33.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:17:33.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:33.718 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:33.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:33.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:33.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:33.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:33.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:34.000 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:34.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:34.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:34.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:34.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:34.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:34.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-14T04:17:34.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:34.272 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:34.294 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:34.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:34.298 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:34 smithi114 ceph-mon[103029]: pgmap v1075: 89 pgs: 89 active+clean; 853 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T04:17:34.298 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:34 smithi114 ceph-mon[106969]: pgmap v1075: 89 pgs: 89 active+clean; 853 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T04:17:34.385 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 24012 23996 166.582 110.625 0.00623983 0.00599823 2024-09-14T04:17:34.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:34 smithi160 ceph-mon[88407]: pgmap v1075: 89 pgs: 89 active+clean; 853 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-14T04:17:34.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:34.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:34.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:34.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:34.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:34.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:34.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:34.719 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:34.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:34.821 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:34.869 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:34.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:34.970 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:34.997 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:34.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:35.098 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:35.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:35.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:35.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:35.260 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:35.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:35.298 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:35.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:35.385 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 26181 26165 163.479 135.562 0.00611374 0.00611206 2024-09-14T04:17:35.393 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:35.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:35.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:35.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:35.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:35.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:35.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:35.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:35.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:35.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:35.795 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:35.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:35.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:35.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:35.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:36.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:36.057 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:36.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:36.158 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:36.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:36.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:36.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:36.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:36.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:36 smithi114 ceph-mon[103029]: pgmap v1076: 89 pgs: 89 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T04:17:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:36 smithi114 ceph-mon[106969]: pgmap v1076: 89 pgs: 89 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T04:17:36.385 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 28216 28200 160.18 127.188 0.00644222 0.00623809 2024-09-14T04:17:36.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:36 smithi160 ceph-mon[88407]: pgmap v1076: 89 pgs: 89 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T04:17:36.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:36.440 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:36.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:36.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:36.574 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:36.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:36.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:36.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:36.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:36.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:36.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:36.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:36.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:36.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:36.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:36.953 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:36.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:37.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:37.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:37.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:37.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:37.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:37.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:37.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:17:37.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:17:37.385 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 29799 29783 155.076 98.9375 0.00726301 0.00644348 2024-09-14T04:17:37.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:37.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:17:37.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:37.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:37.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:37.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:37.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:37.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:37.713 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:37.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:37.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:37.854 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:37.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:37.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:37.982 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:37.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:38.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:38.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:38.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:38.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:38.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:38.258 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:38.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:38.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:38.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:38 smithi114 ceph-mon[106969]: pgmap v1077: 89 pgs: 89 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T04:17:38.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:38.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:38 smithi114 ceph-mon[103029]: pgmap v1077: 89 pgs: 89 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T04:17:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:38.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:38.385 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 31692 31676 152.247 118.312 0.00579011 0.006561 2024-09-14T04:17:38.398 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:38.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:38 smithi160 ceph-mon[88407]: pgmap v1077: 89 pgs: 89 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T04:17:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:38.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:38.522 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:38.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:38.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:38.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:38.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:38.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:38.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:38.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:38.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:38.926 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:38.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:39.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:39.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:39.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:39.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:39.196 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:39.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:39.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:39.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:39.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:39.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:39.386 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 15 33596 33581 149.875 119.062 0.00741894 0.00666522 2024-09-14T04:17:39.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:39.488 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:39.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:39.490 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:39.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:39.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:39.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:39.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:39.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:39.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:39.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:39.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:39.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:39.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:40.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:40.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:40.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:40.138 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:40.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:40.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:40.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:40.273 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:17:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:17:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:17:40.273 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:40 smithi160 ceph-mon[88407]: pgmap v1078: 89 pgs: 89 active+clean; 1.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-09-14T04:17:40.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:40.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:40 smithi114 ceph-mon[106969]: pgmap v1078: 89 pgs: 89 active+clean; 1.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-09-14T04:17:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:40 smithi114 ceph-mon[103029]: pgmap v1078: 89 pgs: 89 active+clean; 1.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-09-14T04:17:40.387 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 34608 34592 144.087 63.1875 0.00538856 0.00693431 2024-09-14T04:17:40.397 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:40.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:40.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:40.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:40.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:40.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:40.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:40.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:40.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:40.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:40.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:40.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:40.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:40.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:41.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:41.073 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:41.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:41.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:41.209 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:41.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:41.311 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:41.352 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:41.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:41.387 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 36503 36487 142.484 118.438 0.00755053 0.00701232 2024-09-14T04:17:41.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:41.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:41.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:41.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:41.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:41.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:41.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:41.761 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:41.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:41.768 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:17:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:17:41] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-09-14T04:17:41.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:41.908 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:41.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:42.010 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:42.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:42.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:42.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:42.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:42.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:42.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:42.285 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:42 smithi160 ceph-mon[88407]: pgmap v1079: 89 pgs: 89 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T04:17:42.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:42.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:42.362 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:42 smithi114 ceph-mon[103029]: pgmap v1079: 89 pgs: 89 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T04:17:42.363 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:42 smithi114 ceph-mon[106969]: pgmap v1079: 89 pgs: 89 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 145 MiB/s wr, 2.32k op/s 2024-09-14T04:17:42.387 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 38438 38422 141.216 120.938 0.0136524 0.0070744 2024-09-14T04:17:42.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:42.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:42.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:42.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:42.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:42.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:42.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:42.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:42.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:42.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:42.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:42.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:42.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:42.997 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:42.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:42.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:43.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:43.153 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:43.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:43.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:43.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:43.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:43.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:43.387 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 40318 40302 139.898 117.5 0.00533612 0.00714238 2024-09-14T04:17:43.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:43.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:43.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:43.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:43.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:43.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:43.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:43.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:43.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:43.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:43.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:43.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:43.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:43 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:43 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:43 smithi160 ceph-mon[88407]: pgmap v1080: 89 pgs: 89 active+clean; 2.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-14T04:17:43.924 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:43.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:43.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:43.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:44.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:44.101 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:44.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:44.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:43 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:44.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:43 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:44.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:43 smithi114 ceph-mon[106969]: pgmap v1080: 89 pgs: 89 active+clean; 2.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-14T04:17:44.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:43 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:44.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:43 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:44.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:43 smithi114 ceph-mon[103029]: pgmap v1080: 89 pgs: 89 active+clean; 2.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-14T04:17:44.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:44.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:44.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:44.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:44.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:44.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:44.387 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 15 42064 42049 138.281 109.188 0.00498552 0.00722681 2024-09-14T04:17:44.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:44.502 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:44.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:44.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:44.631 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:44.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:44.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:44.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:44.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:44.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:44.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:44.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:44.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:45.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:45.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:45.044 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:45.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:45.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:45.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:45.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:45.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:45.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:45.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:45.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:17:45.387482+0000 min lat: 0.00118641 max lat: 0.216139 avg lat: 0.00731176 2024-09-14T04:17:45.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:17:45.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 43762 43746 136.67 106.062 0.00643538 0.00731176 2024-09-14T04:17:45.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:45.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:45.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:45.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:45.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:45.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:45.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:45.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:45.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:45.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:45.837 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:45.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:45.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:45.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:45.996 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:45.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:46.063 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:17:46.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:46.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:46.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:46.149 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:46 smithi114 ceph-mon[106969]: pgmap v1081: 89 pgs: 89 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T04:17:46.149 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/415781047' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:17:46.151 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:46 smithi114 ceph-mon[103029]: pgmap v1081: 89 pgs: 89 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T04:17:46.151 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/415781047' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:17:46.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:46.278 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:46.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:46.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:46.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 45371 45355 134.95 100.562 0.00869326 0.00740466 2024-09-14T04:17:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:46 smithi160 ceph-mon[88407]: pgmap v1081: 89 pgs: 89 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-14T04:17:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/415781047' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:17:46.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:46.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:46.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:46.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:46.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:46.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:46.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:46.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:46.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:46.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:46.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:46.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:46.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:46.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:46.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:47.043 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:17:47.043 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:17:47.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:47.077 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.4 primary-affinity to 0.436282 (8285922) 2024-09-14T04:17:47.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:47.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:47.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/415781047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[106969]: osdmap e289: 8 total, 8 up, 8 in 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/286031351' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.43628202977914354} : dispatch 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.43628202977914354} : dispatch 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2944644749' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/415781047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[103029]: osdmap e289: 8 total, 8 up, 8 in 2024-09-14T04:17:47.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/286031351' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.43628202977914354} : dispatch 2024-09-14T04:17:47.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.43628202977914354} : dispatch 2024-09-14T04:17:47.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2944644749' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:47.136 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:17:47.136 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:17:47.161 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:17:47.161 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:17:47.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:47.196 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:17:47.196 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:17:47.223 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:17:47.223 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:17:47.226 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:47.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:47.242 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:17:47.242 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:17:47.259 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:17:47.259 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:17:47.276 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:17:47.276 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:17:47.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:47.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:47.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:47.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 46642 46626 132.427 79.4375 0.00240372 0.00746914 2024-09-14T04:17:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:47 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:17:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:47 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:17:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:47 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/415781047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:17:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:47 smithi160 ceph-mon[88407]: osdmap e289: 8 total, 8 up, 8 in 2024-09-14T04:17:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:47 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/286031351' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.43628202977914354} : dispatch 2024-09-14T04:17:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:47 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.43628202977914354} : dispatch 2024-09-14T04:17:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:47 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2944644749' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:17:47.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:47.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:47.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:47.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:47.597 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:47.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:47.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:47.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:47.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:47.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:47.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:47.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:47.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:47.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:47.970 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:47.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:48.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:48.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:48.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:48.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:48.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:48.222 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:48 smithi114 ceph-mon[103029]: pgmap v1083: 89 pgs: 89 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-14T04:17:48.222 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:48 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.43628202977914354}]': finished 2024-09-14T04:17:48.222 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:48 smithi114 ceph-mon[103029]: osdmap e290: 8 total, 8 up, 8 in 2024-09-14T04:17:48.223 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:48 smithi114 ceph-mon[106969]: pgmap v1083: 89 pgs: 89 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-14T04:17:48.223 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:48 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.43628202977914354}]': finished 2024-09-14T04:17:48.223 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:48 smithi114 ceph-mon[106969]: osdmap e290: 8 total, 8 up, 8 in 2024-09-14T04:17:48.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:48.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:48.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:48.259 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:17:48.259 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:17:48.259 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.259 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:46859 12.a 12:566425db:::benchmark_data_smithi114_253692_object46858:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-14T04:17:48.259 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:17:48.246906+0000", 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.011135785, 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011167437000000001, 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46859 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.260 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.246906+0000", 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.246906+0000", 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.246907+0000", 2024-09-14T04:17:48.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2219999999999999e-06 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.246926+0000", 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8972000000000001e-05 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.246934+0000", 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.011e-06 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.262 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.246939+0000", 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4539999999999998e-06 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.247407+0000", 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00046722099999999998 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.247530+0000", 2024-09-14T04:17:48.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000123681 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.247652+0000", 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000121867 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.254278+0000", 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0066262109999999999 2024-09-14T04:17:48.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.255993+0000", 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001714134 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46463.0:46870 12.c e291/284 12:31e87bfb:::benchmark_data_smithi114_253692_object46869:head v 291'2986, mlcod=291'2984)", 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:17:48.256835+0000", 2024-09-14T04:17:48.265 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001206377, 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001311546, 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.256835+0000", 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:17:48.266 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.256835+0000", 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.256836+0000", 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0440000000000001e-06 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.256859+0000", 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2274999999999999e-05 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:17:48.267 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.256870+0000", 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1090000000000001e-05 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.256876+0000", 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8430000000000004e-06 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.257107+0000", 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000230855 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:17:48.268 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:17:48.269 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:17:48.269 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.257127+0000", 2024-09-14T04:17:48.269 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9867000000000001e-05 2024-09-14T04:17:48.269 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:17:48.269 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:17:48.269 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:17:48.269 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:17:48.269 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:17:48.269 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 2 2024-09-14T04:17:48.269 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:17:48.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:48.386 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:48.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:48.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 47087 47071 127.879 27.8125 0.00561589 0.00781447 2024-09-14T04:17:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:48 smithi160 ceph-mon[88407]: pgmap v1083: 89 pgs: 89 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-14T04:17:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:48 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.43628202977914354}]': finished 2024-09-14T04:17:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:48 smithi160 ceph-mon[88407]: osdmap e290: 8 total, 8 up, 8 in 2024-09-14T04:17:48.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:48.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:48.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:48.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:48.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:48.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:48.758 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:48.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:48.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:48.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:48.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:48.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:49.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:49.072 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:49.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:49.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:49.215 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:49.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:49.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:49 smithi114 ceph-mon[103029]: osdmap e291: 8 total, 8 up, 8 in 2024-09-14T04:17:49.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:49 smithi114 ceph-mon[106969]: osdmap e291: 8 total, 8 up, 8 in 2024-09-14T04:17:49.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:49.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:49.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:49.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 48801 48785 127.014 107.125 0.00780271 0.00786726 2024-09-14T04:17:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:49 smithi160 ceph-mon[88407]: osdmap e291: 8 total, 8 up, 8 in 2024-09-14T04:17:49.448 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:49.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:49.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:49.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:49.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:49.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:49.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:49.757 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:49.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:49.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:49.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:49.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:49.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:17:49.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:50.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:50.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:50.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:50.221 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:50.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:50.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:50.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:50.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:50.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:50.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:50.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:50 smithi114 ceph-mon[106969]: pgmap v1086: 89 pgs: 89 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-14T04:17:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:50 smithi114 ceph-mon[103029]: pgmap v1086: 89 pgs: 89 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-14T04:17:50.389 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 50349 50333 125.803 96.75 0.0093564 0.00794378 2024-09-14T04:17:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:17:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:17:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:17:50.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:50 smithi160 ceph-mon[88407]: pgmap v1086: 89 pgs: 89 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-14T04:17:50.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:50.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:50.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:50.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-14T04:17:50.527 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:50.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:50.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:50.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:50.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:50.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:50.805 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:50.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:50.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:50.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:50.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:50.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:51.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:51.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:51.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:51.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:51.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:51.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:51.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:51.342 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:51.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:51.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:17:51.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 51763 51747 124.363 88.375 0.00900507 0.00803542 2024-09-14T04:17:51.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:51.490 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:51.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:51.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:51.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:17:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:17:51] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-09-14T04:17:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:51.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:51.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:51.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:51.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:51.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:51.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:51.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:51.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:51.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:52.027 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:52.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:52.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:52.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:52.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:52.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:52.305 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:52.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:52.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:52 smithi114 ceph-mon[103029]: pgmap v1087: 89 pgs: 89 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-14T04:17:52.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:52 smithi114 ceph-mon[106969]: pgmap v1087: 89 pgs: 89 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-14T04:17:52.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 53182 53166 123.042 88.6875 0.00235973 0.00811639 2024-09-14T04:17:52.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:52 smithi160 ceph-mon[88407]: pgmap v1087: 89 pgs: 89 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-14T04:17:52.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:52.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:52.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:52.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:52.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:52.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:52.709 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:52.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:52.811 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:52.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:52.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:52.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:52.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:52.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:53.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:53.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:53.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:53.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:53.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:53.215 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:53.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:53.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:53.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:53.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:53.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:53.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 54569 54553 121.744 86.6875 0.00674946 0.00820881 2024-09-14T04:17:53.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:53.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:53.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:53.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:53.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:53.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:53.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:53.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:53.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:53.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:53.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:53.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:53.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:53 smithi160 ceph-mon[88407]: pgmap v1088: 89 pgs: 89 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T04:17:53.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:54.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:54.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:54.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:54.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:54.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:53 smithi114 ceph-mon[106969]: pgmap v1088: 89 pgs: 89 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T04:17:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:17:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:17:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:53 smithi114 ceph-mon[103029]: pgmap v1088: 89 pgs: 89 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-14T04:17:54.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:54.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:54.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:54.265 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:54.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:54.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:54.388 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 55864 55848 120.336 80.9375 0.0125952 0.00830513 2024-09-14T04:17:54.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:54.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:54.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:54.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:54.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:54.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:54.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:54.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:54.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:54.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:54.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:54.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:54.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:54.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:54.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:55.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:55.039 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:17:55.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:17:55.039 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-14T04:17:55.039 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:17:55.040 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:17:55.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-14T04:17:55.067 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:55.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:55.169 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:55.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:55.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:55.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:55.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:55.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:55.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:55.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:55.389 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 57164 57148 119.033 81.25 0.0104143 0.00839624 2024-09-14T04:17:55.401 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:17:55.438 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:55.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:55.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:55.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:55.576 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:17:55.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:55.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:17:55.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:55.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:55.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:55.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:55.887 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:55.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:55.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:56.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:56.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:56.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:56.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:56.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:56.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:56.285 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:56.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:56.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:56.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:56 smithi114 ceph-mon[106969]: pgmap v1089: 89 pgs: 89 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-14T04:17:56.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:56 smithi114 ceph-mon[103029]: pgmap v1089: 89 pgs: 89 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-14T04:17:56.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:56.389 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 58531 58515 117.949 85.4375 0.0025374 0.00846549 2024-09-14T04:17:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:56 smithi160 ceph-mon[88407]: pgmap v1089: 89 pgs: 89 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-14T04:17:56.408 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:56.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:56.510 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:56.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:56.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:56.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:56.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:56.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:56.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:56.803 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:56.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:56.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:56.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:56.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:57.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:57.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:57.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:57.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:57.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:57.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:57.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:57.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:57.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:57.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:57.389 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 59523 59507 116.201 62 0.00438182 0.00859046 2024-09-14T04:17:57.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:57.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:57.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:57.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:57.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:57.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:57.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:57.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:57.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:57.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:57.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:57.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:57.976 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:57.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:58.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:58.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:58.048 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:17:58.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:58.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:58.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:58.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:17:58 smithi114 ceph-mon[103029]: pgmap v1090: 89 pgs: 89 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-14T04:17:58.165 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:17:58 smithi114 ceph-mon[106969]: pgmap v1090: 89 pgs: 89 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-14T04:17:58.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:58.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:58.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:58.389 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 60793 60777 115.084 79.375 0.00758295 0.00868353 2024-09-14T04:17:58.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:17:58 smithi160 ceph-mon[88407]: pgmap v1090: 89 pgs: 89 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-14T04:17:58.437 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:58.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:58.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:58.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:58.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:58.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:17:58.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:58.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:17:58.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:17:58.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:58.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:17:58.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:58.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:58.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:59.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:59.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:59.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:59.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:59.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:17:59.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:59.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:17:59.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:17:59.380 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:59.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:17:59.389 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 61967 61951 113.858 73.375 0.0196538 0.00877839 2024-09-14T04:17:59.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:17:59.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:59.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:17:59.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:59.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:59.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:17:59.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:17:59.786 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:17:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:59.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:17:59.842 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:17:59.842 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "" 2024-09-14T04:17:59.842 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:17:59.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:17:59.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:17:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:17:59.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:00.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:00.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:00.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:00.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:00 smithi160 ceph-mon[88407]: pgmap v1091: 89 pgs: 89 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-14T04:18:00.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:00.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:00.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:00.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:00.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:00.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:00.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:00 smithi114 ceph-mon[106969]: pgmap v1091: 89 pgs: 89 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-14T04:18:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:00 smithi114 ceph-mon[103029]: pgmap v1091: 89 pgs: 89 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-14T04:18:00.389 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 63243 63227 112.883 79.75 0.0118856 0.00885352 2024-09-14T04:18:00.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:00.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:00.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:00.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:00.569 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:00.574 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:18:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:18:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:18:00.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:00.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:00.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:00.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:00.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:00.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:00.867 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:00.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:00.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:01.012 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:01.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:01.052 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:18:01.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:01.153 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:01.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:01.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:01.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:01.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:01.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:01 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1076802422' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:18:01.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:01 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1076802422' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:18:01.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:01.389 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 64565 64549 112.042 82.625 0.00948456 0.00891937 2024-09-14T04:18:01.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:01 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1076802422' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:18:01.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:01.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:01.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:01.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:01.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:01.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:01.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:01.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:01.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:01.799 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:18:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:18:01] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-09-14T04:18:01.830 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:01.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:01.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:01.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:01.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:02.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:02 smithi114 ceph-mon[103029]: pgmap v1092: 89 pgs: 89 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-14T04:18:02.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:02.088 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:02.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:02.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:02.230 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:02.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:02.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:02 smithi160 ceph-mon[88407]: pgmap v1092: 89 pgs: 89 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-14T04:18:02.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:02 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:18:02.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:02 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1076802422' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:18:02.329 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:02 smithi160 ceph-mon[88407]: osdmap e292: 8 total, 8 up, 8 in 2024-09-14T04:18:02.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:02.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:02 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:18:02.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:02 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1076802422' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:18:02.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:02 smithi114 ceph-mon[103029]: osdmap e292: 8 total, 8 up, 8 in 2024-09-14T04:18:02.354 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:02 smithi114 ceph-mon[106969]: pgmap v1092: 89 pgs: 89 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-14T04:18:02.354 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:02 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:18:02.354 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:02 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1076802422' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:18:02.354 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:02 smithi114 ceph-mon[106969]: osdmap e292: 8 total, 8 up, 8 in 2024-09-14T04:18:02.367 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:02.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:02.389 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 65796 65780 111.093 76.9375 0.0104826 0.008997 2024-09-14T04:18:02.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:02.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:02.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:02.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:02.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:02.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:02.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:02.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:02.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:02.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:02.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:02.891 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:02.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:02.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:03.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:03.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:03.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:03.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:03.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:03.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:03.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:03.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:03.390 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 67026 67010 110.193 76.875 0.0152373 0.00906997 2024-09-14T04:18:03.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:03.423 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:03.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:03.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:03.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:03.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:03.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:03.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:03.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:03.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:03.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:03.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:03.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:03.920 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:03.949 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:03.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:04.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:04.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:04.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:04 smithi160 ceph-mon[88407]: pgmap v1094: 89 pgs: 89 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-14T04:18:04.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:04.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:04.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:04.299 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:04 smithi114 ceph-mon[103029]: pgmap v1094: 89 pgs: 89 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-14T04:18:04.299 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:18:04.300 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:18:04.300 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:04 smithi114 ceph-mon[106969]: pgmap v1094: 89 pgs: 89 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-14T04:18:04.344 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:18:04.344 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:18:04.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:04.375 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:04.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:04.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:04.384 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:18:04.384 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:18:04.391 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 68210 68194 109.265 74 0.00498774 0.00914637 2024-09-14T04:18:04.421 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:18:04.421 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:18:04.460 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:18:04.460 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:18:04.477 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:04.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:04.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:04.512 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:18:04.512 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:18:04.537 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:18:04.537 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:18:04.549 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:18:04.550 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:18:04.587 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:18:04.587 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:18:04.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:04.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:04.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:04.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:04.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:04.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:04.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:04.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:04.932 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:04.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:05.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:05.073 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:05.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:05.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:05 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2229382564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:05.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:05 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2229382564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:05.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:05.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:05.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:05.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:05.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:05.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:05.390 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:18:05.389589+0000 min lat: 0.00118641 max lat: 1.03496 avg lat: 0.00923075 2024-09-14T04:18:05.390 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:18:05.390 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 69318 69302 108.264 69.25 0.0101525 0.00923075 2024-09-14T04:18:05.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:05 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2229382564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:05.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:05.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:18:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:05.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:05.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:05.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:05.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:05.615 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:18:05.615 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:18:05.616 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-14T04:18:05.616 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:18:05.616 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:18:05.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-14T04:18:05.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:05.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:05.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:05.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:05.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:05.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:05.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:05.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:05.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:06.017 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:18:06.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:06.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:06.081 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:06 smithi160 ceph-mon[88407]: pgmap v1095: 89 pgs: 89 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-14T04:18:06.125 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:06.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:06.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:06.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:06.294 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:06.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:06.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:06.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:06 smithi114 ceph-mon[103029]: pgmap v1095: 89 pgs: 89 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-14T04:18:06.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:06 smithi114 ceph-mon[106969]: pgmap v1095: 89 pgs: 89 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-14T04:18:06.390 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 70541 70525 107.488 76.4375 0.0103134 0.00929561 2024-09-14T04:18:06.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:06.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:06.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:06.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:06.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:06.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:06.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:06.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:06.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:06.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:06.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:06.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:06.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:06.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:06.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:06.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:07.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:07.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:07.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:07.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:07.205 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:07.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:07.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:07.307 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:07.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:07.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:07.390 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 71608 71592 106.516 66.6875 0.0114406 0.00938372 2024-09-14T04:18:07.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:07.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:07.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:07.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:07.595 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:07.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:07.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:07.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:07.762 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:07.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:07.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:07.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:07.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:08.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:08.044 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:08.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:08.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:08.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:08.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:08.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:08.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:08.364 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:18:08.364 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-14T04:18:08.364 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:18:08.365 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:08 smithi160 ceph-mon[88407]: pgmap v1096: 89 pgs: 89 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-14T04:18:08.365 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:08.365 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:08.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:08 smithi114 ceph-mon[106969]: pgmap v1096: 89 pgs: 89 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-14T04:18:08.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:08.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:08.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:08 smithi114 ceph-mon[103029]: pgmap v1096: 89 pgs: 89 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-14T04:18:08.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:08.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:08.390 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 72846 72830 105.839 77.375 0.00889886 0.00944296 2024-09-14T04:18:08.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:08.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:08.559 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:08.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-14T04:18:08.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:08.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:08.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:08.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:08.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:08.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:08.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:08.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:08.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:09.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:09.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:09.116 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:09.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:09.173 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:09.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:09.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:09.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:09.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:09.392 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 74005 73989 105.075 72.4375 0.0120624 0.009511 2024-09-14T04:18:09.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:09.445 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:09.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:09.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:09.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:09.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:09.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:09.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:09.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:09.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:09.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:09.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:09.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:09.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:10.007 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:10.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:10.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:10.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:10.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:10.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:10.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:10.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:10.311 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:10.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:10 smithi114 ceph-mon[106969]: pgmap v1097: 89 pgs: 89 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-14T04:18:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:10 smithi114 ceph-mon[103029]: pgmap v1097: 89 pgs: 89 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-14T04:18:10.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:10.393 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 75226 75210 104.435 76.3125 0.0184697 0.00956997 2024-09-14T04:18:10.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:10 smithi160 ceph-mon[88407]: pgmap v1097: 89 pgs: 89 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-14T04:18:10.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:18:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:18:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:18:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:10.434 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:10.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:10.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:10.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:10.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:10.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:10.717 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:10.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:10.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:10.843 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:10.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:10.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:10.984 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:10.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:11.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:18:11.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:11.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:11.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:11.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:11.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:11.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:11.397 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 76377 76361 103.72 71.9375 0.0237719 0.00963497 2024-09-14T04:18:11.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:11.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:11.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:11.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:11.612 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:11.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:11.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:11.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:11.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:11.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:11.868 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:11.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:11.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:18:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:18:11] "GET /metrics HTTP/1.1" 200 36572 "" "Prometheus/2.51.0" 2024-09-14T04:18:11.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:12.003 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:12.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:12.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:12.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:12.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:12.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:12.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:12.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:12.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:12.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:12 smithi114 ceph-mon[103029]: pgmap v1098: 89 pgs: 89 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-14T04:18:12.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:12 smithi114 ceph-mon[106969]: pgmap v1098: 89 pgs: 89 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-14T04:18:12.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:12.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:12.396 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 77553 77537 103.077 73.5 0.0202633 0.00969667 2024-09-14T04:18:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:12 smithi160 ceph-mon[88407]: pgmap v1098: 89 pgs: 89 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-14T04:18:12.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:12.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:12.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:12.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:12.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:12.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:12.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:12.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:12.883 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:12.904 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:12.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:13.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:13.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:13.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:13.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:13.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:13.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:13.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:13.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:13.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:13.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:13.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:13.396 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 78364 78348 101.986 50.6875 0.0075461 0.00980026 2024-09-14T04:18:13.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:13.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:13.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:13.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:13.568 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:13.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:13.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:13.712 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:13.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:13.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:13.840 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:13.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:13.942 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:13.966 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:13.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:14.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:14.123 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:14.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:14.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:14.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:14.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:14.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:14.260 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:18:14.260 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:18:14.261 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-14T04:18:14.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:18:14.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:14.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:14 smithi114 ceph-mon[106969]: pgmap v1099: 89 pgs: 89 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-14T04:18:14.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:14 smithi114 ceph-mon[103029]: pgmap v1099: 89 pgs: 89 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-14T04:18:14.397 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 79454 79438 101.294 68.125 0.0233919 0.00986684 2024-09-14T04:18:14.398 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:14.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:14.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:14 smithi160 ceph-mon[88407]: pgmap v1099: 89 pgs: 89 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-14T04:18:14.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:14.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:14.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:14.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:14.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:14.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:14.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:14.839 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:14.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:14.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:14.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:14.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:15.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:15.130 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:18:15.144 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:15.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:15.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:15.258 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3604499635' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:18:15.258 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:15 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:18:15.258 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3604499635' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:18:15.258 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:15 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:18:15.275 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:15.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:15.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:15.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:15.397 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 80503 80487 100.579 65.5625 0.0190835 0.00993654 2024-09-14T04:18:15.401 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:15.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3604499635' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:18:15.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:15 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:18:15.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:15.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:15.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:15.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:15.667 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:15.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:15.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:15.812 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:15.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:15.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:15.968 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:15.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:15.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:16.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:16.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:16.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:16.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:16.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:16.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:16.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:16.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[103029]: pgmap v1100: 89 pgs: 89 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-14T04:18:16.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:18:16.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:18:16.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[103029]: osdmap e293: 8 total, 8 up, 8 in 2024-09-14T04:18:16.324 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:16.324 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:16.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[106969]: pgmap v1100: 89 pgs: 89 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-14T04:18:16.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:18:16.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:18:16.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[106969]: osdmap e293: 8 total, 8 up, 8 in 2024-09-14T04:18:16.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:16.328 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:16 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:16.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:16.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:16.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:16.397 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 81613 81597 99.9676 69.375 0.00793262 0.00999817 2024-09-14T04:18:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:16 smithi160 ceph-mon[88407]: pgmap v1100: 89 pgs: 89 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-14T04:18:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:16 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:18:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:16 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:18:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:16 smithi160 ceph-mon[88407]: osdmap e293: 8 total, 8 up, 8 in 2024-09-14T04:18:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:16 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:16 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:16.478 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:16.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:16.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:16.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:16.632 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:16.642 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:18:16.642 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:18:16.642 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:16.668 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:16.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:16.769 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:16.809 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:16.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:16.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:16.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:16.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:16.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:17.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:17.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:17.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:17.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:17.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:17.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:17.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:18:17.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:18:17.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:17.292 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:18:17.292 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:18:17.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:17.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:17.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:17.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:17.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:17.397 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 82650 82634 99.2915 64.8125 0.00900611 0.0100665 2024-09-14T04:18:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:18:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:18:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:17.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:17.471 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:17.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:17.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:17.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:17.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:17.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:17.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:17.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-14T04:18:17.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:17.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:17.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:17.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:17.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:17.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:18.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:18.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:18.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:18.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:18.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:18.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:18.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:18.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:18.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:18.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:18.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:18:18.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:18.397 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 83771 83755 98.74 70.0625 0.0102607 0.010122 2024-09-14T04:18:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:18.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:18.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:18.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:18.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:18.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:18.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:18.715 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:18.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:18.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:18.843 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:18.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:18.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:18 smithi114 ceph-mon[103029]: pgmap v1102: 89 pgs: 89 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T04:18:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:18 smithi114 ceph-mon[106969]: pgmap v1102: 89 pgs: 89 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T04:18:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:18 smithi160 ceph-mon[88407]: pgmap v1102: 89 pgs: 89 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-14T04:18:18.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:18.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:18.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:19.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:19.109 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:19.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:19.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:19.239 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:19.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:19.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:19.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:19.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:19.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 84854 84838 98.16 67.6875 0.0159582 0.0101819 2024-09-14T04:18:19.478 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:19.511 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:19.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:19.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:19.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:19.651 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:19.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:19.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:19.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:19.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:19.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:19.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:19.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:19 smithi114 ceph-mon[106969]: pgmap v1103: 89 pgs: 89 active+clean; 5.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 939 op/s 2024-09-14T04:18:19.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:19 smithi114 ceph-mon[103029]: pgmap v1103: 89 pgs: 89 active+clean; 5.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 939 op/s 2024-09-14T04:18:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:19.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:20.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:20.112 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:20.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:20.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:20.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:19 smithi160 ceph-mon[88407]: pgmap v1103: 89 pgs: 89 active+clean; 5.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 939 op/s 2024-09-14T04:18:20.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:20.243 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:20.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:20.344 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:20.372 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:20.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:20.400 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 85979 85963 97.6537 70.3125 0.0152519 0.0102351 2024-09-14T04:18:20.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:20.525 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:20.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:20.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:18:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:18:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:18:20.674 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:20.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:20.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:20.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:20.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:20.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:20.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:20.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:20.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:21.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:21.072 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:21.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:21.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:21.217 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:21.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:21.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:21.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:21.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:21.400 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 87044 87028 97.0985 66.5625 0.0116519 0.0102931 2024-09-14T04:18:21.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:21.511 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:21.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:21.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:21.596 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:18:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:18:21] "GET /metrics HTTP/1.1" 200 36572 "" "Prometheus/2.51.0" 2024-09-14T04:18:21.600 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:21.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:21.669 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:21.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:21.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:21.706 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:18:21.708 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:18:21.709 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:18:21.709 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:18:21.755 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:18:21.756 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:18:21.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:21.796 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:18:21.796 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:18:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:21.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:21.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:21.838 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:18:21.839 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:18:21.871 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:18:21.871 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:18:21.903 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:18:21.903 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:18:21.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:21.934 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:18:21.934 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:18:21.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:21.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:21.967 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:18:21.967 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:18:22.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:22.073 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:22.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:22.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:22 smithi114 ceph-mon[103029]: pgmap v1104: 89 pgs: 89 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-14T04:18:22.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:22 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2278621562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:22.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:22 smithi114 ceph-mon[106969]: pgmap v1104: 89 pgs: 89 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-14T04:18:22.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:22 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2278621562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:22.175 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:22.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:22.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:22.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:22.330 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:22.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:22.400 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 88004 87988 96.4477 60 0.0124546 0.0103635 2024-09-14T04:18:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:22 smithi160 ceph-mon[88407]: pgmap v1104: 89 pgs: 89 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-14T04:18:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:22 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2278621562' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:22.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:22.466 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:22.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:22.568 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:22.595 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:22.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:22.697 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:22.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:22.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:22.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:22.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:22.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:22.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:22.992 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:22.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:23.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:23.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:23.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:23.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:23.278 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:23.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:23 smithi114 ceph-mon[103029]: from='client.56002 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:18:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:23 smithi114 ceph-mon[106969]: from='client.56002 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:18:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:23.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:23.400 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 89080 89064 95.9438 67.25 0.0247345 0.0104175 2024-09-14T04:18:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:23 smithi160 ceph-mon[88407]: from='client.56002 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:18:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:23.419 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:23.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:23.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:23.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:23.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:23.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:23.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:23.711 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:23.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:23.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:23.854 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:23.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:23.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:23.983 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:23.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:23.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:24.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:24.119 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:24.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:24.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:24.221 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:24.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:24.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:24.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:24.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:24 smithi114 ceph-mon[106969]: pgmap v1105: 89 pgs: 89 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-14T04:18:24.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:24 smithi114 ceph-mon[103029]: pgmap v1105: 89 pgs: 89 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-14T04:18:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:24.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:24.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:24.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 90160 90144 95.4617 67.5 0.0119816 0.0104696 2024-09-14T04:18:24.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:24 smithi160 ceph-mon[88407]: pgmap v1105: 89 pgs: 89 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-14T04:18:24.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:24.536 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:24.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:24.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:24.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:24.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:24.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:24.794 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:24.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:24.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:24.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:24.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:24.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:25.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:25.062 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:25.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:25.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:25.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:25.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:25.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:25.302 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1104,"stamp":"2024-09-14T04:18:20.984152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5515581389,"num_objects":84178,"num_object_clones":0,"num_object_copies":252534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84178,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":84364,"num_write_kb":5387387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":595202,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84310,"ondisk_log_size":84310,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23026788,"kb_used_data":16366876,"kb_used_omap":33,"kb_used_meta":6659806,"kb_avail":726934428,"statfs":{"total":767960285184,"available":744380854272,"internally_reserved":0,"allocated":16759681024,"data_stored":16749042034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34628,"internal_metadata":6819641532},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":717684736,"num_objects":10951,"num_object_clones":0,"num_object_copies":32853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10951,"num_write_kb":700864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002698"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":397,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.138502+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:18:15.138502+0000","last_peered":"2024-09-14T04:18:15.138502+0000","last_clean":"2024-09-14T04:18:15.138502+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:18:15.138502+0000","last_undegraded":"2024-09-14T04:18:15.138502+0000","last_fullsized":"2024-09-14T04:18:15.138502+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":292,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.067120+0000","last_change":"2024-09-14T04:13:37.176637+0000","last_active":"2024-09-14T04:18:01.067120+0000","last_peered":"2024-09-14T04:18:01.067120+0000","last_clean":"2024-09-14T04:18:01.067120+0000","last_became_active":"2024-09-14T04:13:37.159759+0000","last_became_peered":"2024-09-14T04:13:37.159759+0000","last_unstale":"2024-09-14T04:18:01.067120+0000","last_undegraded":"2024-09-14T04:18:01.067120+0000","last_fullsized":"2024-09-14T04:18:01.067120+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:56.293937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":258,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060542+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:18:01.060542+0000","last_peered":"2024-09-14T04:18:01.060542+0000","last_clean":"2024-09-14T04:18:01.060542+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:18:01.060542+0000","last_undegraded":"2024-09-14T04:18:01.060542+0000","last_fullsized":"2024-09-14T04:18:01.060542+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":482,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.149664+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:18:15.149664+0000","last_peered":"2024-09-14T04:18:15.149664+0000","last_clean":"2024-09-14T04:18:15.149664+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:18:15.149664+0000","last_undegraded":"2024-09-14T04:18:15.149664+0000","last_fullsized":"2024-09-14T04:18:15.149664+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":467,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145712+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:18:15.145712+0000","last_peered":"2024-09-14T04:18:15.145712+0000","last_clean":"2024-09-14T04:18:15.145712+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:18:15.145712+0000","last_undegraded":"2024-09-14T04:18:15.145712+0000","last_fullsized":"2024-09-14T04:18:15.145712+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":255,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.054113+0000","last_change":"2024-09-14T04:14:30.541064+0000","last_active":"2024-09-14T04:18:01.054113+0000","last_peered":"2024-09-14T04:18:01.054113+0000","last_clean":"2024-09-14T04:18:01.054113+0000","last_became_active":"2024-09-14T04:14:30.540755+0000","last_became_peered":"2024-09-14T04:14:30.540755+0000","last_unstale":"2024-09-14T04:18:01.054113+0000","last_undegraded":"2024-09-14T04:18:01.054113+0000","last_fullsized":"2024-09-14T04:18:01.054113+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:37.189820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":210,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.135403+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:18:15.135403+0000","last_peered":"2024-09-14T04:18:15.135403+0000","last_clean":"2024-09-14T04:18:15.135403+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:18:15.135403+0000","last_undegraded":"2024-09-14T04:18:15.135403+0000","last_fullsized":"2024-09-14T04:18:15.135403+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":267,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.055971+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:18:01.055971+0000","last_peered":"2024-09-14T04:18:01.055971+0000","last_clean":"2024-09-14T04:18:01.055971+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:18:01.055971+0000","last_undegraded":"2024-09-14T04:18:01.055971+0000","last_fullsized":"2024-09-14T04:18:01.055971+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"293'5346","reported_seq":5369,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:18.978547+0000","last_change":"2024-09-14T04:17:48.104166+0000","last_active":"2024-09-14T04:18:18.978547+0000","last_peered":"2024-09-14T04:18:18.978547+0000","last_clean":"2024-09-14T04:18:18.978547+0000","last_became_active":"2024-09-14T04:17:48.083298+0000","last_became_peered":"2024-09-14T04:17:48.083298+0000","last_unstale":"2024-09-14T04:18:18.978547+0000","last_undegraded":"2024-09-14T04:18:18.978547+0000","last_fullsized":"2024-09-14T04:18:18.978547+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5346,"log_dups_size":0,"ondisk_log_size":5346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:53.010206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350355456,"num_objects":5346,"num_object_clones":0,"num_object_copies":16038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5346,"num_write_kb":342144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":277,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.054357+0000","last_change":"2024-09-14T04:14:30.543098+0000","last_active":"2024-09-14T04:18:01.054357+0000","last_peered":"2024-09-14T04:18:01.054357+0000","last_clean":"2024-09-14T04:18:01.054357+0000","last_became_active":"2024-09-14T04:14:30.542773+0000","last_became_peered":"2024-09-14T04:14:30.542773+0000","last_unstale":"2024-09-14T04:18:01.054357+0000","last_undegraded":"2024-09-14T04:18:01.054357+0000","last_fullsized":"2024-09-14T04:18:01.054357+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:06:27.423486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":241,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060064+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:18:01.060064+0000","last_peered":"2024-09-14T04:18:01.060064+0000","last_clean":"2024-09-14T04:18:01.060064+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:18:01.060064+0000","last_undegraded":"2024-09-14T04:18:01.060064+0000","last_fullsized":"2024-09-14T04:18:01.060064+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"293'5317","reported_seq":5333,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:18.937654+0000","last_change":"2024-09-14T04:17:13.575564+0000","last_active":"2024-09-14T04:18:18.937654+0000","last_peered":"2024-09-14T04:18:18.937654+0000","last_clean":"2024-09-14T04:18:18.937654+0000","last_became_active":"2024-09-14T04:17:13.575303+0000","last_became_peered":"2024-09-14T04:17:13.575303+0000","last_unstale":"2024-09-14T04:18:18.937654+0000","last_undegraded":"2024-09-14T04:18:18.937654+0000","last_fullsized":"2024-09-14T04:18:18.937654+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5317,"log_dups_size":0,"ondisk_log_size":5317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:02:07.463876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348454912,"num_objects":5317,"num_object_clones":0,"num_object_copies":15951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":340288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":27,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.143589+0000","last_change":"2024-09-14T04:17:48.084067+0000","last_active":"2024-09-14T04:18:15.143589+0000","last_peered":"2024-09-14T04:18:15.143589+0000","last_clean":"2024-09-14T04:18:15.143589+0000","last_became_active":"2024-09-14T04:17:48.083584+0000","last_became_peered":"2024-09-14T04:17:48.083584+0000","last_unstale":"2024-09-14T04:18:15.143589+0000","last_undegraded":"2024-09-14T04:18:15.143589+0000","last_fullsized":"2024-09-14T04:18:15.143589+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:01.274210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":315,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.143946+0000","last_change":"2024-09-14T04:17:48.083353+0000","last_active":"2024-09-14T04:18:15.143946+0000","last_peered":"2024-09-14T04:18:15.143946+0000","last_clean":"2024-09-14T04:18:15.143946+0000","last_became_active":"2024-09-14T04:17:48.083032+0000","last_became_peered":"2024-09-14T04:17:48.083032+0000","last_unstale":"2024-09-14T04:18:15.143946+0000","last_undegraded":"2024-09-14T04:18:15.143946+0000","last_fullsized":"2024-09-14T04:18:15.143946+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:15:25.358824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"293'5257","reported_seq":5282,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:18.818821+0000","last_change":"2024-09-14T04:17:48.088284+0000","last_active":"2024-09-14T04:18:18.818821+0000","last_peered":"2024-09-14T04:18:18.818821+0000","last_clean":"2024-09-14T04:18:18.818821+0000","last_became_active":"2024-09-14T04:17:48.083349+0000","last_became_peered":"2024-09-14T04:17:48.083349+0000","last_unstale":"2024-09-14T04:18:18.818821+0000","last_undegraded":"2024-09-14T04:18:18.818821+0000","last_fullsized":"2024-09-14T04:18:18.818821+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:56.489946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344522752,"num_objects":5257,"num_object_clones":0,"num_object_copies":15771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5257,"num_write_kb":336448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":54,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060477+0000","last_change":"2024-09-14T04:14:30.561646+0000","last_active":"2024-09-14T04:18:01.060477+0000","last_peered":"2024-09-14T04:18:01.060477+0000","last_clean":"2024-09-14T04:18:01.060477+0000","last_became_active":"2024-09-14T04:14:30.539174+0000","last_became_peered":"2024-09-14T04:14:30.539174+0000","last_unstale":"2024-09-14T04:18:01.060477+0000","last_undegraded":"2024-09-14T04:18:01.060477+0000","last_fullsized":"2024-09-14T04:18:01.060477+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:02:49.397620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":268,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.055922+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:18:01.055922+0000","last_peered":"2024-09-14T04:18:01.055922+0000","last_clean":"2024-09-14T04:18:01.055922+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:18:01.055922+0000","last_undegraded":"2024-09-14T04:18:01.055922+0000","last_fullsized":"2024-09-14T04:18:01.055922+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"293'5410","reported_seq":5439,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:19.324472+0000","last_change":"2024-09-14T04:17:13.574686+0000","last_active":"2024-09-14T04:18:19.324472+0000","last_peered":"2024-09-14T04:18:19.324472+0000","last_clean":"2024-09-14T04:18:19.324472+0000","last_became_active":"2024-09-14T04:17:13.574396+0000","last_became_peered":"2024-09-14T04:17:13.574396+0000","last_unstale":"2024-09-14T04:18:19.324472+0000","last_undegraded":"2024-09-14T04:18:19.324472+0000","last_fullsized":"2024-09-14T04:18:19.324472+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5410,"log_dups_size":0,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:06:20.930337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354549760,"num_objects":5410,"num_object_clones":0,"num_object_copies":16230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":346240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":279,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060423+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:18:01.060423+0000","last_peered":"2024-09-14T04:18:01.060423+0000","last_clean":"2024-09-14T04:18:01.060423+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:18:01.060423+0000","last_undegraded":"2024-09-14T04:18:01.060423+0000","last_fullsized":"2024-09-14T04:18:01.060423+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":503,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.144591+0000","last_change":"2024-09-14T04:13:37.165995+0000","last_active":"2024-09-14T04:18:15.144591+0000","last_peered":"2024-09-14T04:18:15.144591+0000","last_clean":"2024-09-14T04:18:15.144591+0000","last_became_active":"2024-09-14T04:13:37.165699+0000","last_became_peered":"2024-09-14T04:13:37.165699+0000","last_unstale":"2024-09-14T04:18:15.144591+0000","last_undegraded":"2024-09-14T04:18:15.144591+0000","last_fullsized":"2024-09-14T04:18:15.144591+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:21.792342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.4","version":"293'5162","reported_seq":5179,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:18.963496+0000","last_change":"2024-09-14T04:17:13.572270+0000","last_active":"2024-09-14T04:18:18.963496+0000","last_peered":"2024-09-14T04:18:18.963496+0000","last_clean":"2024-09-14T04:18:18.963496+0000","last_became_active":"2024-09-14T04:17:13.571986+0000","last_became_peered":"2024-09-14T04:17:13.571986+0000","last_unstale":"2024-09-14T04:18:18.963496+0000","last_undegraded":"2024-09-14T04:18:18.963496+0000","last_fullsized":"2024-09-14T04:18:18.963496+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:58:04.909769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338296832,"num_objects":5162,"num_object_clones":0,"num_object_copies":15486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5162,"num_write_kb":330368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":447,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.137224+0000","last_change":"2024-09-14T04:13:37.173224+0000","last_active":"2024-09-14T04:18:15.137224+0000","last_peered":"2024-09-14T04:18:15.137224+0000","last_clean":"2024-09-14T04:18:15.137224+0000","last_became_active":"2024-09-14T04:13:37.158630+0000","last_became_peered":"2024-09-14T04:13:37.158630+0000","last_unstale":"2024-09-14T04:18:15.137224+0000","last_undegraded":"2024-09-14T04:18:15.137224+0000","last_fullsized":"2024-09-14T04:18:15.137224+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:41:25.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":284,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.140529+0000","last_change":"2024-09-14T04:17:48.083765+0000","last_active":"2024-09-14T04:18:15.140529+0000","last_peered":"2024-09-14T04:18:15.140529+0000","last_clean":"2024-09-14T04:18:15.140529+0000","last_became_active":"2024-09-14T04:17:48.083385+0000","last_became_peered":"2024-09-14T04:17:48.083385+0000","last_unstale":"2024-09-14T04:18:15.140529+0000","last_undegraded":"2024-09-14T04:18:15.140529+0000","last_fullsized":"2024-09-14T04:18:15.140529+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:13:49.268399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"293'5275","reported_seq":5292,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:18.911568+0000","last_change":"2024-09-14T04:17:13.572972+0000","last_active":"2024-09-14T04:18:18.911568+0000","last_peered":"2024-09-14T04:18:18.911568+0000","last_clean":"2024-09-14T04:18:18.911568+0000","last_became_active":"2024-09-14T04:17:13.572619+0000","last_became_peered":"2024-09-14T04:17:13.572619+0000","last_unstale":"2024-09-14T04:18:18.911568+0000","last_undegraded":"2024-09-14T04:18:18.911568+0000","last_fullsized":"2024-09-14T04:18:18.911568+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5275,"log_dups_size":0,"ondisk_log_size":5275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:07:11.298450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345702400,"num_objects":5275,"num_object_clones":0,"num_object_copies":15825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5275,"num_write_kb":337600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":279,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.056006+0000","last_change":"2024-09-14T04:13:37.166773+0000","last_active":"2024-09-14T04:18:01.056006+0000","last_peered":"2024-09-14T04:18:01.056006+0000","last_clean":"2024-09-14T04:18:01.056006+0000","last_became_active":"2024-09-14T04:13:37.166465+0000","last_became_peered":"2024-09-14T04:13:37.166465+0000","last_unstale":"2024-09-14T04:18:01.056006+0000","last_undegraded":"2024-09-14T04:18:01.056006+0000","last_fullsized":"2024-09-14T04:18:01.056006+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:57:18.786263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":275,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.132294+0000","last_change":"2024-09-14T04:17:48.084357+0000","last_active":"2024-09-14T04:18:15.132294+0000","last_peered":"2024-09-14T04:18:15.132294+0000","last_clean":"2024-09-14T04:18:15.132294+0000","last_became_active":"2024-09-14T04:17:48.084034+0000","last_became_peered":"2024-09-14T04:17:48.084034+0000","last_unstale":"2024-09-14T04:18:15.132294+0000","last_undegraded":"2024-09-14T04:18:15.132294+0000","last_fullsized":"2024-09-14T04:18:15.132294+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:42:00.818438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"293'5258","reported_seq":5287,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:18.395285+0000","last_change":"2024-09-14T04:17:13.575249+0000","last_active":"2024-09-14T04:18:18.395285+0000","last_peered":"2024-09-14T04:18:18.395285+0000","last_clean":"2024-09-14T04:18:18.395285+0000","last_became_active":"2024-09-14T04:17:13.574960+0000","last_became_peered":"2024-09-14T04:17:13.574960+0000","last_unstale":"2024-09-14T04:18:18.395285+0000","last_undegraded":"2024-09-14T04:18:18.395285+0000","last_fullsized":"2024-09-14T04:18:18.395285+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5258,"log_dups_size":0,"ondisk_log_size":5258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:31.199701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344588288,"num_objects":5258,"num_object_clones":0,"num_object_copies":15774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5258,"num_write_kb":336512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":51,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.054015+0000","last_change":"2024-09-14T04:13:37.158807+0000","last_active":"2024-09-14T04:18:01.054015+0000","last_peered":"2024-09-14T04:18:01.054015+0000","last_clean":"2024-09-14T04:18:01.054015+0000","last_became_active":"2024-09-14T04:13:37.158475+0000","last_became_peered":"2024-09-14T04:13:37.158475+0000","last_unstale":"2024-09-14T04:18:01.054015+0000","last_undegraded":"2024-09-14T04:18:01.054015+0000","last_fullsized":"2024-09-14T04:18:01.054015+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:09.005093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":161,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.056220+0000","last_change":"2024-09-14T04:17:48.081945+0000","last_active":"2024-09-14T04:18:01.056220+0000","last_peered":"2024-09-14T04:18:01.056220+0000","last_clean":"2024-09-14T04:18:01.056220+0000","last_became_active":"2024-09-14T04:17:48.081510+0000","last_became_peered":"2024-09-14T04:17:48.081510+0000","last_unstale":"2024-09-14T04:18:01.056220+0000","last_undegraded":"2024-09-14T04:18:01.056220+0000","last_fullsized":"2024-09-14T04:18:01.056220+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:19:48.331487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"293'5138","reported_seq":5167,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:18.414364+0000","last_change":"2024-09-14T04:17:13.572496+0000","last_active":"2024-09-14T04:18:18.414364+0000","last_peered":"2024-09-14T04:18:18.414364+0000","last_clean":"2024-09-14T04:18:18.414364+0000","last_became_active":"2024-09-14T04:17:13.572182+0000","last_became_peered":"2024-09-14T04:17:13.572182+0000","last_unstale":"2024-09-14T04:18:18.414364+0000","last_undegraded":"2024-09-14T04:18:18.414364+0000","last_fullsized":"2024-09-14T04:18:18.414364+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5138,"log_dups_size":0,"ondisk_log_size":5138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:43:40.019232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336723968,"num_objects":5138,"num_object_clones":0,"num_object_copies":15414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5138,"num_write_kb":328832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":210,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.134321+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:18:15.134321+0000","last_peered":"2024-09-14T04:18:15.134321+0000","last_clean":"2024-09-14T04:18:15.134321+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:18:15.134321+0000","last_undegraded":"2024-09-14T04:18:15.134321+0000","last_fullsized":"2024-09-14T04:18:15.134321+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":447,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145873+0000","last_change":"2024-09-14T04:13:37.168484+0000","last_active":"2024-09-14T04:18:15.145873+0000","last_peered":"2024-09-14T04:18:15.145873+0000","last_clean":"2024-09-14T04:18:15.145873+0000","last_became_active":"2024-09-14T04:13:37.168136+0000","last_became_peered":"2024-09-14T04:13:37.168136+0000","last_unstale":"2024-09-14T04:18:15.145873+0000","last_undegraded":"2024-09-14T04:18:15.145873+0000","last_fullsized":"2024-09-14T04:18:15.145873+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:13.681730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"293'5362","reported_seq":5379,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:19.290436+0000","last_change":"2024-09-14T04:17:13.572747+0000","last_active":"2024-09-14T04:18:19.290436+0000","last_peered":"2024-09-14T04:18:19.290436+0000","last_clean":"2024-09-14T04:18:19.290436+0000","last_became_active":"2024-09-14T04:17:13.572451+0000","last_became_peered":"2024-09-14T04:17:13.572451+0000","last_unstale":"2024-09-14T04:18:19.290436+0000","last_undegraded":"2024-09-14T04:18:19.290436+0000","last_fullsized":"2024-09-14T04:18:19.290436+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5362,"log_dups_size":0,"ondisk_log_size":5362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:00:57.163738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351404032,"num_objects":5362,"num_object_clones":0,"num_object_copies":16086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5362,"num_write_kb":343168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":216,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.149446+0000","last_change":"2024-09-14T04:14:30.540124+0000","last_active":"2024-09-14T04:18:15.149446+0000","last_peered":"2024-09-14T04:18:15.149446+0000","last_clean":"2024-09-14T04:18:15.149446+0000","last_became_active":"2024-09-14T04:14:30.539704+0000","last_became_peered":"2024-09-14T04:14:30.539704+0000","last_unstale":"2024-09-14T04:18:15.149446+0000","last_undegraded":"2024-09-14T04:18:15.149446+0000","last_fullsized":"2024-09-14T04:18:15.149446+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:07:53.515714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":272,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060259+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:18:01.060259+0000","last_peered":"2024-09-14T04:18:01.060259+0000","last_clean":"2024-09-14T04:18:01.060259+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:18:01.060259+0000","last_undegraded":"2024-09-14T04:18:01.060259+0000","last_fullsized":"2024-09-14T04:18:01.060259+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":211,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.138042+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:18:15.138042+0000","last_peered":"2024-09-14T04:18:15.138042+0000","last_clean":"2024-09-14T04:18:15.138042+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:18:15.138042+0000","last_undegraded":"2024-09-14T04:18:15.138042+0000","last_fullsized":"2024-09-14T04:18:15.138042+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":261,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.056265+0000","last_change":"2024-09-14T04:13:37.166354+0000","last_active":"2024-09-14T04:18:01.056265+0000","last_peered":"2024-09-14T04:18:01.056265+0000","last_clean":"2024-09-14T04:18:01.056265+0000","last_became_active":"2024-09-14T04:13:37.166057+0000","last_became_peered":"2024-09-14T04:13:37.166057+0000","last_unstale":"2024-09-14T04:18:01.056265+0000","last_undegraded":"2024-09-14T04:18:01.056265+0000","last_fullsized":"2024-09-14T04:18:01.056265+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:40.817585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"293'5297","reported_seq":5326,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:19.340009+0000","last_change":"2024-09-14T04:17:13.570675+0000","last_active":"2024-09-14T04:18:19.340009+0000","last_peered":"2024-09-14T04:18:19.340009+0000","last_clean":"2024-09-14T04:18:19.340009+0000","last_became_active":"2024-09-14T04:17:13.569382+0000","last_became_peered":"2024-09-14T04:17:13.569382+0000","last_unstale":"2024-09-14T04:18:19.340009+0000","last_undegraded":"2024-09-14T04:18:19.340009+0000","last_fullsized":"2024-09-14T04:18:19.340009+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:10:19.648738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347144192,"num_objects":5297,"num_object_clones":0,"num_object_copies":15891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5297,"num_write_kb":339008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":160,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.142591+0000","last_change":"2024-09-14T04:14:30.539202+0000","last_active":"2024-09-14T04:18:15.142591+0000","last_peered":"2024-09-14T04:18:15.142591+0000","last_clean":"2024-09-14T04:18:15.142591+0000","last_became_active":"2024-09-14T04:14:30.538884+0000","last_became_peered":"2024-09-14T04:14:30.538884+0000","last_unstale":"2024-09-14T04:18:15.142591+0000","last_undegraded":"2024-09-14T04:18:15.142591+0000","last_fullsized":"2024-09-14T04:18:15.142591+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:39.511239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":497,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.144524+0000","last_change":"2024-09-14T04:13:37.165079+0000","last_active":"2024-09-14T04:18:15.144524+0000","last_peered":"2024-09-14T04:18:15.144524+0000","last_clean":"2024-09-14T04:18:15.144524+0000","last_became_active":"2024-09-14T04:13:37.164753+0000","last_became_peered":"2024-09-14T04:13:37.164753+0000","last_unstale":"2024-09-14T04:18:15.144524+0000","last_undegraded":"2024-09-14T04:18:15.144524+0000","last_fullsized":"2024-09-14T04:18:15.144524+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:50:32.658239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":479,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145570+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:18:15.145570+0000","last_peered":"2024-09-14T04:18:15.145570+0000","last_clean":"2024-09-14T04:18:15.145570+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:18:15.145570+0000","last_undegraded":"2024-09-14T04:18:15.145570+0000","last_fullsized":"2024-09-14T04:18:15.145570+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"293'5374","reported_seq":5391,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:18.965199+0000","last_change":"2024-09-14T04:17:13.572985+0000","last_active":"2024-09-14T04:18:18.965199+0000","last_peered":"2024-09-14T04:18:18.965199+0000","last_clean":"2024-09-14T04:18:18.965199+0000","last_became_active":"2024-09-14T04:17:13.572399+0000","last_became_peered":"2024-09-14T04:17:13.572399+0000","last_unstale":"2024-09-14T04:18:18.965199+0000","last_undegraded":"2024-09-14T04:18:18.965199+0000","last_fullsized":"2024-09-14T04:18:18.965199+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5374,"log_dups_size":0,"ondisk_log_size":5374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:58.922138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352190464,"num_objects":5374,"num_object_clones":0,"num_object_copies":16122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5374,"num_write_kb":343936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":473,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.133897+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:18:15.133897+0000","last_peered":"2024-09-14T04:18:15.133897+0000","last_clean":"2024-09-14T04:18:15.133897+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:18:15.133897+0000","last_undegraded":"2024-09-14T04:18:15.133897+0000","last_fullsized":"2024-09-14T04:18:15.133897+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":435,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.143723+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:18:15.143723+0000","last_peered":"2024-09-14T04:18:15.143723+0000","last_clean":"2024-09-14T04:18:15.143723+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:18:15.143723+0000","last_undegraded":"2024-09-14T04:18:15.143723+0000","last_fullsized":"2024-09-14T04:18:15.143723+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":204,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.132861+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:18:15.132861+0000","last_peered":"2024-09-14T04:18:15.132861+0000","last_clean":"2024-09-14T04:18:15.132861+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:18:15.132861+0000","last_undegraded":"2024-09-14T04:18:15.132861+0000","last_fullsized":"2024-09-14T04:18:15.132861+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.c","version":"293'5410","reported_seq":5439,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:19.295544+0000","last_change":"2024-09-14T04:17:13.570232+0000","last_active":"2024-09-14T04:18:19.295544+0000","last_peered":"2024-09-14T04:18:19.295544+0000","last_clean":"2024-09-14T04:18:19.295544+0000","last_became_active":"2024-09-14T04:17:13.569953+0000","last_became_peered":"2024-09-14T04:17:13.569953+0000","last_unstale":"2024-09-14T04:18:19.295544+0000","last_undegraded":"2024-09-14T04:18:19.295544+0000","last_fullsized":"2024-09-14T04:18:19.295544+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5410,"log_dups_size":0,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:09:54.113872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354549760,"num_objects":5410,"num_object_clones":0,"num_object_copies":16230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":346240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":273,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060250+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:18:01.060250+0000","last_peered":"2024-09-14T04:18:01.060250+0000","last_clean":"2024-09-14T04:18:01.060250+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:18:01.060250+0000","last_undegraded":"2024-09-14T04:18:01.060250+0000","last_fullsized":"2024-09-14T04:18:01.060250+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":230,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145239+0000","last_change":"2024-09-14T04:17:48.083263+0000","last_active":"2024-09-14T04:18:15.145239+0000","last_peered":"2024-09-14T04:18:15.145239+0000","last_clean":"2024-09-14T04:18:15.145239+0000","last_became_active":"2024-09-14T04:17:48.082791+0000","last_became_peered":"2024-09-14T04:17:48.082791+0000","last_unstale":"2024-09-14T04:18:15.145239+0000","last_undegraded":"2024-09-14T04:18:15.145239+0000","last_fullsized":"2024-09-14T04:18:15.145239+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:20:46.928898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":459,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.150625+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:18:15.150625+0000","last_peered":"2024-09-14T04:18:15.150625+0000","last_clean":"2024-09-14T04:18:15.150625+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:18:15.150625+0000","last_undegraded":"2024-09-14T04:18:15.150625+0000","last_fullsized":"2024-09-14T04:18:15.150625+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"293'5108","reported_seq":5137,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:17.135466+0000","last_change":"2024-09-14T04:17:13.575911+0000","last_active":"2024-09-14T04:18:17.135466+0000","last_peered":"2024-09-14T04:18:17.135466+0000","last_clean":"2024-09-14T04:18:17.135466+0000","last_became_active":"2024-09-14T04:17:13.575627+0000","last_became_peered":"2024-09-14T04:17:13.575627+0000","last_unstale":"2024-09-14T04:18:17.135466+0000","last_undegraded":"2024-09-14T04:18:17.135466+0000","last_fullsized":"2024-09-14T04:18:17.135466+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5108,"log_dups_size":0,"ondisk_log_size":5108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:32:39.359481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334757888,"num_objects":5108,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":326912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":296,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060668+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:18:01.060668+0000","last_peered":"2024-09-14T04:18:01.060668+0000","last_clean":"2024-09-14T04:18:01.060668+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:18:01.060668+0000","last_undegraded":"2024-09-14T04:18:01.060668+0000","last_fullsized":"2024-09-14T04:18:01.060668+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":473,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.151027+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:18:15.151027+0000","last_peered":"2024-09-14T04:18:15.151027+0000","last_clean":"2024-09-14T04:18:15.151027+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:18:15.151027+0000","last_undegraded":"2024-09-14T04:18:15.151027+0000","last_fullsized":"2024-09-14T04:18:15.151027+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":437,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145312+0000","last_change":"2024-09-14T04:13:37.163670+0000","last_active":"2024-09-14T04:18:15.145312+0000","last_peered":"2024-09-14T04:18:15.145312+0000","last_clean":"2024-09-14T04:18:15.145312+0000","last_became_active":"2024-09-14T04:13:37.163234+0000","last_became_peered":"2024-09-14T04:13:37.163234+0000","last_unstale":"2024-09-14T04:18:15.145312+0000","last_undegraded":"2024-09-14T04:18:15.145312+0000","last_fullsized":"2024-09-14T04:18:15.145312+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:53:34.767208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"293'5142","reported_seq":5171,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:18.784744+0000","last_change":"2024-09-14T04:17:13.573945+0000","last_active":"2024-09-14T04:18:18.784744+0000","last_peered":"2024-09-14T04:18:18.784744+0000","last_clean":"2024-09-14T04:18:18.784744+0000","last_became_active":"2024-09-14T04:17:13.573636+0000","last_became_peered":"2024-09-14T04:17:13.573636+0000","last_unstale":"2024-09-14T04:18:18.784744+0000","last_undegraded":"2024-09-14T04:18:18.784744+0000","last_fullsized":"2024-09-14T04:18:18.784744+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5142,"log_dups_size":0,"ondisk_log_size":5142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:34:37.026824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336986112,"num_objects":5142,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5142,"num_write_kb":329088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":434,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.151626+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:18:15.151626+0000","last_peered":"2024-09-14T04:18:15.151626+0000","last_clean":"2024-09-14T04:18:15.151626+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:18:15.151626+0000","last_undegraded":"2024-09-14T04:18:15.151626+0000","last_fullsized":"2024-09-14T04:18:15.151626+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":290,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.067634+0000","last_change":"2024-09-14T04:13:37.162059+0000","last_active":"2024-09-14T04:18:01.067634+0000","last_peered":"2024-09-14T04:18:01.067634+0000","last_clean":"2024-09-14T04:18:01.067634+0000","last_became_active":"2024-09-14T04:13:37.161728+0000","last_became_peered":"2024-09-14T04:13:37.161728+0000","last_unstale":"2024-09-14T04:18:01.067634+0000","last_undegraded":"2024-09-14T04:18:01.067634+0000","last_fullsized":"2024-09-14T04:18:01.067634+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:39:59.671661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":458,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.144710+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:18:15.144710+0000","last_peered":"2024-09-14T04:18:15.144710+0000","last_clean":"2024-09-14T04:18:15.144710+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:18:15.144710+0000","last_undegraded":"2024-09-14T04:18:15.144710+0000","last_fullsized":"2024-09-14T04:18:15.144710+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"293'5145","reported_seq":5174,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:18.799722+0000","last_change":"2024-09-14T04:17:13.576627+0000","last_active":"2024-09-14T04:18:18.799722+0000","last_peered":"2024-09-14T04:18:18.799722+0000","last_clean":"2024-09-14T04:18:18.799722+0000","last_became_active":"2024-09-14T04:17:13.576099+0000","last_became_peered":"2024-09-14T04:17:13.576099+0000","last_unstale":"2024-09-14T04:18:18.799722+0000","last_undegraded":"2024-09-14T04:18:18.799722+0000","last_fullsized":"2024-09-14T04:18:18.799722+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5145,"log_dups_size":0,"ondisk_log_size":5145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:31:19.819415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337182720,"num_objects":5145,"num_object_clones":0,"num_object_copies":15435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5145,"num_write_kb":329280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":426,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.135571+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:18:15.135571+0000","last_peered":"2024-09-14T04:18:15.135571+0000","last_clean":"2024-09-14T04:18:15.135571+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:18:15.135571+0000","last_undegraded":"2024-09-14T04:18:15.135571+0000","last_fullsized":"2024-09-14T04:18:15.135571+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":448,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.148978+0000","last_change":"2024-09-14T04:13:37.169517+0000","last_active":"2024-09-14T04:18:15.148978+0000","last_peered":"2024-09-14T04:18:15.148978+0000","last_clean":"2024-09-14T04:18:15.148978+0000","last_became_active":"2024-09-14T04:13:37.169027+0000","last_became_peered":"2024-09-14T04:13:37.169027+0000","last_unstale":"2024-09-14T04:18:15.148978+0000","last_undegraded":"2024-09-14T04:18:15.148978+0000","last_fullsized":"2024-09-14T04:18:15.148978+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:49:57.369201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":104,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.059932+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:18:01.059932+0000","last_peered":"2024-09-14T04:18:01.059932+0000","last_clean":"2024-09-14T04:18:01.059932+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:18:01.059932+0000","last_undegraded":"2024-09-14T04:18:01.059932+0000","last_fullsized":"2024-09-14T04:18:01.059932+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"293'5151","reported_seq":5180,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:17.139607+0000","last_change":"2024-09-14T04:17:13.577599+0000","last_active":"2024-09-14T04:18:17.139607+0000","last_peered":"2024-09-14T04:18:17.139607+0000","last_clean":"2024-09-14T04:18:17.139607+0000","last_became_active":"2024-09-14T04:17:13.577330+0000","last_became_peered":"2024-09-14T04:17:13.577330+0000","last_unstale":"2024-09-14T04:18:17.139607+0000","last_undegraded":"2024-09-14T04:18:17.139607+0000","last_fullsized":"2024-09-14T04:18:17.139607+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:59:12.058934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337575936,"num_objects":5151,"num_object_clones":0,"num_object_copies":15453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5151,"num_write_kb":329664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":493,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145712+0000","last_change":"2024-09-14T04:13:37.178106+0000","last_active":"2024-09-14T04:18:15.145712+0000","last_peered":"2024-09-14T04:18:15.145712+0000","last_clean":"2024-09-14T04:18:15.145712+0000","last_became_active":"2024-09-14T04:13:37.167009+0000","last_became_peered":"2024-09-14T04:13:37.167009+0000","last_unstale":"2024-09-14T04:18:15.145712+0000","last_undegraded":"2024-09-14T04:18:15.145712+0000","last_fullsized":"2024-09-14T04:18:15.145712+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:28:37.700664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":271,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.056357+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:18:01.056357+0000","last_peered":"2024-09-14T04:18:01.056357+0000","last_clean":"2024-09-14T04:18:01.056357+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:18:01.056357+0000","last_undegraded":"2024-09-14T04:18:01.056357+0000","last_fullsized":"2024-09-14T04:18:01.056357+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":107,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060072+0000","last_change":"2024-09-14T04:17:48.080621+0000","last_active":"2024-09-14T04:18:01.060072+0000","last_peered":"2024-09-14T04:18:01.060072+0000","last_clean":"2024-09-14T04:18:01.060072+0000","last_became_active":"2024-09-14T04:17:48.080300+0000","last_became_peered":"2024-09-14T04:17:48.080300+0000","last_unstale":"2024-09-14T04:18:01.060072+0000","last_undegraded":"2024-09-14T04:18:01.060072+0000","last_fullsized":"2024-09-14T04:18:01.060072+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:14:47.756124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":51,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.054328+0000","last_change":"2024-09-14T04:13:37.162516+0000","last_active":"2024-09-14T04:18:01.054328+0000","last_peered":"2024-09-14T04:18:01.054328+0000","last_clean":"2024-09-14T04:18:01.054328+0000","last_became_active":"2024-09-14T04:13:37.162248+0000","last_became_peered":"2024-09-14T04:13:37.162248+0000","last_unstale":"2024-09-14T04:18:01.054328+0000","last_undegraded":"2024-09-14T04:18:01.054328+0000","last_fullsized":"2024-09-14T04:18:01.054328+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:41.935908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":160,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.067094+0000","last_change":"2024-09-14T04:17:48.080930+0000","last_active":"2024-09-14T04:18:01.067094+0000","last_peered":"2024-09-14T04:18:01.067094+0000","last_clean":"2024-09-14T04:18:01.067094+0000","last_became_active":"2024-09-14T04:17:48.080575+0000","last_became_peered":"2024-09-14T04:17:48.080575+0000","last_unstale":"2024-09-14T04:18:01.067094+0000","last_undegraded":"2024-09-14T04:18:01.067094+0000","last_fullsized":"2024-09-14T04:18:01.067094+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:14:19.265519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":433,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145127+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:18:15.145127+0000","last_peered":"2024-09-14T04:18:15.145127+0000","last_clean":"2024-09-14T04:18:15.145127+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:18:15.145127+0000","last_undegraded":"2024-09-14T04:18:15.145127+0000","last_fullsized":"2024-09-14T04:18:15.145127+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":86,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.135685+0000","last_change":"2024-09-14T04:14:30.539928+0000","last_active":"2024-09-14T04:18:15.135685+0000","last_peered":"2024-09-14T04:18:15.135685+0000","last_clean":"2024-09-14T04:18:15.135685+0000","last_became_active":"2024-09-14T04:14:30.539572+0000","last_became_peered":"2024-09-14T04:14:30.539572+0000","last_unstale":"2024-09-14T04:18:15.135685+0000","last_undegraded":"2024-09-14T04:18:15.135685+0000","last_fullsized":"2024-09-14T04:18:15.135685+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:55:42.675243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":259,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060410+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:18:01.060410+0000","last_peered":"2024-09-14T04:18:01.060410+0000","last_clean":"2024-09-14T04:18:01.060410+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:18:01.060410+0000","last_undegraded":"2024-09-14T04:18:01.060410+0000","last_fullsized":"2024-09-14T04:18:01.060410+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":385,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145585+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:18:15.145585+0000","last_peered":"2024-09-14T04:18:15.145585+0000","last_clean":"2024-09-14T04:18:15.145585+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:18:15.145585+0000","last_undegraded":"2024-09-14T04:18:15.145585+0000","last_fullsized":"2024-09-14T04:18:15.145585+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":481,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145182+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:18:15.145182+0000","last_peered":"2024-09-14T04:18:15.145182+0000","last_clean":"2024-09-14T04:18:15.145182+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:18:15.145182+0000","last_undegraded":"2024-09-14T04:18:15.145182+0000","last_fullsized":"2024-09-14T04:18:15.145182+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":397,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.134402+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:18:15.134402+0000","last_peered":"2024-09-14T04:18:15.134402+0000","last_clean":"2024-09-14T04:18:15.134402+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:18:15.134402+0000","last_undegraded":"2024-09-14T04:18:15.134402+0000","last_fullsized":"2024-09-14T04:18:15.134402+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":268,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.056288+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:18:01.056288+0000","last_peered":"2024-09-14T04:18:01.056288+0000","last_clean":"2024-09-14T04:18:01.056288+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:18:01.056288+0000","last_undegraded":"2024-09-14T04:18:01.056288+0000","last_fullsized":"2024-09-14T04:18:01.056288+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":210,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.134235+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:18:15.134235+0000","last_peered":"2024-09-14T04:18:15.134235+0000","last_clean":"2024-09-14T04:18:15.134235+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:18:15.134235+0000","last_undegraded":"2024-09-14T04:18:15.134235+0000","last_fullsized":"2024-09-14T04:18:15.134235+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":483,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145638+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:18:15.145638+0000","last_peered":"2024-09-14T04:18:15.145638+0000","last_clean":"2024-09-14T04:18:15.145638+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:18:15.145638+0000","last_undegraded":"2024-09-14T04:18:15.145638+0000","last_fullsized":"2024-09-14T04:18:15.145638+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":446,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.144037+0000","last_change":"2024-09-14T04:13:37.163535+0000","last_active":"2024-09-14T04:18:15.144037+0000","last_peered":"2024-09-14T04:18:15.144037+0000","last_clean":"2024-09-14T04:18:15.144037+0000","last_became_active":"2024-09-14T04:13:37.163164+0000","last_became_peered":"2024-09-14T04:13:37.163164+0000","last_unstale":"2024-09-14T04:18:15.144037+0000","last_undegraded":"2024-09-14T04:18:15.144037+0000","last_fullsized":"2024-09-14T04:18:15.144037+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:13:25.324074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":217,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.134762+0000","last_change":"2024-09-14T04:13:37.162587+0000","last_active":"2024-09-14T04:18:15.134762+0000","last_peered":"2024-09-14T04:18:15.134762+0000","last_clean":"2024-09-14T04:18:15.134762+0000","last_became_active":"2024-09-14T04:13:37.162219+0000","last_became_peered":"2024-09-14T04:13:37.162219+0000","last_unstale":"2024-09-14T04:18:15.134762+0000","last_undegraded":"2024-09-14T04:18:15.134762+0000","last_fullsized":"2024-09-14T04:18:15.134762+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:52:47.043189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":51,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.054164+0000","last_change":"2024-09-14T04:13:37.164992+0000","last_active":"2024-09-14T04:18:01.054164+0000","last_peered":"2024-09-14T04:18:01.054164+0000","last_clean":"2024-09-14T04:18:01.054164+0000","last_became_active":"2024-09-14T04:13:37.164654+0000","last_became_peered":"2024-09-14T04:13:37.164654+0000","last_unstale":"2024-09-14T04:18:01.054164+0000","last_undegraded":"2024-09-14T04:18:01.054164+0000","last_fullsized":"2024-09-14T04:18:01.054164+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:49.248470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":475,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.133162+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:18:15.133162+0000","last_peered":"2024-09-14T04:18:15.133162+0000","last_clean":"2024-09-14T04:18:15.133162+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:18:15.133162+0000","last_undegraded":"2024-09-14T04:18:15.133162+0000","last_fullsized":"2024-09-14T04:18:15.133162+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":473,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.144793+0000","last_change":"2024-09-14T04:13:37.166866+0000","last_active":"2024-09-14T04:18:15.144793+0000","last_peered":"2024-09-14T04:18:15.144793+0000","last_clean":"2024-09-14T04:18:15.144793+0000","last_became_active":"2024-09-14T04:13:37.166453+0000","last_became_peered":"2024-09-14T04:13:37.166453+0000","last_unstale":"2024-09-14T04:18:15.144793+0000","last_undegraded":"2024-09-14T04:18:15.144793+0000","last_fullsized":"2024-09-14T04:18:15.144793+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:34:59.229028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":301,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.137148+0000","last_change":"2024-09-14T04:17:48.085379+0000","last_active":"2024-09-14T04:18:15.137148+0000","last_peered":"2024-09-14T04:18:15.137148+0000","last_clean":"2024-09-14T04:18:15.137148+0000","last_became_active":"2024-09-14T04:17:48.084944+0000","last_became_peered":"2024-09-14T04:17:48.084944+0000","last_unstale":"2024-09-14T04:18:15.137148+0000","last_undegraded":"2024-09-14T04:18:15.137148+0000","last_fullsized":"2024-09-14T04:18:15.137148+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:43:26.386934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":289,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060354+0000","last_change":"2024-09-14T04:13:37.174369+0000","last_active":"2024-09-14T04:18:01.060354+0000","last_peered":"2024-09-14T04:18:01.060354+0000","last_clean":"2024-09-14T04:18:01.060354+0000","last_became_active":"2024-09-14T04:13:37.165108+0000","last_became_peered":"2024-09-14T04:13:37.165108+0000","last_unstale":"2024-09-14T04:18:01.060354+0000","last_undegraded":"2024-09-14T04:18:01.060354+0000","last_fullsized":"2024-09-14T04:18:01.060354+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:43.400463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":301,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.138192+0000","last_change":"2024-09-14T04:17:48.081862+0000","last_active":"2024-09-14T04:18:15.138192+0000","last_peered":"2024-09-14T04:18:15.138192+0000","last_clean":"2024-09-14T04:18:15.138192+0000","last_became_active":"2024-09-14T04:17:48.081095+0000","last_became_peered":"2024-09-14T04:17:48.081095+0000","last_unstale":"2024-09-14T04:18:15.138192+0000","last_undegraded":"2024-09-14T04:18:15.138192+0000","last_fullsized":"2024-09-14T04:18:15.138192+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:04:07.654057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":272,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.060630+0000","last_change":"2024-09-14T04:13:37.168004+0000","last_active":"2024-09-14T04:18:01.060630+0000","last_peered":"2024-09-14T04:18:01.060630+0000","last_clean":"2024-09-14T04:18:01.060630+0000","last_became_active":"2024-09-14T04:13:37.159695+0000","last_became_peered":"2024-09-14T04:13:37.159695+0000","last_unstale":"2024-09-14T04:18:01.060630+0000","last_undegraded":"2024-09-14T04:18:01.060630+0000","last_fullsized":"2024-09-14T04:18:01.060630+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:18:40.779932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":210,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.145495+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:18:15.145495+0000","last_peered":"2024-09-14T04:18:15.145495+0000","last_clean":"2024-09-14T04:18:15.145495+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:18:15.145495+0000","last_undegraded":"2024-09-14T04:18:15.145495+0000","last_fullsized":"2024-09-14T04:18:15.145495+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":211,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.135187+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:18:15.135187+0000","last_peered":"2024-09-14T04:18:15.135187+0000","last_clean":"2024-09-14T04:18:15.135187+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:18:15.135187+0000","last_undegraded":"2024-09-14T04:18:15.135187+0000","last_fullsized":"2024-09-14T04:18:15.135187+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":51,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-14T04:18:01.053934+0000","last_change":"2024-09-14T04:13:37.165840+0000","last_active":"2024-09-14T04:18:01.053934+0000","last_peered":"2024-09-14T04:18:01.053934+0000","last_clean":"2024-09-14T04:18:01.053934+0000","last_became_active":"2024-09-14T04:13:37.165508+0000","last_became_peered":"2024-09-14T04:13:37.165508+0000","last_unstale":"2024-09-14T04:18:01.053934+0000","last_undegraded":"2024-09-14T04:18:01.053934+0000","last_fullsized":"2024-09-14T04:18:01.053934+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:24:02.783651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":472,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-14T04:18:15.144721+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:18:15.144721+0000","last_peered":"2024-09-14T04:18:15.144721+0000","last_clean":"2024-09-14T04:18:15.144721+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:18:15.144721+0000","last_undegraded":"2024-09-14T04:18:15.144721+0000","last_fullsized":"2024-09-14T04:18:15.144721+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":5514985472,"num_objects":84152,"num_object_clones":0,"num_object_copies":252456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84152,"num_write_kb":5385728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16495280128,"data_stored":16495280128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84152,"ondisk_log_size":84152,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4834,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430655,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3423456,"kb_used_data":2520408,"kb_used_omap":1,"kb_used_meta":903038,"kb_avail":90321696,"statfs":{"total":95995035648,"available":92489416704,"internally_reserved":0,"allocated":2580897792,"data_stored":2579550513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":924711256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626884,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3249300,"kb_used_data":2343628,"kb_used_omap":1,"kb_used_meta":905662,"kb_avail":90495852,"statfs":{"total":95995035648,"available":92667752448,"internally_reserved":0,"allocated":2399875072,"data_stored":2398539639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927398232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823112,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1879556,"kb_used_data":694204,"kb_used_omap":1,"kb_used_meta":1185342,"kb_avail":91865596,"statfs":{"total":95995035648,"available":94070370304,"internally_reserved":0,"allocated":710864896,"data_stored":709540814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1213790552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917194,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1327320,"kb_used_data":989004,"kb_used_omap":6,"kb_used_meta":338297,"kb_avail":92417832,"statfs":{"total":95995035648,"available":94635859968,"internally_reserved":0,"allocated":1012740096,"data_stored":1011404176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":346417151},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552493,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3400004,"kb_used_data":2656380,"kb_used_omap":4,"kb_used_meta":743611,"kb_avail":90345148,"statfs":{"total":95995035648,"available":92513431552,"internally_reserved":0,"allocated":2720133120,"data_stored":2718809119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":761458109},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435534,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3996932,"kb_used_data":3268984,"kb_used_omap":6,"kb_used_meta":727929,"kb_avail":89748220,"statfs":{"total":95995035648,"available":91902177280,"internally_reserved":0,"allocated":3347439616,"data_stored":3346118763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":745399757},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828004,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2758340,"kb_used_data":1868604,"kb_used_omap":5,"kb_used_meta":889722,"kb_avail":90986812,"statfs":{"total":95995035648,"available":93170495488,"internally_reserved":0,"allocated":1913450496,"data_stored":1912126424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":911075567},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":80,"seq":343597383992,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2991880,"kb_used_data":2025664,"kb_used_omap":5,"kb_used_meta":966202,"kb_avail":90753272,"statfs":{"total":95995035648,"available":92931350528,"internally_reserved":0,"allocated":2074279936,"data_stored":2072952586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":989390908},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2088042496,"data_stored":2088042496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1708195840,"data_stored":1708195840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3372548096,"data_stored":3372548096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2741436416,"data_stored":2741436416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1018167296,"data_stored":1018167296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":709754880,"data_stored":709754880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2418081792,"data_stored":2418081792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2439053312,"data_stored":2439053312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:18:25.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-14T04:18:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:25.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:25.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:25.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:18:25.400650+0000 min lat: 0.00118641 max lat: 1.03496 avg lat: 0.0105236 2024-09-14T04:18:25.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:18:25.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 91223 91207 94.978 66.4375 0.018352 0.0105236 2024-09-14T04:18:25.438 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:25.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:25.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:25.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:25.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:25.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:25.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:25.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:25.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:25.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:25.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:18:25.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:25.909 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:25.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:25.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:26.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:26.036 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:26.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:26.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:26.196 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:26.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:26.268 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:26 smithi114 ceph-mon[106969]: pgmap v1106: 89 pgs: 89 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-14T04:18:26.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:26 smithi114 ceph-mon[103029]: pgmap v1106: 89 pgs: 89 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-14T04:18:26.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:26.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:26.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:26.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 92161 92145 94.3821 58.625 0.0125656 0.0105859 2024-09-14T04:18:26.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:26 smithi160 ceph-mon[88407]: pgmap v1106: 89 pgs: 89 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-14T04:18:26.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:26.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:26.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:26.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:26.581 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:26.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:26.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:26.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:26.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:26.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:26.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:26.863 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:26.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:26.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:26.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:26.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:27.078 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:18:27.078 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:18:27.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.078 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:92830 12.8 12:1a13f4de:::benchmark_data_smithi114_253692_object92829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:27.051049+0000", 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.017387619, 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017417497000000001, 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 92830 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.079 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051049+0000", 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051049+0000", 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:27.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051050+0000", 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.669e-06 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051076+0000", 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5683000000000001e-05 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051096+0000", 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9670999999999999e-05 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051106+0000", 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0118e-05 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051169+0000", 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2978999999999995e-05 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051321+0000", 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000151896 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,2", 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051534+0000", 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021349699999999999 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.061567+0000", 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010032951 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.066674+0000", 2024-09-14T04:18:27.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0051071720000000001 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46463.0:92838 12.4 e293/284 12:27bb9ef0:::benchmark_data_smithi114_253692_object92837:head v 293'5703, mlcod=293'5700)", 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:27.051695+0000", 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.016740893, 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016844412999999999, 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:18:27.084 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051695+0000", 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051695+0000", 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:27.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051696+0000", 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5199999999999998e-07 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051723+0000", 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7356e-05 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051731+0000", 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7849999999999992e-06 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.051736+0000", 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5849999999999999e-06 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.057149+0000", 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054126560000000001 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.057170+0000", 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0752999999999999e-05 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:27.087 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46463.0:92833 12.9 e293/284 12:9f5dfe6f:::benchmark_data_smithi114_253692_object92832:head v 293'5789, mlcod=293'5786)", 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:27.062442+0000", 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0059938600000000002, 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0061452670000000003, 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.088 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.062442+0000", 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.062442+0000", 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.062443+0000", 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9899999999999999e-07 2024-09-14T04:18:27.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.062485+0000", 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2137e-05 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.062499+0000", 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3380000000000001e-05 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:27.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.062507+0000", 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5830000000000007e-06 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.062604+0000", 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.645e-05 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.062624+0000", 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9763999999999999e-05 2024-09-14T04:18:27.091 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:92846 12.2 12:47f16976:::benchmark_data_smithi114_253692_object92845:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:27.067133+0000", 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001303703, 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015033430000000001, 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:27.092 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 92846 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.067133+0000", 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.067133+0000", 2024-09-14T04:18:27.093 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.067135+0000", 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0540000000000002e-06 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.067158+0000", 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3669e-05 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:27.094 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.067176+0000", 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7427000000000001e-05 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.067185+0000", 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2830000000000007e-06 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.067272+0000", 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7435999999999994e-05 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.095 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.067407+0000", 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000134268 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:27.067545+0000", 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013823700000000001 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:18:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 4 2024-09-14T04:18:27.097 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:18:27.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:27.141 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:27.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:27.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:27.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:27.264 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:27.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:27.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:27.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:27.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:27.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 93148 93132 93.8548 61.6875 0.0224475 0.0106496 2024-09-14T04:18:27.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:27.535 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:27.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:27.621 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:27.623 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:27.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:27.703 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:27.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:27.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:27.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:27.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:27.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:27.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:27.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:28.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:28.079 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:18:28.114 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:28.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:28.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-14T04:18:28.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:28.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:28.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:28.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:28 smithi114 ceph-mon[106969]: pgmap v1107: 89 pgs: 89 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-14T04:18:28.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:28 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3830733905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:18:28.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:28 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:18:28.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:28 smithi114 ceph-mon[103029]: pgmap v1107: 89 pgs: 89 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-14T04:18:28.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:28 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3830733905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:18:28.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:28 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:18:28.380 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:28.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:28.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 94104 94088 93.3135 59.75 0.0415279 0.0107108 2024-09-14T04:18:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:28 smithi160 ceph-mon[88407]: pgmap v1107: 89 pgs: 89 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-14T04:18:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:28 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3830733905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:18:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:28 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:18:28.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:28.511 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:28.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:28.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:28.642 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:28.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:28.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:28.769 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:28.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:28.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:28.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:28.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:28.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:29.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:29.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:29.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:29.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:29.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:29.178 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:29.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:29.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:29 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:18:29.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:29 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:18:29.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:29 smithi114 ceph-mon[103029]: osdmap e294: 8 total, 8 up, 8 in 2024-09-14T04:18:29.209 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:29 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:18:29.209 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:29 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:18:29.209 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:29 smithi114 ceph-mon[106969]: osdmap e294: 8 total, 8 up, 8 in 2024-09-14T04:18:29.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:29.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:29.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:29.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 95134 95118 92.8613 64.375 0.0114947 0.0107638 2024-09-14T04:18:29.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:29 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:18:29.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:29 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:18:29.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:29 smithi160 ceph-mon[88407]: osdmap e294: 8 total, 8 up, 8 in 2024-09-14T04:18:29.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:29.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:29.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:29.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:29.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:29.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:29.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:29.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:29.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:29.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:29.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:29.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:30.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:30.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:30.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:30.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:30 smithi114 ceph-mon[106969]: pgmap v1109: 89 pgs: 89 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-14T04:18:30.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:30 smithi114 ceph-mon[103029]: pgmap v1109: 89 pgs: 89 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-14T04:18:30.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:30.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:30.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:30.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:30.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:30.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 96151 96135 92.4106 63.5625 0.00638911 0.0108157 2024-09-14T04:18:30.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:30 smithi160 ceph-mon[88407]: pgmap v1109: 89 pgs: 89 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-14T04:18:30.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:18:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:18:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:18:30.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:30.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:30.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:30.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:30.591 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:30.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:30.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:30.738 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:30.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:30.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:30.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:30.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:30.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:18:30.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:31.064 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:31.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:31.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:31.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:31.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:31.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:31.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:31.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:31.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 97092 97076 91.9015 58.8125 0.00959031 0.0108756 2024-09-14T04:18:31.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:31.501 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:31.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:31.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:31.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:31.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:31.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:31.790 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:31.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:31.871 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:18:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:18:31] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-09-14T04:18:31.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:31.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:31.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:32.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:32.074 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:32.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:32.176 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:32.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:32.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:32.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:32.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:32.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:32.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:32 smithi114 ceph-mon[106969]: pgmap v1110: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-14T04:18:32.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:32 smithi114 ceph-mon[103029]: pgmap v1110: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-14T04:18:32.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 98077 98061 91.4487 61.5625 0.0121916 0.0109296 2024-09-14T04:18:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:32 smithi160 ceph-mon[88407]: pgmap v1110: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-14T04:18:32.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:32.492 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:32.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:32.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:32.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:32.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:32.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:32.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:32.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:32.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:32.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:32.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:33.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:33.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:33.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:33.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:33.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:33.188 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:33.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:33.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:33.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:33.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:33.401 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 98960 98944 90.9155 55.1875 0.0200672 0.0109939 2024-09-14T04:18:33.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:33.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:33.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:33.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:33.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:33.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:33.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:33.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:33.712 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:33.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:33.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:33.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:33.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:33.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:33.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:33.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:33.987 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:33.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:34.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:34.131 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:34.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:34.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:34.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:34.275 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:34.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:34.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:34 smithi114 ceph-mon[106969]: pgmap v1111: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-14T04:18:34.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:34 smithi114 ceph-mon[103029]: pgmap v1111: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-14T04:18:34.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:34.402 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 99803 99787 90.361 52.6875 0.0190164 0.0110617 2024-09-14T04:18:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:34 smithi160 ceph-mon[88407]: pgmap v1111: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-14T04:18:34.414 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:34.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:34.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:34.549 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:34.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:34.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:34.686 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:34.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:34.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:34.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:34.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:34.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:34.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:34.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:34.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:35.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:35.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:35.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:35.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:35.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:35.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:35.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:35.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:35.360 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:35.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:35.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:35.402 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 100685 100669 89.8577 55.125 0.00855225 0.0111234 2024-09-14T04:18:35.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:35.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:35.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:35.595 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:35.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:35.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:35.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:35.762 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:35.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:35.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:35.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:35.895 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:35.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:35.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:36.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:36.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:36.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:36.146 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:36.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:36.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:36.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:36.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:36.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:36.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:36 smithi114 ceph-mon[106969]: pgmap v1112: 89 pgs: 89 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-14T04:18:36.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:36 smithi114 ceph-mon[103029]: pgmap v1112: 89 pgs: 89 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-14T04:18:36.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:36.402 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 101550 101534 89.3532 54.0625 0.0303495 0.011186 2024-09-14T04:18:36.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:36 smithi160 ceph-mon[88407]: pgmap v1112: 89 pgs: 89 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-14T04:18:36.428 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:36.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:36.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:36.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:36.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:36.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:36.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:36.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:36.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:36.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:36.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:36.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:36.961 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:36.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:37.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:37.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:37.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:37.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:37.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:37.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:37.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:37.372 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:37.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:37.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:37.402 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 102519 102503 88.9533 60.5625 0.00629845 0.0112359 2024-09-14T04:18:37.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:37.494 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:37.528 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:37.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:37.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:37.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:37.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:37.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:37.777 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:37.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:37.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:37.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:37.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:37.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:38.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:38.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:38.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:38.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:38.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:38.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:38.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:38.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:38.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:38.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:38 smithi114 ceph-mon[103029]: pgmap v1113: 89 pgs: 89 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-14T04:18:38.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:38.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:38 smithi114 ceph-mon[106969]: pgmap v1113: 89 pgs: 89 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-14T04:18:38.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:38.402 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 103393 103377 88.4831 54.625 0.0257938 0.0112956 2024-09-14T04:18:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:38 smithi160 ceph-mon[88407]: pgmap v1113: 89 pgs: 89 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-14T04:18:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:38.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:38.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:38.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:38.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:38.600 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:38.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:38.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:38.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:38.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:38.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:38.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:38.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:38.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:38.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:38.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:38.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:39.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:39.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:39.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:39.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:39.239 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3403177683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:39.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3403177683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:39.258 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:39.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:39.285 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:18:39.285 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:18:39.285 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:18:39.285 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:18:39.285 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:18:39.286 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:18:39.286 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:18:39.286 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:18:39.311 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:18:39.311 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:18:39.355 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:18:39.356 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:18:39.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:39.385 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:18:39.386 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:18:39.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:39.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:39.402 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 103936 103920 87.7461 33.9375 0.0128762 0.0113905 2024-09-14T04:18:39.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3403177683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:39.417 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:18:39.417 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:18:39.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:39.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:39.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:39.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:39.646 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:39.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:39.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:39.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:39.788 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:39.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:39.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:39.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:39.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:40.020 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:40.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:40.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:40.121 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 1 pgp_num to 1 2024-09-14T04:18:40.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:40.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:40.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:40.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:40.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:40.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:40.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:40 smithi114 ceph-mon[106969]: pgmap v1114: 89 pgs: 89 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-14T04:18:40.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/211823568' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-14T04:18:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:40 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-14T04:18:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:40 smithi114 ceph-mon[103029]: pgmap v1114: 89 pgs: 89 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-14T04:18:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/211823568' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-14T04:18:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:40 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-14T04:18:40.403 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 104875 104859 87.3587 58.6875 0.0146703 0.0114422 2024-09-14T04:18:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:18:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:18:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:18:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:40 smithi160 ceph-mon[88407]: pgmap v1114: 89 pgs: 89 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-14T04:18:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/211823568' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-14T04:18:40.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:40 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-14T04:18:40.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:40.474 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:40.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:40.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:40.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:40.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:40.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:40.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:40.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:40.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:40.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:40.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:40.882 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:40.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:40.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:41.030 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:41.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:41.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:41.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:41.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:41.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:41.299 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:41.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:41.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:41.403 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 105840 105824 87.0028 60.3125 0.0071617 0.011488 2024-09-14T04:18:41.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:41 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-14T04:18:41.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:41 smithi160 ceph-mon[88407]: osdmap e295: 8 total, 8 up, 8 in 2024-09-14T04:18:41.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:41.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:41.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:41 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-14T04:18:41.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:41 smithi114 ceph-mon[103029]: osdmap e295: 8 total, 8 up, 8 in 2024-09-14T04:18:41.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:41 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-14T04:18:41.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:41 smithi114 ceph-mon[106969]: osdmap e295: 8 total, 8 up, 8 in 2024-09-14T04:18:41.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:41.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:41.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:41.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:41.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:41.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:41.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:41.737 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:18:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:18:41] "GET /metrics HTTP/1.1" 200 36569 "" "Prometheus/2.51.0" 2024-09-14T04:18:41.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:41.830 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:41.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:41.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:41.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:41.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:41.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:42.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:42.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:42.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:42.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:42.192 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:42.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:42.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:42.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:42.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:42.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:42.363 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:42 smithi114 ceph-mon[106969]: pgmap v1116: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-14T04:18:42.364 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:42 smithi114 ceph-mon[103029]: pgmap v1116: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-14T04:18:42.403 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 106813 106797 86.6626 60.8125 0.0144901 0.011534 2024-09-14T04:18:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:42 smithi160 ceph-mon[88407]: pgmap v1116: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-14T04:18:42.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:42.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:42.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:42.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:18:42.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:42.612 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:42.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:42.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:42.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:42.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:42.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:42.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:42.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:42.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:42.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:43.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:43.063 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:43.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:43.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:43.165 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:43.191 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:43.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:43.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:43.328 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:43.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:43.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:43.403 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 107707 107691 86.2679 55.875 0.0203056 0.0115866 2024-09-14T04:18:43.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:43.456 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:43.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:43.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:43.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:43.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:43.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:43.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:43.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:43.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:43.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:43.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:43.958 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:18:43.958 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:18:43.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:44.008 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:44.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:44.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:44.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:44.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:44.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:44.288 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:44.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:44.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:44.403 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 108620 108604 85.8982 57.0625 0.0128141 0.0116372 2024-09-14T04:18:44.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:44 smithi160 ceph-mon[88407]: pgmap v1117: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-14T04:18:44.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:44.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:44.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:44 smithi114 ceph-mon[103029]: pgmap v1117: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-14T04:18:44.447 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:44 smithi114 ceph-mon[106969]: pgmap v1117: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-14T04:18:44.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:44.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:44.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:44.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:44.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:44.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:44.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:44.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:44.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:44.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:44.955 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:44.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:45.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-14T04:18:45.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:45.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:45.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:45.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:45.234 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:45.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:45.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:45.388 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:45.397 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:45.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:45.406 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:18:45.403105+0000 min lat: 0.00118641 max lat: 1.03496 avg lat: 0.0116924 2024-09-14T04:18:45.406 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:18:45.407 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 109466 109450 85.4854 52.875 0.0380387 0.0116924 2024-09-14T04:18:45.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:45.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:45.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:45.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:45.689 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:45.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:45.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:45.830 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:45.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:45.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:45.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:45.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:46.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:46.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:46.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:46.166 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:18:46.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:46.241 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:46.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:46.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:46.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:46.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:46.403 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 110327 110311 85.0944 53.8125 0.014834 0.0117468 2024-09-14T04:18:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:46 smithi160 ceph-mon[88407]: pgmap v1118: 89 pgs: 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 55 MiB/s wr, 883 op/s 2024-09-14T04:18:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4122378146' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:18:46.435 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:46.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:46.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:46.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:46.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:46.575 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:46 smithi114 ceph-mon[103029]: pgmap v1118: 89 pgs: 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 55 MiB/s wr, 883 op/s 2024-09-14T04:18:46.575 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4122378146' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:18:46.575 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:46 smithi114 ceph-mon[106969]: pgmap v1118: 89 pgs: 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 55 MiB/s wr, 883 op/s 2024-09-14T04:18:46.575 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4122378146' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:18:46.617 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:18:46.617 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:18:46.618 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-14T04:18:46.618 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-09-14T04:18:46.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-14T04:18:46.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:46.669 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:46.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:46.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:46.795 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:46.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:46.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:46.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:46.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:47.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:47.068 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:47.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:47.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:47.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:47.210 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:47.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:47.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:47.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:47.340 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:47.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:47.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:47 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:18:47.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:47 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:18:47.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4122378146' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:18:47.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:47 smithi114 ceph-mon[103029]: osdmap e296: 8 total, 8 up, 8 in 2024-09-14T04:18:47.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:47 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:18:47.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:47 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:18:47.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:47 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4122378146' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:18:47.349 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:47 smithi114 ceph-mon[106969]: osdmap e296: 8 total, 8 up, 8 in 2024-09-14T04:18:47.403 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 111195 111179 84.7183 54.25 0.0151074 0.0117981 2024-09-14T04:18:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:47 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:18:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:47 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:18:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:47 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4122378146' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:18:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:47 smithi160 ceph-mon[88407]: osdmap e296: 8 total, 8 up, 8 in 2024-09-14T04:18:47.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:47.501 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:47.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:47.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:47.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:47.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:47.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:47.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:47.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:47.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:47.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:47.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:48.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:48.072 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:48.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:48.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:48.200 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:48 smithi114 ceph-mon[103029]: pgmap v1120: 89 pgs: 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 54 MiB/s wr, 859 op/s 2024-09-14T04:18:48.201 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:48 smithi114 ceph-mon[106969]: pgmap v1120: 89 pgs: 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 54 MiB/s wr, 859 op/s 2024-09-14T04:18:48.202 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:48.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:48.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:48.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:48.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:48.403 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 112059 112043 84.3482 54 0.012091 0.0118506 2024-09-14T04:18:48.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:48.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:48.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:48.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:48.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:48.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:48.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:48 smithi160 ceph-mon[88407]: pgmap v1120: 89 pgs: 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 54 MiB/s wr, 859 op/s 2024-09-14T04:18:48.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:48.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:48.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:48.930 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:48.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:48.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:48.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:18:49.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:49.060 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:49.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:49.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:49.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:49.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:49.249 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:18:49.249 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:18:49.281 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:18:49.282 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:18:49.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:49.319 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:18:49.320 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:18:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:49.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:49.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:49.360 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:18:49.360 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:18:49.381 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:18:49.381 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:18:49.405 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 112861 112845 83.9396 50.125 0.0235372 0.0119081 2024-09-14T04:18:49.412 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:18:49.412 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:18:49.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:49.444 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:18:49.444 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:18:49.472 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:18:49.473 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:18:49.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:49.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:49.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:49.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:49.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:49.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:49.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:49.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:49.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:49.837 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:49.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:49.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:49.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:49.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:49.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:50.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:50.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:50.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:50.230 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:50.264 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:50.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:50.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:50.404 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:50.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:50.405 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 113776 113760 83.6249 57.1875 0.0293849 0.0119535 2024-09-14T04:18:50.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:50.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:50.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:50.601 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:50 smithi114 ceph-mon[106969]: pgmap v1121: 89 pgs: 89 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 46 MiB/s wr, 734 op/s 2024-09-14T04:18:50.602 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:50 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/899736834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:50.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:50 smithi114 ceph-mon[103029]: pgmap v1121: 89 pgs: 89 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 46 MiB/s wr, 734 op/s 2024-09-14T04:18:50.602 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:50 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/899736834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:50.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:50 smithi160 ceph-mon[88407]: pgmap v1121: 89 pgs: 89 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 46 MiB/s wr, 734 op/s 2024-09-14T04:18:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:50 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/899736834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:18:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:18:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:18:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:18:50.670 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:50.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:50.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:50.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:50.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:50.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:50.935 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:50.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:50.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:51.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:51.062 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:51.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:51.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:51.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:51.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:51.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:51.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:51.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:51.354 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:51.364 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:18:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:18:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:18:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:18:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172828 10.f e216/199 10:fd430cc2:::benchmark_data_smithi114_188657_object52632:head v 216'10769, mlcod=216'10767)", 2024-09-14T04:18:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.723005+0000", 2024-09-14T04:18:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 495.63619165300003, 2024-09-14T04:18:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32168082799999997, 2024-09-14T04:18:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723005+0000", 2024-09-14T04:18:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723005+0000", 2024-09-14T04:18:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723005+0000", 2024-09-14T04:18:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5099999999999999e-07 2024-09-14T04:18:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723012+0000", 2024-09-14T04:18:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8079999999999999e-06 2024-09-14T04:18:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723023+0000", 2024-09-14T04:18:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0505e-05 2024-09-14T04:18:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723030+0000", 2024-09-14T04:18:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2389999999999996e-06 2024-09-14T04:18:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723091+0000", 2024-09-14T04:18:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0639e-05 2024-09-14T04:18:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723110+0000", 2024-09-14T04:18:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9511e-05 2024-09-14T04:18:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043884+0000", 2024-09-14T04:18:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32077389299999998 2024-09-14T04:18:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044685+0000", 2024-09-14T04:18:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00080148199999999995 2024-09-14T04:18:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172833 10.d e216/199 10:bca70c7d:::benchmark_data_smithi114_188657_object52637:head v 216'10819, mlcod=216'10817)", 2024-09-14T04:18:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.723344+0000", 2024-09-14T04:18:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 495.635852688, 2024-09-14T04:18:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32066517999999999, 2024-09-14T04:18:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723344+0000", 2024-09-14T04:18:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723344+0000", 2024-09-14T04:18:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723344+0000", 2024-09-14T04:18:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8299999999999997e-07 2024-09-14T04:18:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723349+0000", 2024-09-14T04:18:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3370000000000001e-06 2024-09-14T04:18:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723357+0000", 2024-09-14T04:18:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0839999999999992e-06 2024-09-14T04:18:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.723362+0000", 2024-09-14T04:18:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5999999999999997e-06 2024-09-14T04:18:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724401+0000", 2024-09-14T04:18:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010383079999999999 2024-09-14T04:18:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724419+0000", 2024-09-14T04:18:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8598e-05 2024-09-14T04:18:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.729936+0000", 2024-09-14T04:18:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0055167020000000001 2024-09-14T04:18:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044009+0000", 2024-09-14T04:18:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31407296800000001 2024-09-14T04:18:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172830 10.3 e216/199 10:c6093d96:::benchmark_data_smithi114_188657_object52634:head v 216'10738, mlcod=216'10736)", 2024-09-14T04:18:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.724379+0000", 2024-09-14T04:18:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 495.63481684499999, 2024-09-14T04:18:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32095316000000002, 2024-09-14T04:18:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724379+0000", 2024-09-14T04:18:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724379+0000", 2024-09-14T04:18:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724380+0000", 2024-09-14T04:18:51.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7000000000000003e-07 2024-09-14T04:18:51.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724387+0000", 2024-09-14T04:18:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1360000000000002e-06 2024-09-14T04:18:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724397+0000", 2024-09-14T04:18:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0541e-05 2024-09-14T04:18:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724403+0000", 2024-09-14T04:18:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2979999999999997e-06 2024-09-14T04:18:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724457+0000", 2024-09-14T04:18:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3940000000000002e-05 2024-09-14T04:18:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.724475+0000", 2024-09-14T04:18:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.738e-05 2024-09-14T04:18:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043920+0000", 2024-09-14T04:18:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31944481600000002 2024-09-14T04:18:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.045333+0000", 2024-09-14T04:18:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001413079 2024-09-14T04:18:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172832 10.f e216/199 10:f5ac3d97:::benchmark_data_smithi114_188657_object52636:head v 216'10770, mlcod=216'10767)", 2024-09-14T04:18:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.726063+0000", 2024-09-14T04:18:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 495.63313309900002, 2024-09-14T04:18:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32223997999999998, 2024-09-14T04:18:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726063+0000", 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726063+0000", 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726064+0000", 2024-09-14T04:18:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.031e-06 2024-09-14T04:18:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726070+0000", 2024-09-14T04:18:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2570000000000004e-06 2024-09-14T04:18:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726082+0000", 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1039000000000001e-05 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.726088+0000", 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8020000000000003e-06 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.727200+0000", 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001112154 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.727219+0000", 2024-09-14T04:18:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8822e-05 2024-09-14T04:18:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.048262+0000", 2024-09-14T04:18:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32104314299999998 2024-09-14T04:18:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.048303+0000", 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0732000000000002e-05 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172838 10.5 e216/199 10:aaa76445:::benchmark_data_smithi114_188657_object52642:head v 216'10852, mlcod=216'10850)", 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.733532+0000", 2024-09-14T04:18:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 495.62566459599998, 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31986189300000001, 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.733532+0000", 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.733532+0000", 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.733533+0000", 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.003e-06 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.733540+0000", 2024-09-14T04:18:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3150000000000003e-06 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043891+0000", 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31035127000000001 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043905+0000", 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4028e-05 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044699+0000", 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00079361500000000001 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044722+0000", 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2858999999999999e-05 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.053357+0000", 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0086351780000000003 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.053394+0000", 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6625000000000003e-05 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.35997.0:172841 10.4 e216/199 10:225f533d:::benchmark_data_smithi114_188657_object52645:head v 216'10785, mlcod=216'10783)", 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:10:35.738220+0000", 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 495.620975891, 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31266361500000001, 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.738220+0000", 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.738220+0000", 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.738222+0000", 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.235e-06 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:35.738230+0000", 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8519999999999993e-06 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043948+0000", 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.305717303 2024-09-14T04:18:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.043957+0000", 2024-09-14T04:18:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9199999999999993e-06 2024-09-14T04:18:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044068+0000", 2024-09-14T04:18:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111172 2024-09-14T04:18:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.044092+0000", 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3966999999999999e-05 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.050850+0000", 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0067584020000000002 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:10:36.050884+0000", 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3763999999999997e-05 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:180572 11.b 11:d26d0e0b:::benchmark_data_smithi114_223345_object68635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:28.231833+0000", 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 203.127363608, 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.337487492, 2024-09-14T04:18:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:18:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:18:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180572 2024-09-14T04:18:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.231833+0000", 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.231833+0000", 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.231834+0000", 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4950000000000001e-06 2024-09-14T04:18:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.231839+0000", 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5449999999999997e-06 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.246468+0000", 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014629269 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.246483+0000", 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4858000000000001e-05 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.246569+0000", 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6098000000000001e-05 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.246681+0000", 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111637 2024-09-14T04:18:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:18:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.246783+0000", 2024-09-14T04:18:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000102367 2024-09-14T04:18:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.257473+0000", 2024-09-14T04:18:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010690204999999999 2024-09-14T04:18:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.266795+0000", 2024-09-14T04:18:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0093217270000000001 2024-09-14T04:18:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.568824+0000", 2024-09-14T04:18:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30202900999999999 2024-09-14T04:18:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569172+0000", 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00034821400000000001 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569320+0000", 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014806699999999999 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:180578 11.b 11:df0e673f:::benchmark_data_smithi114_223345_object68641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:28.252200+0000", 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 203.10699614500001, 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31729920499999997, 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180578 2024-09-14T04:18:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252200+0000", 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252200+0000", 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252202+0000", 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.181e-06 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252210+0000", 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5379999999999997e-06 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252225+0000", 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4722000000000001e-05 2024-09-14T04:18:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252234+0000", 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1600000000000004e-06 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252663+0000", 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000428923 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252761+0000", 2024-09-14T04:18:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8820000000000006e-05 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252873+0000", 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000111467 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.261327+0000", 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0084537459999999998 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.266852+0000", 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0055253109999999998 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569366+0000", 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30251435199999999 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569415+0000", 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8415000000000003e-05 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569499+0000", 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4569999999999998e-05 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:180582 11.b 11:db3884c7:::benchmark_data_smithi114_223345_object68645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:28.252508+0000", 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 203.10668871199999, 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31736006300000003, 2024-09-14T04:18:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180582 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252508+0000", 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252508+0000", 2024-09-14T04:18:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252509+0000", 2024-09-14T04:18:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0049999999999999e-06 2024-09-14T04:18:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252513+0000", 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.634e-06 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252525+0000", 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1754e-05 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.252532+0000", 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1130000000000003e-06 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.257502+0000", 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0049703689999999997 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.257609+0000", 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000106366 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:18:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.257712+0000", 2024-09-14T04:18:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103081 2024-09-14T04:18:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.266814+0000", 2024-09-14T04:18:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0091023019999999996 2024-09-14T04:18:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.568738+0000", 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30192428199999999 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569553+0000", 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00081403499999999995 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569759+0000", 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020639500000000001 2024-09-14T04:18:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569868+0000", 2024-09-14T04:18:51.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000108727 2024-09-14T04:18:51.432 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.432 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.432 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.433 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:180586 11.19 11:9db75938:::benchmark_data_smithi114_223345_object68649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-14T04:18:51.433 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:28.259834+0000", 2024-09-14T04:18:51.433 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 203.099362353, 2024-09-14T04:18:51.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30985690799999999, 2024-09-14T04:18:51.433 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.434 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.434 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.434 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:51.434 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:18:51.434 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:18:51.434 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180586 2024-09-14T04:18:51.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.434 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259834+0000", 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259834+0000", 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259835+0000", 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3939999999999999e-06 2024-09-14T04:18:51.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259839+0000", 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7570000000000002e-06 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259849+0000", 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.611e-06 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259854+0000", 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7150000000000003e-06 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262763+0000", 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029086400000000001 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262873+0000", 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110426 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.263325+0000", 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00045103999999999999 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.269338+0000", 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0060137100000000002 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.269356+0000", 2024-09-14T04:18:51.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8246000000000001e-05 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569616+0000", 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30025971800000001 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569657+0000", 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0575999999999999e-05 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569691+0000", 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4075000000000002e-05 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:180590 11.9 11:95d16aa3:::benchmark_data_smithi114_223345_object68653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:28.259866+0000", 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 203.09933052, 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30993884199999999, 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.440 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180590 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259866+0000", 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259866+0000", 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259866+0000", 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2100000000000002e-07 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259869+0000", 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6730000000000001e-06 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259875+0000", 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2519999999999996e-06 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.259880+0000", 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8230000000000002e-06 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.265874+0000", 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059937810000000001 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.266010+0000", 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013642999999999999 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:18:51.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.266454+0000", 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00044382200000000002 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.568675+0000", 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30222053100000001 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.568688+0000", 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3594999999999999e-05 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569726+0000", 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001038277 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569772+0000", 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5779999999999999e-05 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569805+0000", 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2456999999999998e-05 2024-09-14T04:18:51.446 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:180591 11.b 11:d2c4a078:::benchmark_data_smithi114_223345_object68654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:28.262025+0000", 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 203.09717144800001, 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30810161400000002, 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:51.447 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180591 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262025+0000", 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262025+0000", 2024-09-14T04:18:51.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262026+0000", 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3820000000000001e-06 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262031+0000", 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6840000000000004e-06 2024-09-14T04:18:51.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262043+0000", 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1736000000000001e-05 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262049+0000", 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7490000000000001e-06 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262100+0000", 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0201999999999998e-05 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262222+0000", 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012193699999999999 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,1", 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.262332+0000", 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110887 2024-09-14T04:18:51.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.266821+0000", 2024-09-14T04:18:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044888549999999999 2024-09-14T04:18:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.568776+0000", 2024-09-14T04:18:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30195498799999998 2024-09-14T04:18:51.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.569914+0000", 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00113747 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.570018+0000", 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104277 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:28.570126+0000", 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000108447 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:46612 12.2 12:4a477a48:::benchmark_data_smithi114_253692_object46611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:17:47.222463+0000", 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 64.136733140999993, 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87429041600000001, 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.454 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46612 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222463+0000", 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222463+0000", 2024-09-14T04:18:51.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222465+0000", 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5099999999999999e-06 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222486+0000", 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1673000000000001e-05 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222502+0000", 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6079e-05 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222510+0000", 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.17e-06 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222577+0000", 2024-09-14T04:18:51.457 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7605000000000001e-05 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222597+0000", 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9324999999999999e-05 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.092490+0000", 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.86989392600000004 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.459 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 114515 114499 83.1866 46.1875 0.0199672 0.0120155 2024-09-14T04:18:51.459 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.092608+0000", 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000117557 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.092742+0000", 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013379900000000001 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.094278+0000", 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015362329999999999 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.096603+0000", 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023254209999999998 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.096680+0000", 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6534999999999998e-05 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.096718+0000", 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8052000000000001e-05 2024-09-14T04:18:51.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.096754+0000", 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5531000000000003e-05 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:103388 12.b 12:df3c71cc:::benchmark_data_smithi114_253692_object103387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.396278+0000", 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 12.962918751, 2024-09-14T04:18:51.462 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.335246877, 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103388 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.396278+0000", 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.396278+0000", 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.396279+0000", 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.451e-06 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.396301+0000", 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1620999999999998e-05 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.396313+0000", 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2122e-05 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.396320+0000", 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1980000000000004e-06 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.398286+0000", 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019662909999999998 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.398407+0000", 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000120668 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.398527+0000", 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012023000000000001 2024-09-14T04:18:51.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404263+0000", 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0057360739999999999 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404325+0000", 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1401000000000004e-05 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:51.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731423+0000", 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32709817299999999 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731477+0000", 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4651999999999998e-05 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731524+0000", 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6996000000000002e-05 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.468 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:103393 12.b 12:d494b209:::benchmark_data_smithi114_253692_object103392:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.403844+0000", 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 12.955352059999999, 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.33982917099999999, 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:18:51.469 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103393 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.403844+0000", 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.403844+0000", 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.470 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.403846+0000", 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9230000000000001e-06 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.403872+0000", 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5939999999999999e-05 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.403885+0000", 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3108000000000001e-05 2024-09-14T04:18:51.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.403893+0000", 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6240000000000002e-06 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404352+0000", 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00045952800000000001 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.472 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404484+0000", 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000131729 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404614+0000", 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013006899999999999 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731410+0000", 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32679558800000003 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.735114+0000", 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037045699999999999 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.743569+0000", 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0084548880000000007 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.475 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.743624+0000", 2024-09-14T04:18:51.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5216000000000001e-05 2024-09-14T04:18:51.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.475 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.475 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.743673+0000", 2024-09-14T04:18:51.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8987999999999999e-05 2024-09-14T04:18:51.475 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.475 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.475 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46463.0:103395 12.3 e294/284 12:c488e3d1:::benchmark_data_smithi114_253692_object103394:head v 294'6565, mlcod=294'6563)", 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.405634+0000", 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 12.953561983, 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.34062460100000003, 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.476 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405634+0000", 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405634+0000", 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405635+0000", 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5999999999999991e-07 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405658+0000", 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3161000000000001e-05 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405670+0000", 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1620999999999999e-05 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405677+0000", 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9160000000000004e-06 2024-09-14T04:18:51.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.479 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.479 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405764+0000", 2024-09-14T04:18:51.479 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6923999999999996e-05 2024-09-14T04:18:51.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.479 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.479 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405781+0000", 2024-09-14T04:18:51.479 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6776e-05 2024-09-14T04:18:51.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.746229+0000", 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.34044843000000002 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.746259+0000", 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9813000000000001e-05 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.481 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.46463.0:103393 12.b e294/284 ondisk, result = 0)", 2024-09-14T04:18:51.481 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.407586+0000", 2024-09-14T04:18:51.481 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 12.951609822, 2024-09-14T04:18:51.481 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32394182100000002, 2024-09-14T04:18:51.481 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.481 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.481 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:18:51.481 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.407586+0000", 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.407586+0000", 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.407587+0000", 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7900000000000007e-07 2024-09-14T04:18:51.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.407597+0000", 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2650000000000002e-06 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.407609+0000", 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1819e-05 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.407616+0000", 2024-09-14T04:18:51.483 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2479999999999999e-06 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731390+0000", 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32377457700000001 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731402+0000", 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1985e-05 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731528+0000", 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012594799999999999 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:103398 12.8 12:1e25e08e:::benchmark_data_smithi114_253692_object103397:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.408340+0000", 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 12.950856352000001, 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.330258681, 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.485 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103398 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.408340+0000", 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.486 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.408340+0000", 2024-09-14T04:18:51.487 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.487 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.487 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.408341+0000", 2024-09-14T04:18:51.487 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2190000000000001e-06 2024-09-14T04:18:51.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.487 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.408361+0000", 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9775e-05 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731363+0000", 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32300205900000001 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731379+0000", 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6245000000000001e-05 2024-09-14T04:18:51.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731449+0000", 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9628999999999994e-05 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731609+0000", 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015996599999999999 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.489 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,2", 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731754+0000", 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014545399999999999 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.735291+0000", 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035363439999999999 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.738013+0000", 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027223080000000001 2024-09-14T04:18:51.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.738370+0000", 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00035674399999999998 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.738523+0000", 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015311099999999999 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.738599+0000", 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5827000000000004e-05 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46463.0:103408 12.4 e294/284 12:27e05a89:::benchmark_data_smithi114_253692_object103407:head v 294'6377, mlcod=294'6375)", 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.415645+0000", 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 12.943551025, 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32604008200000001, 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.492 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.493 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.493 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.493 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.415645+0000", 2024-09-14T04:18:51.493 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.493 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.493 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.493 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.493 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.415645+0000", 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.415647+0000", 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5209999999999999e-06 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.415681+0000", 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4555999999999999e-05 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.494 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731397+0000", 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31571613399999998 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731407+0000", 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8090000000000004e-06 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731510+0000", 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000102274 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.496 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.496 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731534+0000", 2024-09-14T04:18:51.496 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4536e-05 2024-09-14T04:18:51.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.496 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.496 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.741616+0000", 2024-09-14T04:18:51.496 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010081512000000001 2024-09-14T04:18:51.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.741685+0000", 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9740000000000007e-05 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46463.0:103415 12.3 e294/284 12:c72de31a:::benchmark_data_smithi114_253692_object103414:head v 294'6566, mlcod=294'6563)", 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.421845+0000", 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 12.937351746999999, 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32444007200000002, 2024-09-14T04:18:51.497 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421845+0000", 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421845+0000", 2024-09-14T04:18:51.498 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:18:51.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:18:51.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421846+0000", 2024-09-14T04:18:51.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2079999999999999e-06 2024-09-14T04:18:51.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:18:51.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421870+0000", 2024-09-14T04:18:51.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4127000000000001e-05 2024-09-14T04:18:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:18:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731369+0000", 2024-09-14T04:18:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30949863 2024-09-14T04:18:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:18:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731379+0000", 2024-09-14T04:18:51.500 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0063e-05 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731550+0000", 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017157199999999999 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.731573+0000", 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2603e-05 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:18:51.501 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.746264+0000", 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014691274000000001 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.746285+0000", 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0594999999999998e-05 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:18:51.502 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:18:51.503 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:51.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:51.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:51.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:51.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:51.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:51.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:51.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:51.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:51.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:51.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:18:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:18:51] "GET /metrics HTTP/1.1" 200 36569 "" "Prometheus/2.51.0" 2024-09-14T04:18:51.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:51.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:51.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:52.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:52.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:52.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:52.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:52.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:52.242 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:52.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:52.344 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:52.369 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:52.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:52.408 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 115402 115386 82.8676 55.4375 0.0125853 0.0120622 2024-09-14T04:18:52.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:52.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:52.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:52.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:52.546 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:52 smithi114 ceph-mon[106969]: pgmap v1122: 89 pgs: 89 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 MiB/s wr, 886 op/s 2024-09-14T04:18:52.547 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:52 smithi114 ceph-mon[103029]: pgmap v1122: 89 pgs: 89 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 MiB/s wr, 886 op/s 2024-09-14T04:18:52.596 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:52.624 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:52.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:52 smithi160 ceph-mon[88407]: pgmap v1122: 89 pgs: 89 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 MiB/s wr, 886 op/s 2024-09-14T04:18:52.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:52.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:52.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:52.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:52.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:52.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:52.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:53.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:53.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:53.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:53.163 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:53.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:53.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:53.299 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:53.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:53.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:53.408 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 116294 116278 82.5595 55.75 0.026561 0.0121071 2024-09-14T04:18:53.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:53.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:53.448 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:53.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:53.577 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:53.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:53.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:53.707 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:53.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:53.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:53.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:53.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:53.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:53 smithi114 ceph-mon[103029]: pgmap v1123: 89 pgs: 89 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 MiB/s wr, 886 op/s 2024-09-14T04:18:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:53 smithi114 ceph-mon[106969]: pgmap v1123: 89 pgs: 89 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 MiB/s wr, 886 op/s 2024-09-14T04:18:53.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:53.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:53.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:54.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:54.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-14T04:18:54.102 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:18:54.102 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:18:54.102 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:53 smithi160 ceph-mon[88407]: pgmap v1123: 89 pgs: 89 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 55 MiB/s wr, 886 op/s 2024-09-14T04:18:54.112 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:54.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:54.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:54.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:54.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:54.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:18:54.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:54.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 117174 117158 82.2498 55 0.0292231 0.0121524 2024-09-14T04:18:54.437 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:18:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:54.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:54.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:54.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:54.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:54.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:54.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:54.730 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:54.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:54.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:54.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:54.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:54.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:54.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:54.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:55.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:55.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:55.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:55.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:55.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:55.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:55.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:55.397 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:55.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:55.408 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0168 2024-09-14T04:18:55.408 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 117976 2024-09-14T04:18:55.408 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 81.9125 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 36.9369 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 229 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 27.8125 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1310 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 590.991 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3664 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 445 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0122036 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0156623 2024-09-14T04:18:55.409 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 1.03496 2024-09-14T04:18:55.410 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00118641 2024-09-14T04:18:55.415 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-14T04:18:55.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:55.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:55.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:55.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:55.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:55.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:55.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:55.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:55.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:55.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:55.917 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:55.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:56.018 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:56.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:56.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:56.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:56.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:56.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:56.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:56.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:56.315 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:56 smithi160 ceph-mon[88407]: pgmap v1124: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 52 MiB/s wr, 828 op/s 2024-09-14T04:18:56.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:56.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:56.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:56 smithi114 ceph-mon[103029]: pgmap v1124: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 52 MiB/s wr, 828 op/s 2024-09-14T04:18:56.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:56 smithi114 ceph-mon[106969]: pgmap v1124: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 52 MiB/s wr, 828 op/s 2024-09-14T04:18:56.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:56.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:56.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:56.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:18:56.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:56.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:18:56.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:56.725 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:56.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:56.766 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:18:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:18:56.766 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-14T04:18:56.766 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:18:56.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:56.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:56.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:56.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:56.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:57.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:57.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:57.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:57.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:57.242 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:57.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:57.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:57.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:57.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:57.467 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:57.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:57.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:57.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:57.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:57.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:57.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:57.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:57.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:57.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:57.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:57.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:57.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-14T04:18:58.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:18:58.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:58.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:58.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:18:58.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:18:58 smithi114 ceph-mon[106969]: pgmap v1125: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 48 MiB/s wr, 765 op/s 2024-09-14T04:18:58.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:18:58 smithi114 ceph-mon[103029]: pgmap v1125: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 48 MiB/s wr, 765 op/s 2024-09-14T04:18:58.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:58.210 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:58.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:58.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:58.341 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:18:58.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:58.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:18:58 smithi160 ceph-mon[88407]: pgmap v1125: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 48 MiB/s wr, 765 op/s 2024-09-14T04:18:58.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:58.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:58.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:58.489 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:18:58.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:58.595 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:58.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:18:58.697 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:58.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:58.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:58.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:58.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:58.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:58.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:59.019 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:59.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:59.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:18:59.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:59.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:18:59.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:18:59.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:59.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:18:59.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:18:59.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:59.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:18:59.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:18:59.565 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:18:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:59.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:18:59.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:59.698 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:59.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:59.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:18:59.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:59.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:18:59.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:18:59.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:18:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:18:59.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:00.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:00.063 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:19:00.071 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-14T04:19:00.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:00.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:00.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:00.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:00 smithi160 ceph-mon[88407]: pgmap v1126: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 85 B/s rd, 51 MiB/s wr, 965 op/s 2024-09-14T04:19:00.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2029993066' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:19:00.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:00 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:19:00.209 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/741119846' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:19:00.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:00.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:00.225 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:19:00.226 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:19:00.226 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:19:00.226 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:19:00.226 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:19:00.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:00.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:00.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:00.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:00 smithi114 ceph-mon[106969]: pgmap v1126: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 85 B/s rd, 51 MiB/s wr, 965 op/s 2024-09-14T04:19:00.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2029993066' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:19:00.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:00 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:19:00.372 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/741119846' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:19:00.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:00 smithi114 ceph-mon[103029]: pgmap v1126: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 85 B/s rd, 51 MiB/s wr, 965 op/s 2024-09-14T04:19:00.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2029993066' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:19:00.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:00 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:19:00.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/741119846' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:19:00.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:00.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:00.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:00.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:00.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:00.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:00.630 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:19:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:19:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:19:00.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:00.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:00.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:00.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:00.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:00.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:01.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:01.037 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:01.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:01.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:01.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-14T04:19:01.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:01.173 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:01.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:01.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:01.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:01.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:01.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:01 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:01.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:01 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:19:01.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:01 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/741119846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:19:01.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:01 smithi114 ceph-mon[106969]: osdmap e297: 8 total, 8 up, 8 in 2024-09-14T04:19:01.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:01 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:01.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:01 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:19:01.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:01 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/741119846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:19:01.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:01 smithi114 ceph-mon[103029]: osdmap e297: 8 total, 8 up, 8 in 2024-09-14T04:19:01.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:01.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:01 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:01.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:01 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:19:01.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:01 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/741119846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:19:01.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:01 smithi160 ceph-mon[88407]: osdmap e297: 8 total, 8 up, 8 in 2024-09-14T04:19:01.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:01.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:01.493 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:01.541 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:01.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:01.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:01.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:01.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:01.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:01.779 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:19:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:19:01] "GET /metrics HTTP/1.1" 200 36571 "" "Prometheus/2.51.0" 2024-09-14T04:19:01.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:01.850 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:01.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:01.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:01.977 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:01.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:02.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:02.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:02.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:02.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:02.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:02.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:02.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:02.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:02.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:02.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:02 smithi114 ceph-mon[106969]: pgmap v1128: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 798 op/s 2024-09-14T04:19:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:02 smithi114 ceph-mon[103029]: pgmap v1128: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 798 op/s 2024-09-14T04:19:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:02 smithi160 ceph-mon[88407]: pgmap v1128: 89 pgs: 89 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 798 op/s 2024-09-14T04:19:02.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:02.495 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:19:02.495 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:19:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:02.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:02.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:02.552 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:19:02.552 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:19:02.581 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:19:02.581 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:19:02.609 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:02.637 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:02.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:02.645 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:19:02.645 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:19:02.675 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:19:02.675 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:19:02.707 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:19:02.707 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:19:02.738 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:19:02.738 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:19:02.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:02.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:02.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:02.777 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:19:02.777 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:19:02.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:02.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:02.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:03.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:03.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:03.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:03.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:03.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:03.173 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:03.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:03.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:03.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:03.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:03.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:03.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:03 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3978308996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:03.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:03 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3978308996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:03.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:03.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:03 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3978308996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:03.431 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:03.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:03.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:03.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:03.570 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:03.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:03.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:03.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:03.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:03.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:03.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:19:03.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:03.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:03.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:03.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:03.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:03.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:04.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:04.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:04.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:04.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:04.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:04 smithi114 ceph-mon[103029]: pgmap v1129: 89 pgs: 89 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 793 op/s 2024-09-14T04:19:04.133 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:04 smithi114 ceph-mon[106969]: pgmap v1129: 89 pgs: 89 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 793 op/s 2024-09-14T04:19:04.181 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:19:04.181 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:19:04.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:04.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:04.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:04.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:04.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:04.360 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:04.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:04.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:04.410 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:04 smithi160 ceph-mon[88407]: pgmap v1129: 89 pgs: 89 active+clean; 7.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 793 op/s 2024-09-14T04:19:04.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:04.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:04.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:04.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:04.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:04.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:04.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:04.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:04.761 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:04.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:04.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:04.885 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:04.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:04.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:04.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-14T04:19:04.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:05.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:05.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:05.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:05.160 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:05.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:05.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:05.290 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:05.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:05.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:05.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:05.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:05.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:05.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:05.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:05.547 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:05.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:05.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:05.685 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:05.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:05.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:05.821 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:05.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:05.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:05.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:05.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:06.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:06.078 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:06.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:06.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:06.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:06.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:06.214 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:06 smithi114 ceph-mon[106969]: pgmap v1130: 89 pgs: 89 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 812 op/s 2024-09-14T04:19:06.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:06 smithi114 ceph-mon[103029]: pgmap v1130: 89 pgs: 89 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 812 op/s 2024-09-14T04:19:06.309 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:06.334 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:06.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:06.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:06 smithi160 ceph-mon[88407]: pgmap v1130: 89 pgs: 89 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 812 op/s 2024-09-14T04:19:06.436 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:06.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:06.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:06.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:06.610 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:06.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:06.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:06.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:06.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:06.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:06.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:06.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:19:06.894 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:06.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:06.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:07.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:07.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:07.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:07.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:07.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:07.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:07.159 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:19:07.160 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:19:07.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:07.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:07.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:07.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:07.350 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:07.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:07.411 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:07.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:07.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:07.537 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:07.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:07.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:07.668 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:07.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:07.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:07.794 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:07.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:07.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:07.943 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:07.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:08.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:08.079 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:08.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:08.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-14T04:19:08.152 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:19:08.181 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:08.206 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:08.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:08.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:08.308 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:08.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:08.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:08 smithi160 ceph-mon[88407]: pgmap v1131: 89 pgs: 89 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 812 op/s 2024-09-14T04:19:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3059083665' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:08 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:08.435 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:08.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:08.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:08.489 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:08.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:08 smithi114 ceph-mon[106969]: pgmap v1131: 89 pgs: 89 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 812 op/s 2024-09-14T04:19:08.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3059083665' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:08.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:08 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:08.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:08.500 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:08.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:08 smithi114 ceph-mon[103029]: pgmap v1131: 89 pgs: 89 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 812 op/s 2024-09-14T04:19:08.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3059083665' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:08.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:08 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:08.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:08.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:08.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:08.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:08.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:08.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:08.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:08.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:08.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:08.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:08.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:09.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:09.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:09.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:09.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:09.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:09.134 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:19:09.134 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:19:09.134 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-14T04:19:09.134 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:19:09.134 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:19:09.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-14T04:19:09.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:09.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:09.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:09.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:09.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:09 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:19:09.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:09 smithi160 ceph-mon[88407]: osdmap e298: 8 total, 8 up, 8 in 2024-09-14T04:19:09.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:09.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:09.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:09.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:09.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:09.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:09.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:09.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:09 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:09.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:19:09.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:09 smithi114 ceph-mon[106969]: osdmap e298: 8 total, 8 up, 8 in 2024-09-14T04:19:09.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:09 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:09.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:19:09.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:09 smithi114 ceph-mon[103029]: osdmap e298: 8 total, 8 up, 8 in 2024-09-14T04:19:09.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:09.725 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:09.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:09.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:09.883 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:09.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:09.985 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:10.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:10.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:10.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:10.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:10.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:10.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:10.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:10.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:10.356 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:19:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:19:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:19:10.356 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:10 smithi160 ceph-mon[88407]: pgmap v1133: 89 pgs: 89 active+clean; 6.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 839 op/s 2024-09-14T04:19:10.366 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:19:10.366 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:19:10.410 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:10 smithi114 ceph-mon[103029]: pgmap v1133: 89 pgs: 89 active+clean; 6.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 839 op/s 2024-09-14T04:19:10.411 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:10 smithi114 ceph-mon[106969]: pgmap v1133: 89 pgs: 89 active+clean; 6.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 839 op/s 2024-09-14T04:19:10.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:10.453 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:10.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:10.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:10.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:10.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:10.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:10.761 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:10.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:10.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:10.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:10.906 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:10.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:11.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:11.029 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:19:11.030 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:19:11.051 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:11.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:11.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:11.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-14T04:19:11.063 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:19:11.064 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:19:11.091 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:11.101 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:19:11.102 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:19:11.139 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:19:11.139 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:19:11.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:11.170 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:19:11.170 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:19:11.179 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:11.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:11.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:11.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:11 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2031461217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:11.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:11 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2031461217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:11.206 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:19:11.206 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:19:11.241 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:19:11.241 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:19:11.272 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:19:11.273 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:19:11.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:11.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:11.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:11.387 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:11.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:11 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2031461217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:11.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:11.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:11.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:11.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:11.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:11.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:19:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:19:11] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-14T04:19:11.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:11.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:11.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:11.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:11.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:11.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:12.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:12.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:12.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:12.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:12.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:12.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:12.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:12.306 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:12.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:12.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:12.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:12.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:19:12.443 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:12.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:12.477 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:12 smithi114 ceph-mon[106969]: pgmap v1134: 89 pgs: 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-09-14T04:19:12.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:12 smithi114 ceph-mon[103029]: pgmap v1134: 89 pgs: 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-09-14T04:19:12.546 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:12.572 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:12.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:12.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:12.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:12 smithi160 ceph-mon[88407]: pgmap v1134: 89 pgs: 89 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 824 op/s 2024-09-14T04:19:12.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:12.696 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:12.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:12.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:12.842 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:12.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:12.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:12.944 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:12.984 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:12.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:13.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:13.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:13.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:13.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:13.263 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:13.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:13.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:13.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:13.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:13.435 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:19:13.435 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:19:13.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:172828 10.f 10:fd430cc2:::benchmark_data_smithi114_188657_object52632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:10:35.722475+0000", 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 517.71030096200002, 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32259693299999997, 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172828 2024-09-14T04:19:13.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722475+0000", 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722475+0000", 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.438 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722475+0000", 2024-09-14T04:19:13.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:13.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.438 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.438 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722480+0000", 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8060000000000002e-06 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722491+0000", 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1316e-05 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722497+0000", 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1319999999999999e-06 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.439 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.440 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.440 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722549+0000", 2024-09-14T04:19:13.440 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2296999999999998e-05 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722685+0000", 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013533800000000001 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,3", 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.722792+0000", 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000107123 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.441 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.728797+0000", 2024-09-14T04:19:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0060054330000000001 2024-09-14T04:19:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.734105+0000", 2024-09-14T04:19:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0053078530000000004 2024-09-14T04:19:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.442 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.044991+0000", 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31088628200000001 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.045037+0000", 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5812000000000001e-05 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.443 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.045072+0000", 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5227e-05 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:172832 10.f 10:f5ac3d97:::benchmark_data_smithi114_188657_object52636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:10:35.723078+0000", 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 517.70969868999998, 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32550156499999999, 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.444 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172832 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723078+0000", 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.445 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723078+0000", 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723077+0000", 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723081+0000", 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9740000000000004e-06 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.446 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723089+0000", 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3969999999999994e-06 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723094+0000", 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2780000000000004e-06 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.725834+0000", 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0027405300000000001 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.447 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.725909+0000", 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4869000000000004e-05 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,3", 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.725974+0000", 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5067999999999994e-05 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.728905+0000", 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029313120000000002 2024-09-14T04:19:13.448 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.734226+0000", 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.005320361 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.048503+0000", 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31427680400000002 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.449 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.048544+0000", 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1359000000000002e-05 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.048579+0000", 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5317000000000001e-05 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.35997.0:172833 10.d 10:bca70c7d:::benchmark_data_smithi114_188657_object52637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-14T04:19:13.450 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:10:35.723100+0000", 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 517.70967660899998, 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.321905844, 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.35997", 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/2059992764", 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172833 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.451 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723100+0000", 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723100+0000", 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723100+0000", 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:13.452 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723101+0000", 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.64e-06 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723104+0000", 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2449999999999998e-06 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.453 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723107+0000", 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2199999999999999e-06 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723144+0000", 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7110000000000002e-05 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723217+0000", 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3182000000000005e-05 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.454 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,3", 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.723278+0000", 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1227000000000002e-05 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.726221+0000", 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029430139999999999 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:35.735307+0000", 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090862159999999994 2024-09-14T04:19:13.455 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.044872+0000", 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30956473000000001 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.044953+0000", 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0872999999999996e-05 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.456 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:10:36.045006+0000", 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2679000000000002e-05 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127224 11.4 11:21e6e516:::benchmark_data_smithi114_223345_object15287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.551373+0000", 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 283.88140360099999, 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0083590950000001, 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.457 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127224 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551373+0000", 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551373+0000", 2024-09-14T04:19:13.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551371+0000", 2024-09-14T04:19:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551379+0000", 2024-09-14T04:19:13.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4420000000000001e-06 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551393+0000", 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4610000000000001e-05 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551400+0000", 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4579999999999998e-06 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551456+0000", 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6212000000000003e-05 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551467+0000", 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.075e-05 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553349+0000", 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.001882191 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553444+0000", 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4993000000000003e-05 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553534+0000", 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0431000000000002e-05 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559606+0000", 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.006072076 2024-09-14T04:19:13.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559662+0000", 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5921999999999998e-05 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559669+0000", 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0160000000000003e-06 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559702+0000", 2024-09-14T04:19:13.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2906e-05 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559732+0000", 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9733000000000001e-05 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127241 11.14 11:2e01ae4a:::benchmark_data_smithi114_223345_object15304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.572850+0000", 2024-09-14T04:19:13.464 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 283.85992660900001, 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98064877299999997, 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127241 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572850+0000", 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572850+0000", 2024-09-14T04:19:13.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572849+0000", 2024-09-14T04:19:13.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:13.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572853+0000", 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8460000000000001e-06 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572864+0000", 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.098e-05 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572868+0000", 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4410000000000003e-06 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572922+0000", 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3471999999999999e-05 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572935+0000", 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2975e-05 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.548843+0000", 2024-09-14T04:19:13.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.975908687 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.548971+0000", 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724200000000001 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.549084+0000", 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113316 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551528+0000", 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00244378 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553317+0000", 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017892120000000001 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553369+0000", 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1722999999999999e-05 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553463+0000", 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4036000000000006e-05 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553499+0000", 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5998999999999999e-05 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.472 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127248 11.4 11:26baefff:::benchmark_data_smithi114_223345_object15311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.587658+0000", 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 283.84511865000002, 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97257697799999998, 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127248 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587658+0000", 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587658+0000", 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587656+0000", 2024-09-14T04:19:13.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587665+0000", 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3259999999999996e-06 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587683+0000", 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8300000000000001e-05 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587689+0000", 2024-09-14T04:19:13.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7420000000000003e-06 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587749+0000", 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0062000000000002e-05 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587761+0000", 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2285e-05 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555352+0000", 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96759091900000005 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555578+0000", 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000226176 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555671+0000", 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2947000000000003e-05 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559633+0000", 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039615220000000003 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559737+0000", 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104633 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560169+0000", 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043182499999999999 2024-09-14T04:19:13.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560203+0000", 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3831000000000002e-05 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560235+0000", 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1977000000000002e-05 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127262 11.14 11:28c1fb4b:::benchmark_data_smithi114_223345_object15325:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.611254+0000", 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 283.82152247900001, 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94832032799999999, 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127262 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611254+0000", 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611254+0000", 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611252+0000", 2024-09-14T04:19:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611260+0000", 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9840000000000001e-06 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611276+0000", 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5164000000000001e-05 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611282+0000", 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4060000000000002e-06 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611339+0000", 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7014e-05 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611351+0000", 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1579999999999999e-05 2024-09-14T04:19:13.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551549+0000", 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.940198321 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551673+0000", 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012361499999999999 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551780+0000", 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000107011 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555307+0000", 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003527002 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555312+0000", 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7950000000000004e-06 2024-09-14T04:19:13.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559499+0000", 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041868319999999997 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559541+0000", 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1721999999999998e-05 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559574+0000", 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3407e-05 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127264 11.4 11:260f74ee:::benchmark_data_smithi114_223345_object15327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.613838+0000", 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 283.81893889399998, 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94978247400000004, 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.487 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127264 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613838+0000", 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613838+0000", 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613836+0000", 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613843+0000", 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1550000000000004e-06 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613855+0000", 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2069000000000001e-05 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613859+0000", 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3599999999999998e-06 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613896+0000", 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7217e-05 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613904+0000", 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2099999999999996e-06 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557498+0000", 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94359476099999995 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557706+0000", 2024-09-14T04:19:13.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020764700000000001 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557796+0000", 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0552999999999999e-05 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560072+0000", 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022751780000000001 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.562269+0000", 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021972390000000001 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563542+0000", 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001273167 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563586+0000", 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4351000000000003e-05 2024-09-14T04:19:13.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563620+0000", 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3630999999999997e-05 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:46642 12.0 12:046bca62:::benchmark_data_smithi114_253692_object46641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:17:47.286839+0000", 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 86.145937275999998, 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82783152400000004, 2024-09-14T04:19:13.494 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 46642 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286839+0000", 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286839+0000", 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286839+0000", 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286894+0000", 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5449999999999999e-05 2024-09-14T04:19:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286902+0000", 2024-09-14T04:19:13.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2449999999999998e-06 2024-09-14T04:19:13.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286906+0000", 2024-09-14T04:19:13.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3850000000000002e-06 2024-09-14T04:19:13.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286955+0000", 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9156000000000002e-05 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286964+0000", 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6309999999999998e-06 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.102517+0000", 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81555286000000005 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.102653+0000", 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000135635 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.102776+0000", 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012279700000000001 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.104401+0000", 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0016249019999999999 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.105350+0000", 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00094917700000000005 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.114580+0000", 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0092306699999999998 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.114633+0000", 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2324000000000003e-05 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.114671+0000", 2024-09-14T04:19:13.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7902999999999999e-05 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:103408 12.4 12:27e05a89:::benchmark_data_smithi114_253692_object103407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:38.415035+0000", 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.017741614999998, 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.326837565, 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.502 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 103408 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.415035+0000", 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.415035+0000", 2024-09-14T04:19:13.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.415034+0000", 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.415101+0000", 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7324999999999997e-05 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.415117+0000", 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6062e-05 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.415124+0000", 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.443e-06 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.415167+0000", 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2598000000000001e-05 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.415268+0000", 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101545 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.415381+0000", 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113118 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418421+0000", 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030402430000000002 2024-09-14T04:19:13.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418976+0000", 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00055483599999999996 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.741786+0000", 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32281014699999999 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.741839+0000", 2024-09-14T04:19:13.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2512999999999997e-05 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.741872+0000", 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3482999999999998e-05 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:103410 12.1 12:85071585:::benchmark_data_smithi114_253692_object103409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:38.418335+0000", 2024-09-14T04:19:13.508 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.014441929, 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32251458700000002, 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 103410 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418335+0000", 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418335+0000", 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418334+0000", 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418416+0000", 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1959000000000006e-05 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418426+0000", 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0378e-05 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418432+0000", 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.998e-06 2024-09-14T04:19:13.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418462+0000", 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0148999999999999e-05 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418541+0000", 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8914999999999999e-05 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.418627+0000", 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6278000000000003e-05 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.422196+0000", 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0035689749999999998 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.422489+0000", 2024-09-14T04:19:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000292434 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.740749+0000", 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31826015299999999 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.740799+0000", 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0383000000000002e-05 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.740849+0000", 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9873999999999997e-05 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:103420 12.4 12:280c6fa0:::benchmark_data_smithi114_253692_object103419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:38.427259+0000", 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.005517654000002, 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31472644399999999, 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.515 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 103420 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.427259+0000", 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.427259+0000", 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.427258+0000", 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.427311+0000", 2024-09-14T04:19:13.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2707999999999997e-05 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.427320+0000", 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7860000000000008e-06 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.427325+0000", 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3970000000000004e-06 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.427358+0000", 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3161999999999999e-05 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.427437+0000", 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9123999999999997e-05 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,3", 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.427519+0000", 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2275999999999996e-05 2024-09-14T04:19:13.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.430321+0000", 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028011680000000001 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.431481+0000", 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0011602369999999999 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.741911+0000", 2024-09-14T04:19:13.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31043068400000001 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.741954+0000", 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2478000000000002e-05 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:38.741985+0000", 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1337000000000002e-05 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:119405 12.f 12:fb31b8ab:::benchmark_data_smithi114_253692_object1426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.068770+0000", 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 16.36400669, 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41861637699999998, 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119405 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.522 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068770+0000", 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068770+0000", 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068769+0000", 2024-09-14T04:19:13.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:13.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068775+0000", 2024-09-14T04:19:13.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1519999999999997e-06 2024-09-14T04:19:13.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068784+0000", 2024-09-14T04:19:13.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8950000000000004e-06 2024-09-14T04:19:13.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068789+0000", 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0499999999999999e-06 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068945+0000", 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015627999999999999 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.069042+0000", 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6781999999999996e-05 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,5", 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.069107+0000", 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5267999999999999e-05 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073284+0000", 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041767339999999997 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073309+0000", 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5202999999999999e-05 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487312+0000", 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41400287899999999 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487350+0000", 2024-09-14T04:19:13.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7935000000000002e-05 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487386+0000", 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6297999999999997e-05 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:119407 12.0 12:03c363b5:::benchmark_data_smithi114_253692_object1428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.072825+0000", 2024-09-14T04:19:13.529 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 16.359951402, 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42696216799999998, 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119407 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072825+0000", 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072825+0000", 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072824+0000", 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072831+0000", 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8689999999999998e-06 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072841+0000", 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.01e-05 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072847+0000", 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7359999999999998e-06 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073024+0000", 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017740799999999999 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073145+0000", 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000121032 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073253+0000", 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108347 2024-09-14T04:19:13.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077106+0000", 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003852725 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487199+0000", 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41009261000000002 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499662+0000", 2024-09-14T04:19:13.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012463741 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499745+0000", 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2350999999999998e-05 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499787+0000", 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2494999999999999e-05 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46463.0:119401 12.b e296/284 12:d0be7710:::benchmark_data_smithi114_253692_object1422:head v 296'7335, mlcod=296'7332)", 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.072948+0000", 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 16.35982843, 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40976830199999997, 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072948+0000", 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072948+0000", 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072948+0000", 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072952+0000", 2024-09-14T04:19:13.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5759999999999999e-06 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072959+0000", 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1909999999999997e-06 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072963+0000", 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0180000000000003e-06 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073111+0000", 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000148228 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073130+0000", 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8751000000000001e-05 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.482688+0000", 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.409558329 2024-09-14T04:19:13.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.482716+0000", 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8469000000000001e-05 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:119412 12.0 12:09c6de14:::benchmark_data_smithi114_253692_object1433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.073189+0000", 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 16.359587299000001, 2024-09-14T04:19:13.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42671745500000002, 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119412 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073189+0000", 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073189+0000", 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073188+0000", 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073193+0000", 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7269999999999998e-06 2024-09-14T04:19:13.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073202+0000", 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1390000000000001e-06 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073207+0000", 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5750000000000002e-06 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077157+0000", 2024-09-14T04:19:13.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039506919999999996 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077241+0000", 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3615000000000005e-05 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478681+0000", 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40144028700000001 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487230+0000", 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0085489810000000006 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487373+0000", 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014276600000000001 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499796+0000", 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012422812 2024-09-14T04:19:13.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499875+0000", 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8677999999999994e-05 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499907+0000", 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2048000000000002e-05 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:119413 12.5 12:a8b15bec:::benchmark_data_smithi114_253692_object1434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.073417+0000", 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 16.359359829999999, 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.410459833, 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119413 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073417+0000", 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073417+0000", 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073416+0000", 2024-09-14T04:19:13.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073420+0000", 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9700000000000001e-06 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073430+0000", 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0068000000000001e-05 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073435+0000", 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4530000000000001e-06 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073490+0000", 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4994999999999998e-05 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073602+0000", 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111681 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073691+0000", 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8565000000000003e-05 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478791+0000", 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.405100505 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483354+0000", 2024-09-14T04:19:13.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045633610000000002 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483757+0000", 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000402326 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483838+0000", 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1552999999999997e-05 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483876+0000", 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8114000000000001e-05 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46463.0:119403 12.b e296/284 12:d65c6516:::benchmark_data_smithi114_253692_object1424:head v 296'7336, mlcod=296'7332)", 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.076066+0000", 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 16.3567103, 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41100507200000003, 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076066+0000", 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076066+0000", 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076062+0000", 2024-09-14T04:19:13.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999962 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076071+0000", 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8529999999999998e-06 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076079+0000", 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0059999999999992e-06 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076083+0000", 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3699999999999997e-06 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478585+0000", 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40250170400000002 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478612+0000", 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7090999999999999e-05 2024-09-14T04:19:13.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487000+0000", 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0083881979999999995 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487071+0000", 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0820000000000003e-05 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:119416 12.1 12:806ab969:::benchmark_data_smithi114_253692_object1437:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.077231+0000", 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 16.355545338999999, 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41513983199999999, 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119416 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077231+0000", 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077231+0000", 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077230+0000", 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:13.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077236+0000", 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0020000000000003e-06 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077246+0000", 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0258000000000001e-05 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478517+0000", 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40127036599999999 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.479436+0000", 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00091930500000000004 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.479538+0000", 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010226 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.479631+0000", 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3214999999999994e-05 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.491421+0000", 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011790007999999999 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.491487+0000", 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5579000000000004e-05 2024-09-14T04:19:13.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.492268+0000", 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000780655 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.492326+0000", 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8851000000000003e-05 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.492371+0000", 2024-09-14T04:19:13.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4453000000000003e-05 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.46463.0:119413 12.5 e296/284 ondisk, result = 0)", 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.077596+0000", 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 16.355180005000001, 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40575976800000002, 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:19:13.564 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077596+0000", 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077596+0000", 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077596+0000", 2024-09-14T04:19:13.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077601+0000", 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.977e-06 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478511+0000", 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40091058499999999 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478529+0000", 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7541e-05 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483342+0000", 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048131060000000002 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483348+0000", 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3640000000000004e-06 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:13.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:13.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483356+0000", 2024-09-14T04:19:13.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9000000000000006e-06 2024-09-14T04:19:13.568 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.568 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.568 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.568 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:13.568 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:13.568 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:19:13.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:13.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:13.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:13.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:13.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:13.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:13.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:13.856 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:13.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:13.958 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:13.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:13.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:14.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:14.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:14.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:14.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:14.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:14.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:14.373 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:14 smithi160 ceph-mon[88407]: pgmap v1135: 89 pgs: 89 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-09-14T04:19:14.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:19:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:14.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:14.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:14.439 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:14 smithi114 ceph-mon[106969]: pgmap v1135: 89 pgs: 89 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-09-14T04:19:14.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:14 smithi114 ceph-mon[103029]: pgmap v1135: 89 pgs: 89 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-09-14T04:19:14.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:14.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-14T04:19:14.536 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:14.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:14.638 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:14.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:14.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:14.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:14.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:14.803 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:14.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:14.832 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:14.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:14.910 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:14.946 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:14.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:14.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:15.091 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:15.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:15.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:15.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:15.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:15.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:15.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:15.396 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:15.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:15.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:15.498 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:15.525 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:15.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:15.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:15.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:15.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:15.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:15.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:15.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:15.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:15.930 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:15.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:16.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:16.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:16.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:16.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:16.192 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:19:16.193 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "" 2024-09-14T04:19:16.193 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:19:16.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:16.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:16.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:16.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:16.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:16.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:16 smithi160 ceph-mon[88407]: pgmap v1136: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-14T04:19:16.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:16.455 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:16.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:16.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:16.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:16.612 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:16 smithi114 ceph-mon[103029]: pgmap v1136: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-14T04:19:16.612 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:16 smithi114 ceph-mon[106969]: pgmap v1136: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-14T04:19:16.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:16.733 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:16.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:16.775 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:19:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:19:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:135278 12.d 12:bd545b49:::benchmark_data_smithi114_253692_object17299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:19:16.767777+0000", 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0060784280000000003, 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061086489999999998, 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 135278 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767777+0000", 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767777+0000", 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767776+0000", 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767783+0000", 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7610000000000002e-06 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767802+0000", 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9551e-05 2024-09-14T04:19:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767809+0000", 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2389999999999996e-06 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767882+0000", 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2346000000000002e-05 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.768201+0000", 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000318838 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.768313+0000", 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011277000000000001 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:135280 12.d 12:b8168e4c:::benchmark_data_smithi114_253692_object17301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:19:16.767831+0000", 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.006025318, 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061023379999999997, 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 135280 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767831+0000", 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767831+0000", 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767830+0000", 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767833+0000", 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1889999999999999e-06 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767840+0000", 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2230000000000001e-06 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:16.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.767845+0000", 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.459e-06 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.772686+0000", 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048410579999999996 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.772794+0000", 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010840699999999999 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.772913+0000", 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011914200000000001 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46463.0:135281 12.1 e298/284 12:8076637e:::benchmark_data_smithi114_253692_object17302:head v 298'8455, mlcod=298'8452)", 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:19:16.770379+0000", 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0034769110000000001, 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003594725, 2024-09-14T04:19:16.785 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.770379+0000", 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.770379+0000", 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.770378+0000", 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.770385+0000", 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5069999999999997e-06 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.770399+0000", 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3861e-05 2024-09-14T04:19:16.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:16.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.770404+0000", 2024-09-14T04:19:16.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8680000000000004e-06 2024-09-14T04:19:16.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:16.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.771581+0000", 2024-09-14T04:19:16.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001176222 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:16.771602+0000", 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1373e-05 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 3 2024-09-14T04:19:16.789 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:19:16.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:16.867 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:16.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:16.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:17.008 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:17.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:17.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:17.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:17.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:17.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:17.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:17.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:17.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:17.359 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:19:17.359 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:19:17.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:17.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:19:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:17.420 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:17.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:17.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:17.565 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:17.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:17.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:17.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-14T04:19:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:17.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:17.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:17.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:17.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:17.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:17.958 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:17.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:17.997 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:17.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:18.098 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:18.145 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:18.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:18.149 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:18.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:18.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:18.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:18.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:18.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:18.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:18.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:18.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:18.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:18.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:18.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:18.576 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:18 smithi114 ceph-mon[106969]: pgmap v1137: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-14T04:19:18.577 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.577 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.577 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.577 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:18 smithi114 ceph-mon[103029]: pgmap v1137: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-14T04:19:18.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:18 smithi160 ceph-mon[88407]: pgmap v1137: 89 pgs: 89 active+clean; 6.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-14T04:19:18.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:18.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:18.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:18.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:18.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:18.837 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:18.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:18.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:18.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:18.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:18.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:19.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:19.098 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:19.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:19.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:19.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:19.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:19.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:19.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:19.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:19.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:19.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:19.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:19.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:19.513 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:19:19.513 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:19:19.530 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:19:19.531 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:19:19.565 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:19:19.565 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:19:19.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:19.606 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:19:19.607 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:19:19.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:19.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:19.639 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:19:19.640 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:19:19.673 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:19:19.673 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:19:19.708 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:19:19.708 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:19:19.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:19.741 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:19:19.742 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:19:19.773 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:19.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:19.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:19.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:19.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:19.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:20.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:20.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:20.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:20.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:20.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:20.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:20.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:20.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:20.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:20.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:20.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:19:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:19:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:19:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:20 smithi160 ceph-mon[88407]: pgmap v1138: 89 pgs: 89 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-09-14T04:19:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:20 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2470670525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:20.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:20.449 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:20.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:20.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:20.578 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:20.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:20 smithi114 ceph-mon[106969]: pgmap v1138: 89 pgs: 89 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-09-14T04:19:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:20 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2470670525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:20 smithi114 ceph-mon[103029]: pgmap v1138: 89 pgs: 89 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-09-14T04:19:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:20 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2470670525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:20.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:20.723 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:20.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:20.731 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:19:20.731 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:19:20.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:20.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:20.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:20.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:20.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:20.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:21.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:21.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:21.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:21.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:21.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:21.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:21.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:21.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:21.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:21.478 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-14T04:19:21.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:21.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:21.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:21.616 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:19:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:19:21] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-14T04:19:21.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:21.703 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:21.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:21.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:21.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:21.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:21.942 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:21.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:21.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:22.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:22.098 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:19:22.098 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:19:22.099 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-14T04:19:22.099 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:19:22.099 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:19:22.099 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-14T04:19:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:22.123 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:22.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:22.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:22.267 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:22.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:22.328 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:22.330 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:19:22.330 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:22 smithi160 ceph-mon[88407]: pgmap v1139: 89 pgs: 89 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T04:19:22.331 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:22 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/376558305' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:19:22.331 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:22 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:19:22.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:22.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:22.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:22.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:22.554 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:22.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:22.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:22 smithi114 ceph-mon[103029]: pgmap v1139: 89 pgs: 89 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T04:19:22.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:22 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/376558305' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:19:22.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:22 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:19:22.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:22 smithi114 ceph-mon[106969]: pgmap v1139: 89 pgs: 89 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T04:19:22.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:22 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/376558305' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:19:22.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:22 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:19:22.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:22.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:22.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:22.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:22.841 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:22.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:22.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:22.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:22.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:23.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:19:23.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:23.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:23.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:23.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:23.252 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:23.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:23.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:23.399 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:23.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:23 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:23 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:19:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:23 smithi160 ceph-mon[88407]: osdmap e299: 8 total, 8 up, 8 in 2024-09-14T04:19:23.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:23.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:23.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:23.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:23.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:23 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:23.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:23 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:19:23.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:23 smithi114 ceph-mon[106969]: osdmap e299: 8 total, 8 up, 8 in 2024-09-14T04:19:23.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:23.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:23 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:23.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:23 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:19:23.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:23 smithi114 ceph-mon[103029]: osdmap e299: 8 total, 8 up, 8 in 2024-09-14T04:19:23.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:23.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:23.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:23.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:23.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:23.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:23.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:23.858 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:23.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:23.968 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:23.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:23.985 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:23.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:24.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:24.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:24.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:24.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:24.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:24.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:24.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:24.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:24.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:24.400 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:24.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:24.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:24.549 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:24.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:24.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:24 smithi114 ceph-mon[103029]: pgmap v1141: 89 pgs: 89 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 802 op/s 2024-09-14T04:19:24.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:24 smithi114 ceph-mon[106969]: pgmap v1141: 89 pgs: 89 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 802 op/s 2024-09-14T04:19:24.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:24 smithi160 ceph-mon[88407]: pgmap v1141: 89 pgs: 89 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 802 op/s 2024-09-14T04:19:24.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:24.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:24.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:24.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:24.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:24.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:24.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:24.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:25.000 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:25.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:25.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:25.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:25.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:25.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:19:25.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:25.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:25.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:25.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:25.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:25.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:25.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:25.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:25.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:25.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:25.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:25.620 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:25.648 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:25.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:25.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:25.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:25.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:25.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:25.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:25.927 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:25.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:26.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:26.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:26.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:26.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:26.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:26.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:26.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:26.313 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:26.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:26.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:26.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:26.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:26.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:26.577 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:26.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:26.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:26 smithi114 ceph-mon[103029]: pgmap v1142: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 789 op/s 2024-09-14T04:19:26.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:26 smithi114 ceph-mon[106969]: pgmap v1142: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 789 op/s 2024-09-14T04:19:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:26 smithi160 ceph-mon[88407]: pgmap v1142: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 789 op/s 2024-09-14T04:19:26.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:26.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:26.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:26.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:26.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:26.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:26.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:26.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:26.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:27.066 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:19:27.066 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:19:27.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:27.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:27.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:27.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:27.264 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:27.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:27.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:27.400 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:27.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:27.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:27.530 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:27.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:27.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:27.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:27.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:27.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:27.792 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:27.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:27.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:27.935 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:27.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:28.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:28.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:28.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:28.165 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:28.195 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:28.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:28.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:28.334 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:28.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:28.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:19:28.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:28.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:28.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:28.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:28.592 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:28.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:28.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:28 smithi114 ceph-mon[106969]: pgmap v1143: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 789 op/s 2024-09-14T04:19:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:28 smithi114 ceph-mon[103029]: pgmap v1143: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 789 op/s 2024-09-14T04:19:28.644 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:28 smithi160 ceph-mon[88407]: pgmap v1143: 89 pgs: 89 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 789 op/s 2024-09-14T04:19:28.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:28.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:28.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:28.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:28.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:28.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:28.924 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:19:28.925 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "" 2024-09-14T04:19:28.925 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:19:28.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:29.006 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:29.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:29.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:29.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:29.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:29.248 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:29.275 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:29.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:29.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:29.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:29.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:29.512 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:19:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:19:29.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:29.548 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:29.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:29.550 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:19:29.550 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:19:29.588 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:19:29.588 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:19:29.631 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:19:29.632 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:19:29.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:29.662 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:19:29.663 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:19:29.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:29.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:29.690 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:19:29.690 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:19:29.724 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:19:29.725 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:19:29.755 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:19:29.755 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:19:29.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:29.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:29.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:29.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:29.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:29.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:30.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:30.088 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:30.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:30.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:30.108 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:30.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:30.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:30.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:30.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:30.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:30.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:30.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:19:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:19:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:19:30.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:30 smithi160 ceph-mon[88407]: pgmap v1144: 89 pgs: 89 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-14T04:19:30.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:30 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/9082529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:30.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:30.489 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:30.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:30.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:30.615 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:30 smithi114 ceph-mon[106969]: pgmap v1144: 89 pgs: 89 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-14T04:19:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:30 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/9082529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:30 smithi114 ceph-mon[103029]: pgmap v1144: 89 pgs: 89 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-14T04:19:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:30 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/9082529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:30.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:30.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:30.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:30.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:30.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:30.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:30.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:30.895 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:30.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:30.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:31.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:31.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:31.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:31.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:31.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:31.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:31.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:31.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:31.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:31.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:31.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:31.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:31.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:31.459 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:31.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:31.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:31.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:31.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:31.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:31.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:31.622 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:19:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:19:31] "GET /metrics HTTP/1.1" 200 36573 "" "Prometheus/2.51.0" 2024-09-14T04:19:31.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:31.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:31.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:31.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:31.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:31.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:31.843 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:31.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:31.945 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:31.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:31.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:31.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:32.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:32.115 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:32.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:32.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:32.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:32.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:32.244 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:32.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:32.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:32.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:32.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:32.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:32.513 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:32.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:32.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:32.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:32.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:32.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:32.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:32.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:32 smithi160 ceph-mon[88407]: pgmap v1145: 89 pgs: 89 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-14T04:19:32.736 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:32 smithi114 ceph-mon[103029]: pgmap v1145: 89 pgs: 89 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-14T04:19:32.737 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:32 smithi114 ceph-mon[106969]: pgmap v1145: 89 pgs: 89 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-14T04:19:32.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:32.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:32.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:32.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:32.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:32.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:33.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:33.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:33.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:33.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:19:33.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:33.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:33.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:33.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:33.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:33.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:33.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:33.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:33.438 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:33.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:33.540 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:33.581 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:33.584 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:33.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:33.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:33.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:33.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:33.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:33.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:33.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:33.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:33.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:33.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:33.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:34.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:34.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:34.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:34.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:34.232 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:34.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:34.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:34.367 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:34.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:34.412 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:19:34.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:34.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:34.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:34.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:34.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:34.636 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:34.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:34 smithi160 ceph-mon[88407]: pgmap v1146: 89 pgs: 89 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 737 op/s 2024-09-14T04:19:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:34 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/312572476' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:34.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:34.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:34.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:34.805 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:34 smithi114 ceph-mon[106969]: pgmap v1146: 89 pgs: 89 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 737 op/s 2024-09-14T04:19:34.805 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:34 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/312572476' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:34.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:34 smithi114 ceph-mon[103029]: pgmap v1146: 89 pgs: 89 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 737 op/s 2024-09-14T04:19:34.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:34 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/312572476' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:34.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:34.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:34.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:34.956 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:19:34.956 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:19:34.956 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-14T04:19:34.956 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:19:34.957 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:19:34.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-14T04:19:35.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:35.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:35.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:35.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:35.198 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:35.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:35.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:35.343 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:35.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:35.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:35.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:35.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:35.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:35.583 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:19:35.583 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:19:35.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:35.583 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46463.0:149629 12.0 e300/290 12:01f545c5:::benchmark_data_smithi114_253692_object31650:head v 300'9512, mlcod=300'9510)", 2024-09-14T04:19:35.583 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:19:35.565686+0000", 2024-09-14T04:19:35.583 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00096245100000000002, 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00099624299999999995, 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:35.565686+0000", 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:35.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:35.565686+0000", 2024-09-14T04:19:35.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:35.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:35.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:35.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:35.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:35.565685+0000", 2024-09-14T04:19:35.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:35.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:35.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:35.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:35.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:35.565693+0000", 2024-09-14T04:19:35.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8739999999999995e-06 2024-09-14T04:19:35.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:35.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:35.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:35.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:35.565707+0000", 2024-09-14T04:19:35.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4796e-05 2024-09-14T04:19:35.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:35.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:35.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:35.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:35.565713+0000", 2024-09-14T04:19:35.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9649999999999996e-06 2024-09-14T04:19:35.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:35.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:35.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:35.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:35.565766+0000", 2024-09-14T04:19:35.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2367999999999997e-05 2024-09-14T04:19:35.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:35.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:35.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:35.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:19:35.565786+0000", 2024-09-14T04:19:35.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0696999999999999e-05 2024-09-14T04:19:35.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:35.590 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:35.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:35.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:35.590 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:19:35.590 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 1 2024-09-14T04:19:35.590 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:19:35.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:35.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:35.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:19:35.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:19:35.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/312572476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:19:35.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[106969]: osdmap e300: 8 total, 8 up, 8 in 2024-09-14T04:19:35.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:35.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:19:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:19:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/312572476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:19:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[103029]: osdmap e300: 8 total, 8 up, 8 in 2024-09-14T04:19:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:35.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:35 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:35 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:19:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:35 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:19:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:35 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/312572476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:19:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:35 smithi160 ceph-mon[88407]: osdmap e300: 8 total, 8 up, 8 in 2024-09-14T04:19:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:35 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:35 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:35.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:35.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:35.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:35.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:35.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:35.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:35.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:36.007 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:36.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:36.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:36.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:36.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:36.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:36.265 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:36.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:36.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-14T04:19:36.352 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:36.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:36.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:36.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:36.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:36.526 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:36 smithi160 ceph-mon[88407]: pgmap v1148: 89 pgs: 89 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-09-14T04:19:36.526 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:36 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:19:36.526 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:36 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:19:36.526 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:36 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:36.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:36.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:36.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:36.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:36 smithi114 ceph-mon[106969]: pgmap v1148: 89 pgs: 89 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-09-14T04:19:36.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:36 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:19:36.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:36 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:19:36.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:36 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:36.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:36 smithi114 ceph-mon[103029]: pgmap v1148: 89 pgs: 89 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-09-14T04:19:36.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:36 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:19:36.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:36 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:19:36.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:36 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:36.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:36.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:36.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:36.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:36.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:36.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:36.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:36.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:36.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:37.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:37.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:37.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:37.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:37.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:37.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:37.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:37.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:37.335 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:37.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:37.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:37.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:37.449 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:19:37.450 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:19:37.477 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:19:37.481 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:19:37.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:37.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:37.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:19:37.517 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:19:37.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:37 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1995150444' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:37.518 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:37 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1995150444' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:19:37.556 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:19:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:19:37.585 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:19:37.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:19:37.616 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:19:37.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:37.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:37.647 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:19:37.647 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:19:37.679 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:19:37.679 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:19:37.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:37.752 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:37 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1995150444' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:37.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:37.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:37.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:37.916 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:37.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:37.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:38.018 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:38.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:38.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:38.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:38.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:38.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:38.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:38.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:38.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:38.390 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:19:38.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:38.463 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:38.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:38.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:38.590 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:38.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:38.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:38.728 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:38 smithi160 ceph-mon[88407]: pgmap v1149: 89 pgs: 89 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-09-14T04:19:38.728 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:38.729 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:38.747 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:38.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:38.839 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:38.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:38 smithi114 ceph-mon[103029]: pgmap v1149: 89 pgs: 89 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-09-14T04:19:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:38 smithi114 ceph-mon[106969]: pgmap v1149: 89 pgs: 89 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 766 op/s 2024-09-14T04:19:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:38.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:38.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:38.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:39.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:39.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:39.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:39.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:39.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:39.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:39.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:39.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:39.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:39.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:39.429 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:19:39.429 INFO:teuthology.orchestra.run.smithi160.stdout: "success": "" 2024-09-14T04:19:39.429 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:19:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:39.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:39.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:39.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:39.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:39.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:39.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:39.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:39.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:39.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:39.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:39.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:39.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:39.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:40.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:40.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:40.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:40.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:40.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:40.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:40.293 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:40.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:40.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:40.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:40.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:19:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:19:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:19:40.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:40.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:40.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:40.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:40.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:19:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:40.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:40.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:40.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:40.713 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:40.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:40.718 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:40 smithi114 ceph-mon[106969]: pgmap v1150: 89 pgs: 89 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-14T04:19:40.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:40 smithi114 ceph-mon[103029]: pgmap v1150: 89 pgs: 89 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-14T04:19:40.723 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:40 smithi160 ceph-mon[88407]: pgmap v1150: 89 pgs: 89 active+clean; 5.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 745 op/s 2024-09-14T04:19:40.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:40.865 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:40.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:40.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:41.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:41.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:41.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:41.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:41.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:41.236 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:41.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:41.273 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:19:41.273 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:19:41.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:41.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:41.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:41.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:41.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:41.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:41.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:41.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:41.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:41.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:41.656 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:41.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:41.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:41.722 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:19:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:19:41] "GET /metrics HTTP/1.1" 200 36571 "" "Prometheus/2.51.0" 2024-09-14T04:19:41.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:41.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:41.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:41.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:41.970 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:41.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:42.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-14T04:19:42.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:42.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:42.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:42.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:42.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:42.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:42.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:42.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:42.373 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:19:42.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:42.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:42.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:42.472 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:19:42.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:42.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:42.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:42.556 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:42 smithi114 ceph-mon[106969]: pgmap v1151: 89 pgs: 89 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 744 op/s 2024-09-14T04:19:42.557 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/514352281' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:19:42.557 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:42 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:19:42.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:42 smithi114 ceph-mon[103029]: pgmap v1151: 89 pgs: 89 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 744 op/s 2024-09-14T04:19:42.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/514352281' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:19:42.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:42 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:19:42.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:42.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:42.696 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:42 smithi160 ceph-mon[88407]: pgmap v1151: 89 pgs: 89 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 744 op/s 2024-09-14T04:19:42.697 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/514352281' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:19:42.697 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:42 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:19:42.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:42.814 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:42.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:42.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:42.977 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:42.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:43.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:43.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:43.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:43.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:43.234 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:43.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:43.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:43.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:43.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:43.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:43.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:43.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:43.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:43.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:43.614 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:43 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:43.614 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:43 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:19:43.614 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:43 smithi114 ceph-mon[106969]: osdmap e301: 8 total, 8 up, 8 in 2024-09-14T04:19:43.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:43 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:43.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:43 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:19:43.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:43 smithi114 ceph-mon[103029]: osdmap e301: 8 total, 8 up, 8 in 2024-09-14T04:19:43.634 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:43.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:43.721 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:43 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:43.721 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:43 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:19:43.721 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:43 smithi160 ceph-mon[88407]: osdmap e301: 8 total, 8 up, 8 in 2024-09-14T04:19:43.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:43.767 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:43.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:43.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:43.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:43.916 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:43.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:44.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:44.041 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:19:44.041 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:19:44.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:44.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:44.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:44.078 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:19:44.078 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:19:44.114 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:19:44.114 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:19:44.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:44.153 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:19:44.154 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:19:44.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:44.183 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:19:44.184 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:19:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:44.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:44.214 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:19:44.214 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:19:44.243 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:19:44.243 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:19:44.272 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:19:44.272 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:19:44.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:44.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:44.320 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:44.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:44.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:44.460 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:44.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:44.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:44.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:44.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:44.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:44.698 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:44 smithi160 ceph-mon[88407]: pgmap v1153: 89 pgs: 89 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 579 op/s 2024-09-14T04:19:44.698 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:44 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/32329480' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:44.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:44.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:44.728 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:19:44.728 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:19:44.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:19:44.728 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80126 11.e 11.68f758ce (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.163263+0000", 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 368.56211807, 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98678501200000002, 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80126 2024-09-14T04:19:44.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163263+0000", 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163263+0000", 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163262+0000", 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163438+0000", 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017587599999999999 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163450+0000", 2024-09-14T04:19:44.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2442e-05 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163457+0000", 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0249999999999997e-06 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163549+0000", 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1340000000000003e-05 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163556+0000", 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.452e-06 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150039+0000", 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98648345100000001 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150048+0000", 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8729999999999996e-06 2024-09-14T04:19:44.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.734 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.734 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.735 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80128 11.e 11.ea33aae (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:19:44.735 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.163487+0000", 2024-09-14T04:19:44.735 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 368.56189478800002, 2024-09-14T04:19:44.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98659235700000003, 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80128 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163487+0000", 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163487+0000", 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163486+0000", 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163557+0000", 2024-09-14T04:19:44.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1472999999999998e-05 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163566+0000", 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4169999999999992e-06 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163571+0000", 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2830000000000001e-06 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163625+0000", 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3544000000000001e-05 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163630+0000", 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6699999999999999e-06 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150071+0000", 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98644103500000002 2024-09-14T04:19:44.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150079+0000", 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7090000000000002e-06 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80131 11.e 11.74c6056e (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.163582+0000", 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 368.56179944199999, 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98653437300000002, 2024-09-14T04:19:44.740 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80131 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163582+0000", 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163582+0000", 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163581+0000", 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163648+0000", 2024-09-14T04:19:44.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6358000000000004e-05 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163656+0000", 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9179999999999994e-06 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163660+0000", 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4800000000000003e-06 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163704+0000", 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3847000000000002e-05 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163710+0000", 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8030000000000002e-06 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150108+0000", 2024-09-14T04:19:44.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98639813399999998 2024-09-14T04:19:44.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150116+0000", 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3299999999999999e-06 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80140 11.e 11.37f8f16e (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.163719+0000", 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 368.561662426, 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98642381300000004, 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:19:44.746 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80140 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163719+0000", 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163719+0000", 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163718+0000", 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163776+0000", 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7970999999999998e-05 2024-09-14T04:19:44.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163784+0000", 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9789999999999993e-06 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163788+0000", 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3760000000000001e-06 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163832+0000", 2024-09-14T04:19:44.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3648e-05 2024-09-14T04:19:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163837+0000", 2024-09-14T04:19:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1900000000000003e-06 2024-09-14T04:19:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150136+0000", 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98629912600000003 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150143+0000", 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3500000000000002e-06 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80152 11.e 11.769ce69e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:19:44.751 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.163803+0000", 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 368.56157799800002, 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98636712900000001, 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80152 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163803+0000", 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163803+0000", 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163803+0000", 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163865+0000", 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2187999999999995e-05 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163873+0000", 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4950000000000002e-06 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163877+0000", 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8040000000000001e-06 2024-09-14T04:19:44.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163897+0000", 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9806999999999999e-05 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.163900+0000", 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.424e-06 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150165+0000", 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98626506199999997 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150171+0000", 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.823e-06 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80168 11.e 11.ed9c115e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:19:44.756 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.189840+0000", 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 368.53554111800003, 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96035531900000004, 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80168 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.189840+0000", 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.189840+0000", 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.189839+0000", 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.189903+0000", 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4314999999999997e-05 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.189921+0000", 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7628000000000001e-05 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.189925+0000", 2024-09-14T04:19:44.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8579999999999999e-06 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.189959+0000", 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3589e-05 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.189965+0000", 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.553e-06 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150189+0000", 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96022467600000005 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150196+0000", 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3430000000000001e-06 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.761 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80179 11.e 11.a944b67e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.201895+0000", 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 368.52348643099998, 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94832534099999999, 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80179 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.201895+0000", 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.201895+0000", 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.201893+0000", 2024-09-14T04:19:44.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.201951+0000", 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7583000000000001e-05 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.201969+0000", 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8306999999999999e-05 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.201977+0000", 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.089e-06 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.202033+0000", 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5723999999999997e-05 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.202041+0000", 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7500000000000003e-06 2024-09-14T04:19:44.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150215+0000", 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94817393400000005 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150220+0000", 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7780000000000004e-06 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80189 11.e 11.adc7717e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.213297+0000", 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 368.51208424599997, 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93694906099999997, 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80189 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.213297+0000", 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.213297+0000", 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.213296+0000", 2024-09-14T04:19:44.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.213348+0000", 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2370999999999997e-05 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.213363+0000", 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5554999999999999e-05 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.213369+0000", 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7780000000000004e-06 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.213407+0000", 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7370999999999998e-05 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.213413+0000", 2024-09-14T04:19:44.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3069999999999999e-06 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150239+0000", 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93682646999999997 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.150246+0000", 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7959999999999998e-06 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:117974 11.e e260/252 11:74842812:::benchmark_data_smithi114_223345_object6037:head v 260'7457, mlcod=0'0)", 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:19.932207+0000", 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 324.79317461900001, 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.063806346999999999, 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:19.932207+0000", 2024-09-14T04:19:44.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:19.932207+0000", 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:19.932206+0000", 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:19.932215+0000", 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8319999999999995e-06 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:19.932242+0000", 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7614e-05 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:19.932250+0000", 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7729999999999999e-06 2024-09-14T04:19:44.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:19.937977+0000", 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0057268450000000004 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:19.937998+0000", 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0613000000000001e-05 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:19.995973+0000", 2024-09-14T04:19:44.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057975108999999997 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:19.996013+0000", 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0487e-05 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:129182 11.e e262/252 11:75061052:::benchmark_data_smithi114_223345_object17245:head v 262'8197, mlcod=0'0)", 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:32.634495+0000", 2024-09-14T04:19:44.776 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 312.09088685199998, 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.067261294999999999, 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:32.634495+0000", 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:32.634495+0000", 2024-09-14T04:19:44.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:32.634494+0000", 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:32.634501+0000", 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4660000000000002e-06 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:32.634516+0000", 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4572999999999999e-05 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:32.634521+0000", 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7409999999999998e-06 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:32.658359+0000", 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.023837145000000001 2024-09-14T04:19:44.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:32.658382+0000", 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3807000000000001e-05 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:32.701722+0000", 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.043339919999999997 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:32.701756+0000", 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3581000000000003e-05 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:162017 11.d e263/252 11:bf97f932:::benchmark_data_smithi114_223345_object50080:head v 263'10173, mlcod=240'3185)", 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:07.204045+0000", 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 277.52133601899999, 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060777609000000003, 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.781 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:07.204045+0000", 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:07.204045+0000", 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:07.204045+0000", 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:07.204050+0000", 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4750000000000001e-06 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:07.204063+0000", 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2218000000000001e-05 2024-09-14T04:19:44.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:07.204067+0000", 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6410000000000001e-06 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:07.212802+0000", 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.008735184 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:07.212860+0000", 2024-09-14T04:19:44.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7318000000000002e-05 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:07.264770+0000", 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.051910592999999998 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:07.264823+0000", 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2655999999999999e-05 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.785 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:167506 11.c e264/252 11:30469c72:::benchmark_data_smithi114_223345_object55569:head v 264'10299, mlcod=0'0)", 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:13.383248+0000", 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 271.342133322, 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061231286000000003, 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:13.383248+0000", 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:13.383248+0000", 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:13.383247+0000", 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:13.383256+0000", 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8410000000000006e-06 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:13.383273+0000", 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7680999999999999e-05 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:13.383280+0000", 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6239999999999996e-06 2024-09-14T04:19:44.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:13.384791+0000", 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015105299999999999 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:13.384834+0000", 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2889000000000002e-05 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:13.444391+0000", 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.059557854 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:13.444479+0000", 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7980999999999997e-05 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:183192 11.d e267/252 11:bf7b800b:::benchmark_data_smithi114_223345_object71255:head v 267'11457, mlcod=250'4470)", 2024-09-14T04:19:44.790 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:31.474272+0000", 2024-09-14T04:19:44.791 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 253.25110936300001, 2024-09-14T04:19:44.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.071018998, 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:31.474272+0000", 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:31.474272+0000", 2024-09-14T04:19:44.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:31.474271+0000", 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:31.474281+0000", 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0159999999999999e-05 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:31.474296+0000", 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5284000000000001e-05 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:31.474303+0000", 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4080000000000003e-06 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:31.485705+0000", 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0114019 2024-09-14T04:19:44.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:31.485736+0000", 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1443999999999999e-05 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:31.545132+0000", 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.059396374000000002 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:31.545291+0000", 2024-09-14T04:19:44.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000158636 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:184607 11.d e267/252 11:ba6aa869:::benchmark_data_smithi114_223345_object72670:head v 267'11562, mlcod=250'4573)", 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:33.174942+0000", 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 251.55043889699999, 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060412822999999997, 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:33.174942+0000", 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:33.174942+0000", 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:33.174941+0000", 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:33.174949+0000", 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3140000000000004e-06 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:33.174967+0000", 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7917000000000001e-05 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:33.174974+0000", 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1590000000000004e-06 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:33.184660+0000", 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0096857179999999994 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:33.184673+0000", 2024-09-14T04:19:44.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.331e-05 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:33.235307+0000", 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.050633800999999999 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:33.235355+0000", 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8165999999999999e-05 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:185601 11.e e267/252 11:7c0ff55f:::benchmark_data_smithi114_223345_object73664:head v 267'11711, mlcod=0'0)", 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:34.416280+0000", 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 250.30910138199999, 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.069496888000000007, 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.416280+0000", 2024-09-14T04:19:44.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.416280+0000", 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.416280+0000", 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.416283+0000", 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5159999999999999e-06 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.416291+0000", 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7340000000000008e-06 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.416294+0000", 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1690000000000001e-06 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.430430+0000", 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014136085 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.430452+0000", 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.126e-05 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.485740+0000", 2024-09-14T04:19:44.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.055288110000000001 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.485777+0000", 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7262999999999999e-05 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:185605 11.e e267/252 11:7204c674:::benchmark_data_smithi114_223345_object73668:head v 267'11712, mlcod=0'0)", 2024-09-14T04:19:44.805 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:34.420091+0000", 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 250.30529077899999, 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.065705973000000001, 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.420091+0000", 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.420091+0000", 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.420090+0000", 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.420095+0000", 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4419999999999997e-06 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.420110+0000", 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4982e-05 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.420116+0000", 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8980000000000001e-06 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.437654+0000", 2024-09-14T04:19:44.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017538193000000001 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.437675+0000", 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0933000000000001e-05 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.485780+0000", 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.048105201 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:34.485797+0000", 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6197000000000001e-05 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:191004 11.d e269/252 11:b83d3032:::benchmark_data_smithi114_223345_object79067:head v 269'11944, mlcod=250'4956)", 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:40.363138+0000", 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 244.36224353700001, 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.067106458999999993, 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.810 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:40.363138+0000", 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:40.363138+0000", 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:40.363137+0000", 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:40.363147+0000", 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0798999999999999e-05 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:40.363163+0000", 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5885000000000001e-05 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:40.363171+0000", 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7309999999999993e-06 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:40.363236+0000", 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5038999999999999e-05 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:40.363277+0000", 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1482e-05 2024-09-14T04:19:44.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:40.430113+0000", 2024-09-14T04:19:44.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.066835633000000005 2024-09-14T04:19:44.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:40.430244+0000", 2024-09-14T04:19:44.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000131204 2024-09-14T04:19:44.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:192184 11.e e269/252 11:738b08d2:::benchmark_data_smithi114_223345_object80247:head v 269'12130, mlcod=0'0)", 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:15:41.659321+0000", 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 243.066060233, 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061693645999999998, 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:41.659321+0000", 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:41.659321+0000", 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:41.659321+0000", 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:19:44.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:41.659327+0000", 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4579999999999998e-06 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:41.659336+0000", 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8389999999999996e-06 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:41.659343+0000", 2024-09-14T04:19:44.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2609999999999996e-06 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:41.665650+0000", 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0063065359999999997 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:41.665672+0000", 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2229999999999999e-05 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:41.720977+0000", 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.055304741999999997 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:15:41.721015+0000", 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8177999999999999e-05 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:207911 11.c e271/252 11:335f0686:::benchmark_data_smithi114_223345_object95974:head v 271'12815, mlcod=0'0)", 2024-09-14T04:19:44.819 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:16:00.052082+0000", 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 224.67329983100001, 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061937303999999999, 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:00.052082+0000", 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:00.052082+0000", 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:00.052081+0000", 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:00.052088+0000", 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1860000000000003e-06 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:00.052104+0000", 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5903999999999999e-05 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:00.052110+0000", 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6100000000000002e-06 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:00.069605+0000", 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017494315 2024-09-14T04:19:44.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:00.069645+0000", 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0074999999999998e-05 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:00.113878+0000", 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.044233424 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:00.114019+0000", 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014078699999999999 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.36273.0:210522 11.e e272/252 11:7bf0a319:::benchmark_data_smithi114_223345_object98585:head v 272'13338, mlcod=0'0)", 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:16:03.339646+0000", 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 221.385735345, 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.066456324999999997, 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:19:44.824 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:03.339646+0000", 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:03.339646+0000", 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:19:44.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:03.339645+0000", 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:03.339653+0000", 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.644e-06 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:03.339669+0000", 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5967000000000001e-05 2024-09-14T04:19:44.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:03.339675+0000", 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7089999999999996e-06 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:03.339736+0000", 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0448999999999997e-05 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:19:44.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:03.339763+0000", 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7129999999999999e-05 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:03.406001+0000", 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.066237914999999994 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:16:03.406102+0000", 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010149000000000001 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.828 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:19:44.829 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:19:44.829 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:19:44.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:44.868 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:44.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:44 smithi114 ceph-mon[106969]: pgmap v1153: 89 pgs: 89 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 579 op/s 2024-09-14T04:19:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:44 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/32329480' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:44.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:44 smithi114 ceph-mon[103029]: pgmap v1153: 89 pgs: 89 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 579 op/s 2024-09-14T04:19:44.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:44 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/32329480' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:19:44.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:45.006 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:45.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:45.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:45.151 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:45.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:45.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:45.299 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:45.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:45.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:19:45.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:45.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:45.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:45.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:45.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:45.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:45.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:45.690 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:45.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:45.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:45.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-14T04:19:45.760 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:19:45.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-14T04:19:45.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:45.929 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:45.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:46.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:46.055 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:46.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:46.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:46.235 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:46.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:46.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:46.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:46.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:46.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:46.516 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:46.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:46.618 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:46.645 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:46.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:46.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:46.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:46.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:46.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:46.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:46 smithi114 ceph-mon[106969]: pgmap v1154: 89 pgs: 89 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-09-14T04:19:46.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:46 smithi114 ceph-mon[103029]: pgmap v1154: 89 pgs: 89 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-09-14T04:19:46.875 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:46 smithi160 ceph-mon[88407]: pgmap v1154: 89 pgs: 89 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-09-14T04:19:46.912 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:46.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:47.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:47.051 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:47.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:47.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:47.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:19:47.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:47.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:47.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:47.360 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:47.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:47.409 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:47.417 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:47.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:47.489 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:47.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:47.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:47.645 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:47.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:47.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:47.775 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:47.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:47.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:47.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:47.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:48.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:48.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:48.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:48.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:48.171 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:48.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:48.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:48.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:48.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:48.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:48.458 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:48.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:48.559 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:48.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:48.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:48.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:48.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:48.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:48.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:48.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:48.842 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:48 smithi114 ceph-mon[103029]: pgmap v1155: 89 pgs: 89 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-09-14T04:19:48.842 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:48 smithi114 ceph-mon[106969]: pgmap v1155: 89 pgs: 89 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-09-14T04:19:48.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:48.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:48 smithi160 ceph-mon[88407]: pgmap v1155: 89 pgs: 89 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-09-14T04:19:48.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:48.975 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:48.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:49.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:49.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:49.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:49.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:49.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:49.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:49.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:49.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:49.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:49.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:49.516 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:49.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:49.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:49.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:49.646 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:49.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:49.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:49.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:49.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:49.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:49.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:49.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:49.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:50.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:50.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:50.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:50.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:50.155 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:50.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:50.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:50.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:50.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:50.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:50.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:50.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:50.480 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:19:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:19:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:19:50.502 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:19:50.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:50.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:50.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:50.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:50.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:50.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:50.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:50.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:50.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:50.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:50.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:50 smithi114 ceph-mon[106969]: pgmap v1156: 89 pgs: 89 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 774 op/s 2024-09-14T04:19:50.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:50 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/823215513' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:50.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:50 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:50.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:50 smithi114 ceph-mon[103029]: pgmap v1156: 89 pgs: 89 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 774 op/s 2024-09-14T04:19:50.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:50 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/823215513' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:50.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:50 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:50 smithi160 ceph-mon[88407]: pgmap v1156: 89 pgs: 89 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 774 op/s 2024-09-14T04:19:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:50 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/823215513' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:50 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:19:50.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:50.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:50.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:51.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:51.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:51.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:51.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:51.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:51.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:51.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:51.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:51.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:51.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:51.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:51.484 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:51.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:51.518 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-14T04:19:51.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:51.612 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:51.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:51.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:51.738 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:51.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:51.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:51.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:51.861 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:51.861 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:19:51.861 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-mon[106969]: osdmap e302: 8 total, 8 up, 8 in 2024-09-14T04:19:51.861 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3139901729' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:19:51.861 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:19:51.862 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:19:51] "GET /metrics HTTP/1.1" 200 36571 "" "Prometheus/2.51.0" 2024-09-14T04:19:51.862 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:51.862 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:19:51.862 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-mon[103029]: osdmap e302: 8 total, 8 up, 8 in 2024-09-14T04:19:51.862 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3139901729' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:19:51.862 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:51 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:19:51.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:51.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:51.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:51 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:19:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:51 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:19:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:51 smithi160 ceph-mon[88407]: osdmap e302: 8 total, 8 up, 8 in 2024-09-14T04:19:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:51 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3139901729' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:19:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:51 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:19:51.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:51.994 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:51.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:52.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:52.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:52.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:52.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:52.156 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:52.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:52.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:52.286 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:52.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:52.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:52.427 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:52.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:52.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:52.515 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:52 smithi114 ceph-mon[103029]: pgmap v1158: 89 pgs: 89 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 835 op/s 2024-09-14T04:19:52.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:52.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:52.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:52.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-14T04:19:52.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:52.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:52.708 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:52.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:52.785 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:52 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:19:52.785 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:52 smithi114 ceph-mon[103029]: osdmap e303: 8 total, 8 up, 8 in 2024-09-14T04:19:52.786 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:52 smithi114 ceph-mon[106969]: pgmap v1158: 89 pgs: 89 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 835 op/s 2024-09-14T04:19:52.786 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:52 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:19:52.786 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:52 smithi114 ceph-mon[106969]: osdmap e303: 8 total, 8 up, 8 in 2024-09-14T04:19:52.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:52.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:52.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:52.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:52 smithi160 ceph-mon[88407]: pgmap v1158: 89 pgs: 89 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 835 op/s 2024-09-14T04:19:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:52 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:19:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:52 smithi160 ceph-mon[88407]: osdmap e303: 8 total, 8 up, 8 in 2024-09-14T04:19:52.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:52.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:52.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:52.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:53.091 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:53.116 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:53.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:53.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:53.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:53.244 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:53.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:53.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:53.347 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:19:53.347 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:19:53.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.347 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:163289 12.3 12:c798ec7e:::benchmark_data_smithi114_253692_object45310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-14T04:19:53.347 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:19:53.331909+0000", 2024-09-14T04:19:53.347 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.012893824999999999, 2024-09-14T04:19:53.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012927723, 2024-09-14T04:19:53.347 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 163289 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:19:53.348 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.331909+0000", 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.331909+0000", 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.331911+0000", 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5260000000000001e-06 2024-09-14T04:19:53.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.331915+0000", 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.763e-06 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.331926+0000", 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0845999999999999e-05 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:19:53.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.331938+0000", 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2279999999999999e-05 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.333651+0000", 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017125199999999999 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.333767+0000", 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115683 2024-09-14T04:19:53.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.333884+0000", 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011760399999999999 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.338695+0000", 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004811063 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.342410+0000", 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037146599999999998 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:163290 12.c 12:343531be:::benchmark_data_smithi114_253692_object45311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:19:53.337406+0000", 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0073971080000000003, 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0075057370000000002, 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:19:53.353 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 163290 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.337406+0000", 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:19:53.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.355 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:19:53.355 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.337406+0000", 2024-09-14T04:19:53.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:19:53.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.355 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:19:53.355 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.337408+0000", 2024-09-14T04:19:53.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9870000000000002e-06 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.337414+0000", 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6830000000000002e-06 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.337435+0000", 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0800000000000001e-05 2024-09-14T04:19:53.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.337445+0000", 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4050000000000006e-06 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.337624+0000", 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017923599999999999 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.337973+0000", 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00034909999999999997 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:53.338386+0000", 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00041355999999999999 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:19:53.358 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 2 2024-09-14T04:19:53.359 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:19:53.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:53.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:53.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:53.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:53.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:53.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:53.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:53.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:53.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:53.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:53.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:53.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:53.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:53.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:53.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:53.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:54.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:54.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:54.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:54.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:54.122 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:53 smithi114 ceph-mon[106969]: pgmap v1160: 89 pgs: 89 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 530 op/s 2024-09-14T04:19:54.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:54.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:54.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:53 smithi114 ceph-mon[103029]: pgmap v1160: 89 pgs: 89 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 530 op/s 2024-09-14T04:19:54.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:54.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:19:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:19:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:53 smithi160 ceph-mon[88407]: pgmap v1160: 89 pgs: 89 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 530 op/s 2024-09-14T04:19:54.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:54.276 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:54.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:54.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:54.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:54.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:54.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:54.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:54.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:54.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:54.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:54.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:54.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:54.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:54.822 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:54.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:54.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:54.968 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:54.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:55.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:55.112 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:55.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:55.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:55.236 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:55.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:55.338 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:55.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:55.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:55.467 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:55.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:55.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:55.596 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:55.625 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:55.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:55.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:55.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:55.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:55.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:55.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-14T04:19:55.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:55.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:55.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:55.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:56.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:56.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:56.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:56.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:56.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:56.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:56.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:56.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:56.364 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:56 smithi114 ceph-mon[103029]: pgmap v1161: 89 pgs: 89 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-09-14T04:19:56.365 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:56 smithi114 ceph-mon[106969]: pgmap v1161: 89 pgs: 89 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-09-14T04:19:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:56 smithi160 ceph-mon[88407]: pgmap v1161: 89 pgs: 89 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 947 op/s 2024-09-14T04:19:56.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:56.431 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:56.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:56.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:56.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:56.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:56.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:56.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:56.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:56.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:56.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:56.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:56.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:56.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:56.959 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:56.994 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:56.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:57.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:19:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:57.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:57.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:19:57.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:57.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:57.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:57.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:57.412 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:57.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:57.428 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:19:57.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:19:57.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:57.565 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:57.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:57.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:57.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:57.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:57.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:57.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:57.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:57.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:57.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:57.980 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:57.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:57.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:58.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:58.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:58.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:58.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:58.234 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:58.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:58.293 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:19:58 smithi114 ceph-mon[106969]: pgmap v1162: 89 pgs: 89 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 648 op/s 2024-09-14T04:19:58.293 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:19:58 smithi114 ceph-mon[103029]: pgmap v1162: 89 pgs: 89 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 648 op/s 2024-09-14T04:19:58.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:58.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:58.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:19:58 smithi160 ceph-mon[88407]: pgmap v1162: 89 pgs: 89 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 648 op/s 2024-09-14T04:19:58.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:58.499 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:58.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:58.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:58.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:58.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:58.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:19:58.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:58.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:19:58.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:19:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:58.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:58.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:19:58.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:19:59.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:59.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:59.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:19:59.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:59.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:59.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:59.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:59.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:59.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:59.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:19:59.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:59.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:59.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:19:59.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:19:59.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:19:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:59.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:19:59.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:19:59.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:59.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:19:59.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:59.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:59.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:19:59.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:19:59.968 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:19:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:19:59.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:00.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:00.095 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:00.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:00.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:00.107 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-14T04:20:00.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:00.226 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:00.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:00.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:00.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:00 smithi114 ceph-mon[103029]: pgmap v1163: 89 pgs: 89 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T04:20:00.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/460308609' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:20:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:00 smithi114 ceph-mon[103029]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-14T04:20:00.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:00 smithi114 ceph-mon[103029]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-14T04:20:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:00 smithi114 ceph-mon[106969]: pgmap v1163: 89 pgs: 89 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T04:20:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/460308609' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:20:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:00 smithi114 ceph-mon[106969]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-14T04:20:00.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:00 smithi114 ceph-mon[106969]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-14T04:20:00.372 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:00.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:20:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:20:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:20:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:00 smithi160 ceph-mon[88407]: pgmap v1163: 89 pgs: 89 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T04:20:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/460308609' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:20:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:00 smithi160 ceph-mon[88407]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-14T04:20:00.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:00 smithi160 ceph-mon[88407]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-14T04:20:00.474 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:00.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:00.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:00.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:00.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:00.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:00.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:00.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:00.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:00.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:00.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:00.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:01.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:01.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:01.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:01.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:01.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:01.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:01.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:01.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:01.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:01.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:01.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:01 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/460308609' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:20:01.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:01 smithi160 ceph-mon[88407]: osdmap e304: 8 total, 8 up, 8 in 2024-09-14T04:20:01.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:01.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:01 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/460308609' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:20:01.433 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:01 smithi114 ceph-mon[106969]: osdmap e304: 8 total, 8 up, 8 in 2024-09-14T04:20:01.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:01 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/460308609' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:20:01.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:01 smithi114 ceph-mon[103029]: osdmap e304: 8 total, 8 up, 8 in 2024-09-14T04:20:01.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:01.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:01.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:01.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:01.582 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:01.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:01.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:01.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:01.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:01.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:01.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:01.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:01.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:01.872 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:20:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:20:01] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-14T04:20:01.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:01.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:01.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:02.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:02.131 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:02.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:02.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:02.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:02.276 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:02.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:02.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:02 smithi114 ceph-mon[106969]: pgmap v1165: 89 pgs: 89 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-14T04:20:02.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:02 smithi114 ceph-mon[103029]: pgmap v1165: 89 pgs: 89 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-14T04:20:02.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:02 smithi160 ceph-mon[88407]: pgmap v1165: 89 pgs: 89 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-14T04:20:02.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:02.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:02.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:02.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:02.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:02.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:02.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:02.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:02.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:02.806 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:02.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:02.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:02.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:02.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:03.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:03.067 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:03.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:03.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:03.169 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:03.196 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:03.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:03.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:03.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:03.339 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:03.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:03.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:03.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:03.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:03.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:03.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:03.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:03.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:03.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:03.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:03.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:03.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:03.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:03.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:04.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:04.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:04.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:04.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:04.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:04.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:04.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:04.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:04.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:04.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:04.380 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:04 smithi114 ceph-mon[106969]: pgmap v1166: 89 pgs: 89 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 773 op/s 2024-09-14T04:20:04.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:04 smithi114 ceph-mon[103029]: pgmap v1166: 89 pgs: 89 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 773 op/s 2024-09-14T04:20:04.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:04.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:04.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:04 smithi160 ceph-mon[88407]: pgmap v1166: 89 pgs: 89 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 773 op/s 2024-09-14T04:20:04.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:04.545 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:04.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:04.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:04.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:04.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:04.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:04.803 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:04.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:04.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:04.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:04.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:04.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:04.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:04.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:05.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:05.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:05.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:05.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:05.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:05.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:05.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:05.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:05.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:05.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:05.382 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:20:05.382 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:20:05.383 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:20:05.383 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:20:05.383 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:20:05.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:05.506 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:05.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:05.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:05.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:05.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:05.708 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:20:05.709 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:20:05.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:05.757 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:20:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:20:05.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:05.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:05.801 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:20:05.801 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:20:05.834 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:20:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:20:05.864 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:20:05.864 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:20:05.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:05.896 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:20:05.897 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:20:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:05.907 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:05.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:05.924 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:20:05.925 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:20:05.956 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:20:05.956 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:20:06.008 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:06.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:06.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:06.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:06.169 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:06.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:06.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:06.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:06.295 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:06.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:06.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:06 smithi114 ceph-mon[106969]: pgmap v1167: 89 pgs: 89 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 771 op/s 2024-09-14T04:20:06.373 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/916631861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:06.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:06 smithi114 ceph-mon[103029]: pgmap v1167: 89 pgs: 89 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 771 op/s 2024-09-14T04:20:06.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/916631861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:06.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:06.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:06 smithi160 ceph-mon[88407]: pgmap v1167: 89 pgs: 89 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 771 op/s 2024-09-14T04:20:06.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/916631861' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:06.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:06.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:06.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:06.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:06.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:06.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:06.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:06.686 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:06.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:06.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:06.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:06.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:06.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:06.950 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:06.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:07.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:07.079 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:07.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:07.181 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:07.218 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:07.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:07.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:07.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:07.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:07.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:07.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:07.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:07.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-14T04:20:07.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:07.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:07.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:07.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:07.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:07.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:07.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:07.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:07.954 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:07.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:07.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-14T04:20:08.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:08.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:08.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:08.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:08.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:08.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:08.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:08 smithi114 ceph-mon[103029]: pgmap v1168: 89 pgs: 89 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 771 op/s 2024-09-14T04:20:08.348 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:08.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:08 smithi114 ceph-mon[106969]: pgmap v1168: 89 pgs: 89 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 771 op/s 2024-09-14T04:20:08.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:08.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:08.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:08 smithi160 ceph-mon[88407]: pgmap v1168: 89 pgs: 89 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 771 op/s 2024-09-14T04:20:08.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:08.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:08.504 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:08.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:08.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:08.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:08.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:08.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:08.749 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:08.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:08.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:08.777 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:08.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:08.896 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:08.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:08.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:09.044 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:09.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:09.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:09.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:09.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:09.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:09.344 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:09.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:09.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:09.488 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:09.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:09.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:09.597 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:09.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:09.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:09.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:09.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:09.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:09.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:09.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:09.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:09.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:10.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:10.057 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:10.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:10.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:10.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:10.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:10.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:10.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:10.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:10.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:10.439 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:10.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:10.507 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:20:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:20:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:20:10.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:10.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:10.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:10.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:10.719 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:10.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:10.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:20:10.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:10 smithi114 ceph-mon[103029]: pgmap v1169: 89 pgs: 89 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-09-14T04:20:10.773 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:10 smithi114 ceph-mon[106969]: pgmap v1169: 89 pgs: 89 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-09-14T04:20:10.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:10.872 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:10.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:10.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:10 smithi160 ceph-mon[88407]: pgmap v1169: 89 pgs: 89 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-09-14T04:20:10.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:11.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:11.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:11.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:11.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:11.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:11.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:11.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:11.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:11.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:11.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:11.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:11.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:11.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:11.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:11.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:11.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:11.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:11.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:11.738 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:20:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:20:11] "GET /metrics HTTP/1.1" 200 36572 "" "Prometheus/2.51.0" 2024-09-14T04:20:11.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:11.790 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:11.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:11.892 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:11.917 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:11.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:12.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:12.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:12.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:12.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:12.201 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:12.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:12.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:12.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:12.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:12.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:12.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:12.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:12.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:12.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:12.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:12.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:12 smithi114 ceph-mon[106969]: pgmap v1170: 89 pgs: 89 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-09-14T04:20:12.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:12 smithi114 ceph-mon[103029]: pgmap v1170: 89 pgs: 89 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-09-14T04:20:12.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:12.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:12.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:12.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:12.894 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:12.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:12.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:12 smithi160 ceph-mon[88407]: pgmap v1170: 89 pgs: 89 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 663 op/s 2024-09-14T04:20:12.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:13.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:13.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:13.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:13.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:13.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:13.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:13.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:13.306 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:13.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:13.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:13.432 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:13.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:13.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:13.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:13.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:13.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:13.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:13.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:13.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:13.699 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:20:13.699 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:20:13.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:13.825 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:13.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:13.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:13.960 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:13.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:14.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:14.098 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:14.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:14.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:14.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:14.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:14.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:14.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:14.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:14.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:14.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:14.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:14.537 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:20:14.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:14.642 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:14.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:14.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:14.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:14.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:14.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:14.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:14.782 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:14 smithi114 ceph-mon[106969]: pgmap v1171: 89 pgs: 89 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 640 op/s 2024-09-14T04:20:14.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:14 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4243227320' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:20:14.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:14 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:20:14.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:14 smithi114 ceph-mon[103029]: pgmap v1171: 89 pgs: 89 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 640 op/s 2024-09-14T04:20:14.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:14 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4243227320' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:20:14.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:14 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:20:14.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:14.900 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:14.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:14 smithi160 ceph-mon[88407]: pgmap v1171: 89 pgs: 89 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 640 op/s 2024-09-14T04:20:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:14 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4243227320' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:20:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:14 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:20:15.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:15.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:15.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:15.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:15.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:15.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-14T04:20:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:15.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:15.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:15.220 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:15.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:15.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:15.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:15.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:15.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:15.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:15.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:15.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:15.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:15.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:15.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:15.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:15.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:15.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:15.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:15.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:15.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:15 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:20:15.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:15 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:20:15.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:15 smithi114 ceph-mon[103029]: osdmap e305: 8 total, 8 up, 8 in 2024-09-14T04:20:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:15 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:20:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:15 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:20:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:15 smithi114 ceph-mon[106969]: osdmap e305: 8 total, 8 up, 8 in 2024-09-14T04:20:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:15 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:20:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:15 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:20:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:15 smithi160 ceph-mon[88407]: osdmap e305: 8 total, 8 up, 8 in 2024-09-14T04:20:15.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:15.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:15.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:16.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:16.089 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:16.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:16.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:16.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:16.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:16.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:16.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:16.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:16.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:16.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:16.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:16.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:16.626 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:16.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:16.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:16.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:16.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:16.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:16.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:16 smithi114 ceph-mon[103029]: pgmap v1173: 89 pgs: 89 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s 2024-09-14T04:20:16.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:16 smithi114 ceph-mon[106969]: pgmap v1173: 89 pgs: 89 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s 2024-09-14T04:20:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:16.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:16.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:16.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:16 smithi160 ceph-mon[88407]: pgmap v1173: 89 pgs: 89 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s 2024-09-14T04:20:17.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:17.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:17.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:17.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:17.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:17.174 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:17.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:17.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:17.310 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:17.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:17.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:17.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:17.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:17.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:17.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:17.543 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-14T04:20:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:17.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:17.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:17.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:17.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:17.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:17.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:17.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:17.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:17.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:17 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/263247180' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:20:17.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:17 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:20:17.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:17 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/263247180' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:20:17.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:17 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:20:17.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:17 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/263247180' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:20:17.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:17 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:20:17.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:17.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:17.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:18.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:20:18.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:18.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:18.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:18.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:18.279 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:18.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:18.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:18.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:18.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:18.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:18.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:18.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:18.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:18.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:18.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:18.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:18.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:18 smithi114 ceph-mon[103029]: pgmap v1174: 89 pgs: 89 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s 2024-09-14T04:20:18.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:18 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:20:18.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:18 smithi114 ceph-mon[103029]: osdmap e306: 8 total, 8 up, 8 in 2024-09-14T04:20:18.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:18 smithi114 ceph-mon[106969]: pgmap v1174: 89 pgs: 89 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s 2024-09-14T04:20:18.820 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:18 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:20:18.820 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:18 smithi114 ceph-mon[106969]: osdmap e306: 8 total, 8 up, 8 in 2024-09-14T04:20:18.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:18.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:18 smithi160 ceph-mon[88407]: pgmap v1174: 89 pgs: 89 active+clean; 3.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 728 op/s 2024-09-14T04:20:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:18 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:20:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:18 smithi160 ceph-mon[88407]: osdmap e306: 8 total, 8 up, 8 in 2024-09-14T04:20:18.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:18.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:18.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:19.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:19.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:19.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:19.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:19.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:19.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:19.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-14T04:20:19.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:19.390 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:19.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:19.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:19.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:19.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:19.639 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:19.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:19.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:19.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:19.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:19.799 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:19.837 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:19.841 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:19.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:19.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:19.942 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:19.970 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:19.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:20.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:20.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:20.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:20.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:20.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:20.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:20.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:20.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:20.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:20.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:20.514 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:20.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:20.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:20.550 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:20:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:20:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:20:20.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:20.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:20.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:20.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:20.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:20.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:20.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:20.812 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:20 smithi114 ceph-mon[103029]: pgmap v1176: 89 pgs: 89 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-14T04:20:20.812 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:20 smithi114 ceph-mon[106969]: pgmap v1176: 89 pgs: 89 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-14T04:20:20.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:20.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:20 smithi160 ceph-mon[88407]: pgmap v1176: 89 pgs: 89 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-14T04:20:20.921 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:20.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:20.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:21.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:21.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:21.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:21.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:21.188 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:21.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:21.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:21.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:21.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:21.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:21.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:21.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:21.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:21.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:21.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:21.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:21.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:21.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:21.839 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1752627239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:21.840 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:20:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:20:21] "GET /metrics HTTP/1.1" 200 36572 "" "Prometheus/2.51.0" 2024-09-14T04:20:21.840 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1752627239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:21.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:21.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:21.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:21.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1752627239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:21.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:22.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:22.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:22.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:20:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:20:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:20:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:20:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:20:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:20:22.150 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:20:22.151 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:20:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:20:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:20:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:20:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:20:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:20:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:20:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:20:22.151 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:20:22.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:22.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:22.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:22.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:22.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:22.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:22.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:22.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:22.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:22.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:22.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:22.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:22.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:22.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:22.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:22.689 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:22.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:22.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:22.838 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:22.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:22.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:22 smithi114 ceph-mon[103029]: pgmap v1177: 89 pgs: 89 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-14T04:20:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:22 smithi114 ceph-mon[106969]: pgmap v1177: 89 pgs: 89 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-14T04:20:22.900 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:22 smithi160 ceph-mon[88407]: pgmap v1177: 89 pgs: 89 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-14T04:20:22.940 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:22.966 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:22.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:23.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:23.093 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:23.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:23.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:23.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:23.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:23.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:23.350 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:23.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:23.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:23.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:23.495 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:23.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:23.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:23.642 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:23.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:23.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:23.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:23.767 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:23.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:23.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:23.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:23.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:23.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:23.922 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:23.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:24.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:24.072 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:24.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:24.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:24.215 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:24.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:24.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:24.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:24.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:24.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:24.521 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:24.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:24.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:24 smithi114 ceph-mon[106969]: pgmap v1178: 89 pgs: 89 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 555 op/s 2024-09-14T04:20:24.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:24 smithi114 ceph-mon[103029]: pgmap v1178: 89 pgs: 89 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 555 op/s 2024-09-14T04:20:24.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:24.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:24.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:24.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:24.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:24.889 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:24 smithi160 ceph-mon[88407]: pgmap v1178: 89 pgs: 89 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 555 op/s 2024-09-14T04:20:24.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:20:24.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:24.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:24.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:25.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:25.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:25.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:25.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:25.193 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:25.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:25.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:25.322 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:25.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:25.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:25.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:25.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:25.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:25.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:25.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:25.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:25.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:25.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:25.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:25.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:25.873 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:25.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:25.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:26.027 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:26.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:26.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:26.168 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:26.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:26.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:26.303 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:26.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:26.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:26.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:26.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:26.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:26.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:26.582 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:26.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:26.593 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:20:26.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:26.729 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:26.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:26.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:26.854 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:26.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:26.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:26 smithi114 ceph-mon[106969]: pgmap v1179: 89 pgs: 89 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-14T04:20:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:26 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2609882068' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:26.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:26 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:26.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:26 smithi114 ceph-mon[103029]: pgmap v1179: 89 pgs: 89 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-14T04:20:26.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:26 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2609882068' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:26.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:26 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:26 smithi160 ceph-mon[88407]: pgmap v1179: 89 pgs: 89 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-14T04:20:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:26 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2609882068' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:26 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:26.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:26.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:26.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:27.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:27.137 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:27.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:27.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:27.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:27.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:27.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:27.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:27.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:27.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:27.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:27.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:27.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:27.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:27.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:27.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:27.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:27.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:27.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:27.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:27 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:20:27.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:27 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:20:27.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:27 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:20:27.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:27 smithi114 ceph-mon[106969]: osdmap e307: 8 total, 8 up, 8 in 2024-09-14T04:20:27.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:27 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:20:27.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:27 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:20:27.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:27 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:20:27.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:27 smithi114 ceph-mon[103029]: osdmap e307: 8 total, 8 up, 8 in 2024-09-14T04:20:27.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:27.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:27 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:20:27.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:27 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:20:27.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:27 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:20:27.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:27 smithi160 ceph-mon[88407]: osdmap e307: 8 total, 8 up, 8 in 2024-09-14T04:20:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:27.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:27.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:28.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:28.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:28.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:28.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:28.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:28.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:28.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:28.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:28.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:28.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:28.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:28.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:28.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:28.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:28.571 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:28.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:28.633 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-14T04:20:28.673 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:28.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:28.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:28.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:28.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:28.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:28.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:28 smithi114 ceph-mon[103029]: pgmap v1181: 89 pgs: 89 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T04:20:28.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:28 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4185558065' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:20:28.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:28 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:20:28.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:28 smithi114 ceph-mon[106969]: pgmap v1181: 89 pgs: 89 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T04:20:28.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:28 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4185558065' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:20:28.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:28 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:20:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:28 smithi160 ceph-mon[88407]: pgmap v1181: 89 pgs: 89 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 732 op/s 2024-09-14T04:20:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:28 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4185558065' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:20:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:28 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:20:28.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:28.955 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:28.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:28.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:29.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:29.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:29.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:29.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:29.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:29.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:29.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:29.351 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:29.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:29.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:29.499 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:29.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:29.508 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:29.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-14T04:20:29.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:29.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:29.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:29.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:29.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:29.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:29.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:29.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:29.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:29 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:20:29.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:29 smithi114 ceph-mon[106969]: osdmap e308: 8 total, 8 up, 8 in 2024-09-14T04:20:29.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:29 smithi114 ceph-mon[106969]: pgmap v1183: 89 pgs: 89 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 764 op/s 2024-09-14T04:20:29.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:29 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:20:29.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:29 smithi114 ceph-mon[103029]: osdmap e308: 8 total, 8 up, 8 in 2024-09-14T04:20:29.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:29 smithi114 ceph-mon[103029]: pgmap v1183: 89 pgs: 89 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 764 op/s 2024-09-14T04:20:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:29.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:29.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:29.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:29 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:20:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:29 smithi160 ceph-mon[88407]: osdmap e308: 8 total, 8 up, 8 in 2024-09-14T04:20:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:29 smithi160 ceph-mon[88407]: pgmap v1183: 89 pgs: 89 active+clean; 2.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 764 op/s 2024-09-14T04:20:29.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:30.045 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:30.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:30.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:30.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:30.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:30.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:30.321 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:30.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:30.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:30.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:30.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:30.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:30.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:30.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:20:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:20:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:20:30.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:30.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:30.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:30.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:30.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:30.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:30.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:31.015 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:31.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:31.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:31.164 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:31.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:31.246 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:20:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:20:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46463.0:191460 12.5 e308/284 12:a27cbe8d:::benchmark_data_smithi114_253692_object73481:head v 308'12085, mlcod=308'12077)", 2024-09-14T04:20:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:20:31.225946+0000", 2024-09-14T04:20:31.246 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.017151369999999999, 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017177899999999999, 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.225946+0000", 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.225946+0000", 2024-09-14T04:20:31.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:31.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.248 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:31.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.225947+0000", 2024-09-14T04:20:31.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8999999999999996e-07 2024-09-14T04:20:31.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:31.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.225953+0000", 2024-09-14T04:20:31.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1179999999999997e-06 2024-09-14T04:20:31.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:31.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.225963+0000", 2024-09-14T04:20:31.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3870000000000001e-06 2024-09-14T04:20:31.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:31.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.225968+0000", 2024-09-14T04:20:31.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7450000000000001e-06 2024-09-14T04:20:31.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.236651+0000", 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010683101 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.236672+0000", 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0134000000000001e-05 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46463.0:191469 12.d e308/284 12:b30346c3:::benchmark_data_smithi114_253692_object73490:head v 308'12019, mlcod=308'12017)", 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:20:31.237968+0000", 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0051292849999999999, 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0052182599999999997, 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:31.252 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.237968+0000", 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.237968+0000", 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.237969+0000", 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6500000000000008e-07 2024-09-14T04:20:31.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.237976+0000", 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7479999999999996e-06 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.237987+0000", 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1060000000000001e-05 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.237994+0000", 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7379999999999997e-06 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.241223+0000", 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032291590000000001 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:31.255 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.241241+0000", 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8286000000000001e-05 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:191473 12.6 12:63de3355:::benchmark_data_smithi114_253692_object73494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:20:31.238173+0000", 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0049251440000000002, 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0050639600000000002, 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:31.256 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 191473 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.238173+0000", 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.257 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.238173+0000", 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.238175+0000", 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.26e-06 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.238181+0000", 2024-09-14T04:20:31.258 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7849999999999997e-06 2024-09-14T04:20:31.259 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.238197+0000", 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5988999999999999e-05 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.238204+0000", 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3350000000000001e-06 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.260 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.239860+0000", 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0016561869999999999 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.239979+0000", 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000119053 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,7", 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.240092+0000", 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011272099999999999 2024-09-14T04:20:31.261 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46463.0:191471 12.d e308/284)", 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:20:31.241542+0000", 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001555346, 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001749153, 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.262 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.241542+0000", 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.241542+0000", 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.241543+0000", 2024-09-14T04:20:31.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0499999999999999e-06 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.241550+0000", 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.477e-06 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.241563+0000", 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3122000000000001e-05 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.241571+0000", 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5599999999999996e-06 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46463.0:191476 12.5 e308/284)", 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:20:31.242754+0000", 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00034349700000000001, 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00057142900000000003, 2024-09-14T04:20:31.265 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.242754+0000", 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.242754+0000", 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:31.266 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.242755+0000", 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6199999999999997e-07 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.242761+0000", 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7490000000000004e-06 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:31.267 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.242773+0000", 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2233e-05 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:20:31.242779+0000", 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5339999999999996e-06 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 5 2024-09-14T04:20:31.268 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:20:31.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:31.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:31.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:31.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:31.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:31.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:31.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:31.553 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:31.570 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:20:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:20:31] "GET /metrics HTTP/1.1" 200 36572 "" "Prometheus/2.51.0" 2024-09-14T04:20:31.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:31.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:31.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:31.711 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:31.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:31.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:31.846 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:31.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:31.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:31.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:31.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:31.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:32.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:20:32.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:32.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:32.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:32.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:32.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:32.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:32.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:32 smithi114 ceph-mon[106969]: pgmap v1184: 89 pgs: 89 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-09-14T04:20:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:32 smithi114 ceph-mon[103029]: pgmap v1184: 89 pgs: 89 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-09-14T04:20:32.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:32 smithi160 ceph-mon[88407]: pgmap v1184: 89 pgs: 89 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 880 op/s 2024-09-14T04:20:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:32.411 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:32.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:32.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:32.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:32.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:32.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:32.664 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:32.712 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:32.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:32.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-14T04:20:32.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:32.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:32.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:32.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:32.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:32.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:32.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:32.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:33.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:33.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:33.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:33.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:33.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:33.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:33.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:33.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:33.406 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:33.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:33.459 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:33.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:33.532 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:20:33.532 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:20:33.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:33.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:33.569 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:20:33.569 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:20:33.620 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:20:33.621 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:20:33.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:33.662 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:33.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:33.679 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:20:33.679 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:20:33.707 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:20:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:20:33.749 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:20:33.749 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:20:33.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:33.780 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:20:33.781 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:20:33.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:33.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:33.813 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:20:33.814 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:20:33.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:33.930 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:33.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:34.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:34.061 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:34.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:34.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:34 smithi114 ceph-mon[103029]: pgmap v1185: 89 pgs: 89 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 516 op/s 2024-09-14T04:20:34.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:34 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3407586978' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:34.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:34 smithi114 ceph-mon[106969]: pgmap v1185: 89 pgs: 89 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 516 op/s 2024-09-14T04:20:34.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:34.208 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:34.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:34.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:34.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:34.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:34.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:34 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3407586978' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:34 smithi160 ceph-mon[88407]: pgmap v1185: 89 pgs: 89 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 516 op/s 2024-09-14T04:20:34.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:34 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3407586978' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:34.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:34.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:34.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:34.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:34.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:34.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:34.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:34.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:34.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:34.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:34.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:34.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:34.881 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:34.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:34.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:35.008 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:35.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:35.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:35.140 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:35.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:35.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:35.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:35.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:35.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:35.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:35.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:35.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:35.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:35.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:35.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:35.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:35.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:35.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:35.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:35.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:35.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:35.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:35.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:35.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:36.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:36.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:36.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:36.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:36.240 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:36.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:36.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:36.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:36.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:36.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:36.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:36.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:36.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:36.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:36.629 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:36 smithi160 ceph-mon[88407]: pgmap v1186: 89 pgs: 89 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 857 op/s 2024-09-14T04:20:36.629 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:36 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:20:36.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:36.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:36.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:36.780 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:36 smithi114 ceph-mon[103029]: pgmap v1186: 89 pgs: 89 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 857 op/s 2024-09-14T04:20:36.780 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:36 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:20:36.780 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:36 smithi114 ceph-mon[106969]: pgmap v1186: 89 pgs: 89 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 857 op/s 2024-09-14T04:20:36.780 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:36 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:20:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:36.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:36.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:36.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:36.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:36.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:37.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:37.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:37.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:37.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:37.178 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:37.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:37.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:37.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:37.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:37.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:37.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:37.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:37.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:37.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:37.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:37.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:37.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:37.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:37.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:37.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:37.875 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:37.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:37.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:38.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:38.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:38.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:38.110 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:38.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:38.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:38.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:38.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:38.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:38.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:38.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:38.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:38.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:38.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:38.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:38.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:38 smithi160 ceph-mon[88407]: pgmap v1187: 89 pgs: 89 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-14T04:20:38.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:38.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:38.607 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2085706676' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:20:38.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:38.662 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:20:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-14T04:20:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:38.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:38.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:38.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:20:38.771 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:38 smithi114 ceph-mon[103029]: pgmap v1187: 89 pgs: 89 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-14T04:20:38.771 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:38.771 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:38.771 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2085706676' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:20:38.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:38 smithi114 ceph-mon[106969]: pgmap v1187: 89 pgs: 89 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 725 op/s 2024-09-14T04:20:38.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:38.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:38.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2085706676' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:20:38.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:38.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:38.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:38.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:38.931 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:38.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:39.033 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:39.059 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:39.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:39.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:39.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:39.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:39.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:39.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:39.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:39.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:39.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:39.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:39.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:39.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:39.604 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:39.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:39.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:39.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:39.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:39.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:39.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:39.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:39.870 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:39.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2377422195' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:20:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:20:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:20:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2085706676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:20:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:20:39.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[106969]: osdmap e309: 8 total, 8 up, 8 in 2024-09-14T04:20:39.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2377422195' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:20:39.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:20:39.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:20:39.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2085706676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:20:39.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:20:39.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:39 smithi114 ceph-mon[103029]: osdmap e309: 8 total, 8 up, 8 in 2024-09-14T04:20:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2377422195' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:20:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:39 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:20:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:39 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:20:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2085706676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:20:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:39 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:20:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:39 smithi160 ceph-mon[88407]: osdmap e309: 8 total, 8 up, 8 in 2024-09-14T04:20:39.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:39.992 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:39.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:39.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:40.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:40.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:40.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:40.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:40.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:40.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:40.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:40.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:40.388 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:40.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:40.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:40.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:40.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:40.517 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:20:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:20:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:20:40.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:40.638 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:40.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:40.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:40.775 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:40.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:40.778 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:40 smithi114 ceph-mon[103029]: pgmap v1189: 89 pgs: 89 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 734 op/s 2024-09-14T04:20:40.779 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:40 smithi114 ceph-mon[106969]: pgmap v1189: 89 pgs: 89 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 734 op/s 2024-09-14T04:20:40.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:40.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:40 smithi160 ceph-mon[88407]: pgmap v1189: 89 pgs: 89 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 734 op/s 2024-09-14T04:20:40.918 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:40.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:41.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:41.067 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:41.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:41.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:41.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:41.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:41.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:41.350 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:41.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:41.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:41.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:41.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:41.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:41.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:41.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:20:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:20:41] "GET /metrics HTTP/1.1" 200 36571 "" "Prometheus/2.51.0" 2024-09-14T04:20:41.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:41.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:41.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:41.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:41.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:41.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:41 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:41 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:41.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:41 smithi160 ceph-mon[88407]: pgmap v1190: 89 pgs: 89 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 729 op/s 2024-09-14T04:20:41.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:42.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:42.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:41 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:42.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:41 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:42.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:41 smithi114 ceph-mon[103029]: pgmap v1190: 89 pgs: 89 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 729 op/s 2024-09-14T04:20:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:41 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:41 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:42.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:41 smithi114 ceph-mon[106969]: pgmap v1190: 89 pgs: 89 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 729 op/s 2024-09-14T04:20:42.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:42.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:42.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:42.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-14T04:20:42.248 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:42.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:42.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:42.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:42.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:42.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:42.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:42.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:42.582 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:42.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:42.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:42.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:42.720 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:42.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:42.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:42.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:42.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:42.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:43.022 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:43.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:43.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:43.148 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:43.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:43.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:43.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:43.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:43.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:43.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:43.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:43.431 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:43.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:43.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:43.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:43.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:43.572 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:43.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:43.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:43.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:43.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:43.865 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:43.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:43.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:43.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:43.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:44.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:44.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:44.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:44.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:44.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:44 smithi114 ceph-mon[103029]: pgmap v1191: 89 pgs: 89 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 726 op/s 2024-09-14T04:20:44.213 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:44 smithi114 ceph-mon[106969]: pgmap v1191: 89 pgs: 89 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 726 op/s 2024-09-14T04:20:44.248 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:44.275 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:44.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:44.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:44.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:44.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:44.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:44 smithi160 ceph-mon[88407]: pgmap v1191: 89 pgs: 89 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 726 op/s 2024-09-14T04:20:44.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:44.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:44.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:44.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:44.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:44.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:44.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:44.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:44.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:44.885 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:20:44.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:44.973 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:44.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:44.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:45.049 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:45.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:45.098 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:45.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:45.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:45.260 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:45.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:45.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:45.386 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:45.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:45.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:45.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:45.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:45.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:45.623 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:20:45.623 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:20:45.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:45.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:45.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:45.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:45.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:45.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:45.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:45.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:45.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:46.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:46.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:46.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:46.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:46.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:46.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:46.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:46.332 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:46.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:46.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:46 smithi114 ceph-mon[106969]: pgmap v1192: 89 pgs: 89 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T04:20:46.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:46 smithi114 ceph-mon[103029]: pgmap v1192: 89 pgs: 89 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T04:20:46.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:46 smithi160 ceph-mon[88407]: pgmap v1192: 89 pgs: 89 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T04:20:46.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:46.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-14T04:20:46.458 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:46.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:46.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:46.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:46.615 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:46.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:46.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:46.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:46.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:46.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:46.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:46.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:46.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:47.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:47.070 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:47.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:47.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:47.205 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:47.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:47.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:47.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:47.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:47.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:47.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:47.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:47.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:47.631 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:47.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:47.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:47.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:47.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:47.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:47.906 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:47.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:48.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:48.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:48.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:48.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:48.175 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:48.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:48.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:48.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:48.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:48 smithi160 ceph-mon[88407]: pgmap v1193: 89 pgs: 89 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T04:20:48.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:48.445 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:48 smithi114 ceph-mon[103029]: pgmap v1193: 89 pgs: 89 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T04:20:48.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:48 smithi114 ceph-mon[106969]: pgmap v1193: 89 pgs: 89 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-14T04:20:48.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:48.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:48.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:48.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:48.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:48.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:48.760 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:48.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:48.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:48.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:49.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:49.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:49.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:49.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:49.139 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:49.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:49.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:49.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:49.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:49.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:49.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:49 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2068422127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:49.348 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:49 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2068422127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:49.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:49.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:49 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2068422127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:20:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:49.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:49.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:49.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:49.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:49.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:49.724 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:49.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:49.742 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:20:49.742 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:20:49.742 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:20:49.742 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:20:49.742 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:20:49.742 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:20:49.742 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:20:49.743 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:20:49.743 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:20:49.743 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:20:49.743 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:20:49.743 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:20:49.743 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:20:49.743 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:20:49.743 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:20:49.743 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:20:49.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:49.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:49.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:49.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:49.986 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:49.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:49.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:50.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:50.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:50.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:50.146 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-14T04:20:50.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:50.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:50.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:50.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:50.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:50 smithi114 ceph-mon[103029]: pgmap v1194: 89 pgs: 89 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 689 op/s 2024-09-14T04:20:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:50 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/652390675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:20:50.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:50 smithi114 ceph-mon[106969]: pgmap v1194: 89 pgs: 89 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 689 op/s 2024-09-14T04:20:50.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:50 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/652390675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:20:50.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:50.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:50.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:20:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:20:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:20:50.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:50 smithi160 ceph-mon[88407]: pgmap v1194: 89 pgs: 89 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 689 op/s 2024-09-14T04:20:50.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:50 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/652390675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:20:50.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:50.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:50.638 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:50.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:50.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:50.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:50.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:50.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:50.853 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:50.854 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:20:50.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:50.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:50.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:51.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:51.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:51.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:51.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:51.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:51.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:51.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:51.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:51 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/652390675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:20:51.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:51 smithi114 ceph-mon[106969]: osdmap e310: 8 total, 8 up, 8 in 2024-09-14T04:20:51.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/652390675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:20:51.288 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:51 smithi114 ceph-mon[103029]: osdmap e310: 8 total, 8 up, 8 in 2024-09-14T04:20:51.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:51.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:51.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-14T04:20:51.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:51.472 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:51.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:51.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:51.584 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:20:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:20:51] "GET /metrics HTTP/1.1" 200 36571 "" "Prometheus/2.51.0" 2024-09-14T04:20:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:51.601 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:51.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:51 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/652390675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:20:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:51 smithi160 ceph-mon[88407]: osdmap e310: 8 total, 8 up, 8 in 2024-09-14T04:20:51.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:51.757 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:51.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:51.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:51.905 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:51.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:52.008 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:52.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:52.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:52.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:52.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:52.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:52.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:52.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:52.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:52.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:52.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:52.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:52.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:52 smithi114 ceph-mon[103029]: pgmap v1196: 89 pgs: 89 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 719 op/s 2024-09-14T04:20:52.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:52 smithi114 ceph-mon[106969]: pgmap v1196: 89 pgs: 89 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 719 op/s 2024-09-14T04:20:52.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:52.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:20:52.584 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:52.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:52 smithi160 ceph-mon[88407]: pgmap v1196: 89 pgs: 89 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 719 op/s 2024-09-14T04:20:52.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:52.712 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:52.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:52.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:52.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:52.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:52.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:52.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:52.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:53.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:53.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:53.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:53.174 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:53.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:53.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:53.313 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:53.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:53.414 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:53.458 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:53.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:53.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:53.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:53.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:53.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:53.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:53.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:53.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:53.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:53.891 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:53.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:53.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:54.021 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:54.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:54.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:54.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:53 smithi114 ceph-mon[103029]: pgmap v1197: 89 pgs: 89 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-09-14T04:20:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:54.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:53 smithi114 ceph-mon[106969]: pgmap v1197: 89 pgs: 89 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-09-14T04:20:54.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:54.154 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:54.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:54.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:54.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:20:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:20:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:53 smithi160 ceph-mon[88407]: pgmap v1197: 89 pgs: 89 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 722 op/s 2024-09-14T04:20:54.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:54.294 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:54.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:54.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:54.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:54.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:54.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:54.547 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:54.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:54.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:54.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:54.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:54.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:54.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:54.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:54.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:54.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:54.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:54.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:54.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:55.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:55.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:55.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:55.171 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:55.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:55.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:55.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:55.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:55.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:55.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:55.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:55.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:55.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:55.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:55.582 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:55.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:55.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:55.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:55.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:55.811 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:55.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:55.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:55.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:55.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:55.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:56.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:56.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:56.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:56.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:56.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:56.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:56.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:56.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:56.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:56.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:56 smithi160 ceph-mon[88407]: pgmap v1198: 89 pgs: 89 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-09-14T04:20:56.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:56.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:56.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:56.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:56.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:56 smithi114 ceph-mon[106969]: pgmap v1198: 89 pgs: 89 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-09-14T04:20:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:56 smithi114 ceph-mon[103029]: pgmap v1198: 89 pgs: 89 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-09-14T04:20:56.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:56.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:56.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:56.799 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:56.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:56.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:56.939 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:56.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:57.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:57.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:57.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:57.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:57.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:57.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:57.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:57.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:57.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:57.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:57.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:57.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:57.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:57.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:57.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:57.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:57.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:57.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:57.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:57.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:57.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:57.895 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:57.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:57.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:20:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:58.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:20:58.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:58.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:58.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:58.183 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:58 smithi114 ceph-mon[106969]: pgmap v1199: 89 pgs: 89 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-09-14T04:20:58.183 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:58 smithi114 ceph-mon[103029]: pgmap v1199: 89 pgs: 89 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-09-14T04:20:58.227 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:20:58.227 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:20:58.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:20:58.227 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:20:58.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.227 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:76785 11.10 11:080615bb:::benchmark_data_smithi114_223345_object76784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:20:58.227 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:33.180485+0000", 2024-09-14T04:20:58.227 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 445.04321125799999, 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.28725904400000002, 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 76785 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.180485+0000", 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.180485+0000", 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.180486+0000", 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3790000000000001e-06 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.180510+0000", 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4223000000000002e-05 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.180521+0000", 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1051e-05 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.180528+0000", 2024-09-14T04:20:58.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8469999999999998e-06 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.188161+0000", 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0076328810000000002 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.188299+0000", 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013778999999999999 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,2,6", 2024-09-14T04:20:58.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.188447+0000", 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014835099999999999 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.191780+0000", 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033328379999999999 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.214762+0000", 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.022981556 2024-09-14T04:20:58.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.214776+0000", 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4369000000000001e-05 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.466318+0000", 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.25154167500000002 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.466390+0000", 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2106000000000004e-05 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.467744+0000", 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001353978 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:76795 11.10 11:095a3640:::benchmark_data_smithi114_223345_object76794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:33.185269+0000", 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 445.03842666399999, 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.283611259, 2024-09-14T04:20:58.235 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 76795 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.185269+0000", 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.185269+0000", 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.185271+0000", 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.714e-06 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.237 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.185293+0000", 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1991e-05 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.185305+0000", 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2276000000000001e-05 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.185312+0000", 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8140000000000003e-06 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.190194+0000", 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0048818999999999998 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.190329+0000", 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000134492 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,2,6", 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.190474+0000", 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014534300000000001 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.214631+0000", 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024157326999999999 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.214782+0000", 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015071099999999999 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.215010+0000", 2024-09-14T04:20:58.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00022808799999999999 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.467795+0000", 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.25278508199999999 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.467855+0000", 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9864999999999997e-05 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.468881+0000", 2024-09-14T04:20:58.242 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001025656 2024-09-14T04:20:58.242 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.242 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.242 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.243 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:76799 11.10 11:0efd1410:::benchmark_data_smithi114_223345_object76798:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:20:58.243 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:33.187112+0000", 2024-09-14T04:20:58.243 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 445.03658417899999, 2024-09-14T04:20:58.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.28285289299999999, 2024-09-14T04:20:58.243 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.243 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.243 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.243 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.243 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.243 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 76799 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.187112+0000", 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.187112+0000", 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.187114+0000", 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.669e-06 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.187139+0000", 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5476000000000001e-05 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.187148+0000", 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3859999999999996e-06 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.187155+0000", 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9789999999999996e-06 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.191819+0000", 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0046638629999999999 2024-09-14T04:20:58.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.191969+0000", 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014977299999999999 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,2,6", 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.192126+0000", 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000156668 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.214657+0000", 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.022531332000000001 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.214933+0000", 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00027607700000000001 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.215048+0000", 2024-09-14T04:20:58.248 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000115293 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.468935+0000", 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.25388612599999999 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.468988+0000", 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3208999999999998e-05 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.469965+0000", 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00097704200000000001 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:76804 11.10 11:0b45b37e:::benchmark_data_smithi114_223345_object76803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:33.192661+0000", 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 445.03103472599997, 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.28619028099999999, 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.250 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 76804 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.192661+0000", 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.192661+0000", 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.192663+0000", 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5719999999999999e-06 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.192685+0000", 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2266e-05 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.192697+0000", 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.148e-05 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.192705+0000", 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9200000000000004e-06 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.194819+0000", 2024-09-14T04:20:58.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002114084 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.194977+0000", 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015865000000000001 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,2,6", 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.195128+0000", 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015080200000000001 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.214829+0000", 2024-09-14T04:20:58.255 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.019700481999999998 2024-09-14T04:20:58.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.255 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.255 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.214939+0000", 2024-09-14T04:20:58.255 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000110223 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.215085+0000", 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000146644 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.475057+0000", 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.25997157100000001 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.256 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.475681+0000", 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00062348899999999997 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:33.478852+0000", 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031710979999999998 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.257 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80194 11.c 11:3acb4443:::benchmark_data_smithi114_223345_object80193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:36.218943+0000", 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 442.004752639, 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1812058510000001, 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80194 2024-09-14T04:20:58.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.218943+0000", 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.218943+0000", 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.259 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.218945+0000", 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5319999999999999e-06 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.218971+0000", 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6033999999999998e-05 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.218987+0000", 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5903e-05 2024-09-14T04:20:58.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.218995+0000", 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9680000000000006e-06 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.219066+0000", 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1068999999999993e-05 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:20:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.219074+0000", 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4009999999999997e-06 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.159175+0000", 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94010038600000001 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.159187+0000", 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2153e-05 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.379505+0000", 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2203178000000001 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.379644+0000", 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013889600000000001 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.379809+0000", 2024-09-14T04:20:58.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016546700000000001 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.394441+0000", 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014631473000000001 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.394487+0000", 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6011999999999999e-05 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.394563+0000", 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6545000000000007e-05 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.394780+0000", 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021684800000000001 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.400149+0000", 2024-09-14T04:20:58.265 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0053693639999999997 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:80208 11.c 11:3c6c799b:::benchmark_data_smithi114_223345_object80207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:13:36.266604+0000", 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 441.95709196600001, 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.141265422, 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.266 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 80208 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.266604+0000", 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.266604+0000", 2024-09-14T04:20:58.267 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.266606+0000", 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6610000000000001e-06 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.266625+0000", 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9117999999999999e-05 2024-09-14T04:20:58.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.266636+0000", 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0969e-05 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.266642+0000", 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9680000000000003e-06 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.269 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.266696+0000", 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3668999999999997e-05 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:36.266703+0000", 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2620000000000001e-06 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.159213+0000", 2024-09-14T04:20:58.270 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.89251029500000001 2024-09-14T04:20:58.271 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.271 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.271 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:20:58.271 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:37.159221+0000", 2024-09-14T04:20:58.271 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9200000000000004e-06 2024-09-14T04:20:58.271 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.271 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.271 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.271 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.390865+0000", 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2316443719999999 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.391065+0000", 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019969599999999999 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.391234+0000", 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016911399999999999 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.272 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.394507+0000", 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003273064 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.403959+0000", 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0094516340000000004 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.404008+0000", 2024-09-14T04:20:58.273 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9159999999999997e-05 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.404194+0000", 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018599600000000001 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:13:38.407870+0000", 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036755239999999999 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:123299 11.b e260/235 11:ddf171f5:::benchmark_data_smithi114_223345_object11362:head v 260'7678, mlcod=260'7676)", 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.365391+0000", 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 392.85830531800002, 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.28030756099999998, 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365391+0000", 2024-09-14T04:20:58.275 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.276 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.276 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.276 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365391+0000", 2024-09-14T04:20:58.276 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.276 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.276 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365392+0000", 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3599999999999999e-06 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365401+0000", 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8009999999999993e-06 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365417+0000", 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5979999999999999e-05 2024-09-14T04:20:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365425+0000", 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8760000000000005e-06 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365553+0000", 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012819099999999999 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.278 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365572+0000", 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9137999999999999e-05 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.645650+0000", 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.280077509 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.645698+0000", 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8705999999999998e-05 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.279 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:123294 11.f e260/235 11:f165a627:::benchmark_data_smithi114_223345_object11357:head v 260'7628, mlcod=260'7624)", 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.365460+0000", 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 392.85823638800002, 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.27701592899999999, 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.280 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.281 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365460+0000", 2024-09-14T04:20:58.281 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.281 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.281 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.281 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.281 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365460+0000", 2024-09-14T04:20:58.281 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.281 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.281 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.281 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365460+0000", 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0699999999999998e-07 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365465+0000", 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4469999999999999e-06 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365471+0000", 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5599999999999999e-06 2024-09-14T04:20:58.282 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.365476+0000", 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4719999999999997e-06 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.369931+0000", 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044557210000000002 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.369951+0000", 2024-09-14T04:20:58.283 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9313e-05 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.642398+0000", 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.27244782699999998 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.642476+0000", 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7181999999999994e-05 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.284 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:123308 11.13 e260/243 11:cdce9a5b:::benchmark_data_smithi114_223345_object11371:head v 260'5651, mlcod=260'5649)", 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.373955+0000", 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 392.84974114599999, 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.27177727099999999, 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.285 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373955+0000", 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373955+0000", 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373956+0000", 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.37e-07 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.286 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373959+0000", 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.72e-06 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373966+0000", 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9240000000000001e-06 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373971+0000", 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7559999999999999e-06 2024-09-14T04:20:58.287 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.288 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.288 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.288 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.374861+0000", 2024-09-14T04:20:58.288 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00089008800000000003 2024-09-14T04:20:58.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.288 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.288 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:58.290 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.374879+0000", 2024-09-14T04:20:58.290 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8136000000000001e-05 2024-09-14T04:20:58.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.290 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.290 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.290 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.645702+0000", 2024-09-14T04:20:58.290 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.27082262200000001 2024-09-14T04:20:58.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.645732+0000", 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0388000000000001e-05 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:123309 11.3 e260/243 11:c637ab3c:::benchmark_data_smithi114_223345_object11372:head v 260'5650, mlcod=260'5647)", 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.373984+0000", 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 392.849712496, 2024-09-14T04:20:58.291 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.271778933, 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373984+0000", 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.292 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373984+0000", 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373984+0000", 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3999999999999998e-07 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373987+0000", 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5729999999999999e-06 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373994+0000", 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5379999999999999e-06 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.373998+0000", 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4700000000000004e-06 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.375904+0000", 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001905693 2024-09-14T04:20:58.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.375922+0000", 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7513999999999999e-05 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.645735+0000", 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26981288199999998 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.295 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.645763+0000", 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8022999999999999e-05 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:123313 11.c 11:3539afb9:::benchmark_data_smithi114_223345_object11376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.376361+0000", 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 392.84733469299999, 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.28574513800000001, 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.296 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123313 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376361+0000", 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.297 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376361+0000", 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376363+0000", 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8819999999999999e-06 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376369+0000", 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1999999999999999e-06 2024-09-14T04:20:58.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376382+0000", 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2858e-05 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376391+0000", 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5189999999999992e-06 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376452+0000", 2024-09-14T04:20:58.299 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1538000000000006e-05 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376603+0000", 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015013 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.638345+0000", 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26174265699999999 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.641568+0000", 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032227079999999999 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.646573+0000", 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0050055020000000002 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.658559+0000", 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011985772 2024-09-14T04:20:58.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.658613+0000", 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3508999999999999e-05 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.662107+0000", 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034938629999999998 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.302 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.36273.0:123314 11.13 e260/243 11:c83b6f58:::benchmark_data_smithi114_223345_object11377:head v 260'5652, mlcod=260'5649)", 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:25.376705+0000", 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 392.84699113800002, 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.27086001500000001, 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376705+0000", 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376705+0000", 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376706+0000", 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2099999999999996e-07 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.376712+0000", 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5050000000000004e-06 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.638262+0000", 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.26154948099999997 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.638272+0000", 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0543999999999999e-05 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.639050+0000", 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000777749 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.639067+0000", 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7105999999999999e-05 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.647502+0000", 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0084344840000000008 2024-09-14T04:20:58.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:25.647565+0000", 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3424999999999997e-05 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:127243 11.f 11:fbb4aa71:::benchmark_data_smithi114_223345_object15306:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:29.573066+0000", 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 388.650630218, 2024-09-14T04:20:58.307 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0061637379999999, 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 127243 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573066+0000", 2024-09-14T04:20:58.308 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573066+0000", 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573068+0000", 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0770000000000002e-06 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.309 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573074+0000", 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9479999999999996e-06 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573088+0000", 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4579000000000001e-05 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573096+0000", 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5950000000000001e-06 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573160+0000", 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3986999999999996e-05 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573169+0000", 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9150000000000002e-06 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.553115+0000", 2024-09-14T04:20:58.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97994636099999999 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.555491+0000", 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023760389999999999 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.555765+0000", 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00027377800000000002 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.567674+0000", 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011908372 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.568212+0000", 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00053813199999999998 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.579134+0000", 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010922189000000001 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.579179+0000", 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5142000000000002e-05 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.579230+0000", 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0624000000000002e-05 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.314 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:127260 11.f 11:f98b95da:::benchmark_data_smithi114_223345_object15323:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:29.606687+0000", 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 388.61700872, 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97418977699999998, 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 127260 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606687+0000", 2024-09-14T04:20:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606687+0000", 2024-09-14T04:20:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606689+0000", 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8199999999999999e-06 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606694+0000", 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3950000000000003e-06 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606707+0000", 2024-09-14T04:20:58.317 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3210999999999999e-05 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606714+0000", 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2649999999999999e-06 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606770+0000", 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5887999999999998e-05 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:20:58.318 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606787+0000", 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7133999999999999e-05 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.568243+0000", 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96145623899999999 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.568340+0000", 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6434000000000005e-05 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.568452+0000", 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000112323 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.574771+0000", 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0063187019999999998 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.577844+0000", 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00307355 2024-09-14T04:20:58.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.579275+0000", 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0014305190000000001 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.580790+0000", 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015148749999999999 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.321 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.580877+0000", 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7422000000000004e-05 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206303 11.c 11:3418ab87:::benchmark_data_smithi114_223345_object94366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.960541+0000", 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 300.26315480900001, 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.33908080299999999, 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.322 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.323 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.323 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.323 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.323 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206303 2024-09-14T04:20:58.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.323 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.323 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.323 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.323 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960541+0000", 2024-09-14T04:20:58.323 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960541+0000", 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960543+0000", 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8509999999999999e-06 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.324 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960548+0000", 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6739999999999996e-06 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960563+0000", 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5281000000000002e-05 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960571+0000", 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1009999999999992e-06 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.325 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.965441+0000", 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0048702119999999996 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.965585+0000", 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000143156 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.965712+0000", 2024-09-14T04:20:58.326 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012754899999999999 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.972223+0000", 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.006511317 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.972368+0000", 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014410399999999999 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.327 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.297537+0000", 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32516935299999999 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.297660+0000", 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122853 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.299622+0000", 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019623520000000001 2024-09-14T04:20:58.328 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206312 11.f 11:f5f3464b:::benchmark_data_smithi114_223345_object94375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.973030+0000", 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 300.250666502, 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32513937399999998, 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.329 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206312 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973030+0000", 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973030+0000", 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973031+0000", 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1069999999999999e-06 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973035+0000", 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8299999999999998e-06 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973044+0000", 2024-09-14T04:20:58.331 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.217e-06 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973049+0000", 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1309999999999997e-06 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973106+0000", 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7473999999999999e-05 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.332 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973236+0000", 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013012000000000001 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973368+0000", 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000131524 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977449+0000", 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0040805299999999997 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977463+0000", 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4245e-05 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.297832+0000", 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32036935300000002 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.298033+0000", 2024-09-14T04:20:58.334 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020055 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.298169+0000", 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000136293 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206314 11.c 11:34914179:::benchmark_data_smithi114_223345_object94377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.973883+0000", 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 300.24981333300002, 2024-09-14T04:20:58.335 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32750190400000001, 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206314 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973883+0000", 2024-09-14T04:20:58.336 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973883+0000", 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973884+0000", 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8400000000000003e-07 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.337 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973888+0000", 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1250000000000003e-06 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973902+0000", 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4521e-05 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973909+0000", 2024-09-14T04:20:58.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1110000000000002e-06 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973979+0000", 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9716999999999997e-05 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.974121+0000", 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014170099999999999 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:20:58.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.974257+0000", 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013602300000000001 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.980526+0000", 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062688589999999999 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.981607+0000", 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010813590000000001 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.299736+0000", 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31812907899999998 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.299790+0000", 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4342999999999998e-05 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.301385+0000", 2024-09-14T04:20:58.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001594182 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206317 11.c 11:3f6a083c:::benchmark_data_smithi114_223345_object94380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.977365+0000", 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 300.24633064699998, 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32865713099999999, 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.342 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206317 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977365+0000", 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977365+0000", 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977367+0000", 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.33e-06 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977371+0000", 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5730000000000001e-06 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977380+0000", 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1799999999999996e-06 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977384+0000", 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6349999999999997e-06 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977443+0000", 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8427999999999997e-05 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977552+0000", 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000109881 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977628+0000", 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5611000000000005e-05 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.980534+0000", 2024-09-14T04:20:58.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029058560000000001 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.987282+0000", 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0067482999999999996 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.304204+0000", 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31692202899999999 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.347 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.348 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.304393+0000", 2024-09-14T04:20:58.348 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000188238 2024-09-14T04:20:58.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.348 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.348 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.306023+0000", 2024-09-14T04:20:58.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00163007 2024-09-14T04:20:58.349 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.349 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.349 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.349 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206323 11.f 11:f3df4957:::benchmark_data_smithi114_223345_object94386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.991379+0000", 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 300.23231714000002, 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30869569899999999, 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206323 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.350 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991379+0000", 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991379+0000", 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991380+0000", 2024-09-14T04:20:58.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.438e-06 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991385+0000", 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7720000000000002e-06 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991397+0000", 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1904e-05 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991403+0000", 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6389999999999997e-06 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991458+0000", 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5751000000000002e-05 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991588+0000", 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000129682 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:20:58.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991722+0000", 2024-09-14T04:20:58.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000133776 2024-09-14T04:20:58.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.994513+0000", 2024-09-14T04:20:58.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027913439999999999 2024-09-14T04:20:58.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.994555+0000", 2024-09-14T04:20:58.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1752000000000003e-05 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.299991+0000", 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.305436229 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.300038+0000", 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6699000000000001e-05 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.300075+0000", 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6712999999999999e-05 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:148239 12.6 12:65351756:::benchmark_data_smithi114_253692_object30260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:19:33.546419+0000", 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 84.677276857999999, 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.211842855, 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:20:58.358 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 148239 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.546419+0000", 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:20:58.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.546419+0000", 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.546420+0000", 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7300000000000001e-07 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.546422+0000", 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5100000000000001e-06 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.546428+0000", 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1519999999999997e-06 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.546433+0000", 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9599999999999999e-06 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.556733+0000", 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010299564000000001 2024-09-14T04:20:58.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.556873+0000", 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014041100000000001 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,7", 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.745008+0000", 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.18813480199999999 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:20:58.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.749379+0000", 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043712170000000002 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.749408+0000", 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8802999999999998e-05 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.757950+0000", 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0085419480000000006 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.758079+0000", 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012887700000000001 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:19:33.758262+0000", 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018313800000000001 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:20:58.364 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:20:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:58.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:58.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:58 smithi160 ceph-mon[88407]: pgmap v1199: 89 pgs: 89 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 740 op/s 2024-09-14T04:20:58.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:58.509 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:58.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:58.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:20:58.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:58.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:20:58.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:58.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:58.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:58.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:58.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:20:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:58.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:20:58.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:59.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:59.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:59.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:59.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:59.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:59.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:20:59.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:59.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:20:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:59 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1602122407' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:59.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:20:59 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:59 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1602122407' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:59.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:20:59 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:59.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:20:59.398 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:59.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:20:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:59 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1602122407' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:59.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:20:59 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:20:59.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:20:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:59.549 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:59.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:20:59.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:20:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:59.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:20:59.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:20:59.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-14T04:20:59.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:59.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:59.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:20:59.915 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:20:59.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:20:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:20:59.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:00.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:00.062 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:00.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:00.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:00.187 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:00.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:00.282 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:21:00.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:00 smithi114 ceph-mon[106969]: pgmap v1200: 89 pgs: 89 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T04:21:00.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:00 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:00.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:00 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:21:00.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:00 smithi114 ceph-mon[106969]: osdmap e311: 8 total, 8 up, 8 in 2024-09-14T04:21:00.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:00 smithi114 ceph-mon[103029]: pgmap v1200: 89 pgs: 89 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T04:21:00.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:00 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:00.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:00 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:21:00.284 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:00 smithi114 ceph-mon[103029]: osdmap e311: 8 total, 8 up, 8 in 2024-09-14T04:21:00.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:00.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:00.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:21:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:21:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:21:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:00 smithi160 ceph-mon[88407]: pgmap v1200: 89 pgs: 89 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T04:21:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:00 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:00 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:21:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:00 smithi160 ceph-mon[88407]: osdmap e311: 8 total, 8 up, 8 in 2024-09-14T04:21:00.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:00.445 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:00.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:00.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:00.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:00.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:00.691 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:00.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:00.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:00.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:00.859 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:00.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:00.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:00.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:00.996 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:00.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:01.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:01.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:01.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:01.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:01.282 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:01.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:01.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:01.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:01.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:01.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:01.539 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:01.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:01.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:01.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:01.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:01.773 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:01.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:01.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:01.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:21:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:21:01] "GET /metrics HTTP/1.1" 200 36573 "" "Prometheus/2.51.0" 2024-09-14T04:21:01.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:01.922 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:01.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:02.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:02.096 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:02.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:02.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:02.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:02.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:02.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:02.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:02 smithi114 ceph-mon[103029]: pgmap v1202: 89 pgs: 89 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T04:21:02.267 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:02 smithi114 ceph-mon[106969]: pgmap v1202: 89 pgs: 89 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T04:21:02.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:02.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:02.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:02 smithi160 ceph-mon[88407]: pgmap v1202: 89 pgs: 89 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T04:21:02.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:02.525 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:02.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:02.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:02.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:02.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:02.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:02.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:02.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:02.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:02.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:02.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:02.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:02.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:03.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:03.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:03.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:03.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:03.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:03.187 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:03.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:03.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:21:03.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:03.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:03.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:03.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:03.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:03.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:03.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:03.587 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:03.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:03.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:03.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:03.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:03.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:03.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:03.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:03.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:03.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:03.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:04.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:04.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:04.125 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:04.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:04.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:04.254 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:04.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:04.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:04.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:04 smithi114 ceph-mon[106969]: pgmap v1203: 89 pgs: 89 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T04:21:04.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:04 smithi114 ceph-mon[103029]: pgmap v1203: 89 pgs: 89 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T04:21:04.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:04.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:04.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:04.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:04 smithi160 ceph-mon[88407]: pgmap v1203: 89 pgs: 89 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 753 op/s 2024-09-14T04:21:04.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:04.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:04.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:04.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:04.662 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:04.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:04.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:04.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:04.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:04.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:04.940 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:04.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:05.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:05.079 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:05.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:05.181 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:05.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:05.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:05.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:05.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:05.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:05.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:05.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:05.504 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:05.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:05.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:05.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:05.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:05.719 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:21:05.720 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:21:05.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:05.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:05.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:05.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:05.758 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:21:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:21:05.803 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:21:05.803 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:21:05.835 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:21:05.835 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:21:05.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:05.872 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:21:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:21:05.884 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:05.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:05.901 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:21:05.902 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:21:05.931 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:21:05.932 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:21:05.961 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:21:05.961 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:21:05.985 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:06.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:06.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:06.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:06.165 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:06.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:06.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:06.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:06.432 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:06.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:06 smithi114 ceph-mon[103029]: pgmap v1204: 89 pgs: 89 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:06.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3589334856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:06.450 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:06 smithi114 ceph-mon[106969]: pgmap v1204: 89 pgs: 89 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:06.450 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3589334856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:06.456 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:06.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:06.558 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:06.584 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:06.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:06 smithi160 ceph-mon[88407]: pgmap v1204: 89 pgs: 89 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3589334856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:06.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:06.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:06.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:06.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:06.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:06.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:06.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:06.990 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:06.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:06.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:07.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:07.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:07.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:07.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:07.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:07.294 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:07.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:07.369 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:07.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:07.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:21:07.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:21:07.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:07.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:07.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:07.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:21:07.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:21:07.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:07.381 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:07.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:07.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:07.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:07.456 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:07.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:07.569 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:07.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:21:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:21:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:07.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:07.694 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:07.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:07.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:07.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:07.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:07.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:07.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:07.975 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:07.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:08.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:08.119 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:08.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:08.195 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-14T04:21:08.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:08 smithi114 ceph-mon[103029]: pgmap v1205: 89 pgs: 89 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:08.199 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:08.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:08.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:08.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:08.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:08.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:08.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:08.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:08.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:08.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:08.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2816282641' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:21:08.577 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:08 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:21:08.578 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:08 smithi114 ceph-mon[106969]: pgmap v1205: 89 pgs: 89 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:08.578 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:08.578 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2816282641' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:21:08.578 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:08 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:21:08.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:08 smithi160 ceph-mon[88407]: pgmap v1205: 89 pgs: 89 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2816282641' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:21:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:08 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:21:08.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:08.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:08.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:08.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:08.792 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:08.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:08.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:08.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:08.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:08.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:09.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:09.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:09.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:09.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:21:09.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:09.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:09.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:09.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:09.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:09.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:09.485 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:09.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:09.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:09.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:21:09.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:09 smithi114 ceph-mon[103029]: osdmap e312: 8 total, 8 up, 8 in 2024-09-14T04:21:09.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:21:09.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:09 smithi114 ceph-mon[106969]: osdmap e312: 8 total, 8 up, 8 in 2024-09-14T04:21:09.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:09.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:09.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:09.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:21:09.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:09 smithi160 ceph-mon[88407]: osdmap e312: 8 total, 8 up, 8 in 2024-09-14T04:21:09.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:09.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:09.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:09.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:09.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:09.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:09.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:10.012 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:10.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:10.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:10.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:10.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:10.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:10.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:10.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:10.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:10.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:10.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:10.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:10.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:10.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:10.613 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:10 smithi114 ceph-mon[106969]: pgmap v1207: 89 pgs: 89 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-09-14T04:21:10.614 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:10 smithi114 ceph-mon[103029]: pgmap v1207: 89 pgs: 89 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-09-14T04:21:10.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:10 smithi160 ceph-mon[88407]: pgmap v1207: 89 pgs: 89 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-09-14T04:21:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:21:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:21:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:21:10.665 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:10.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:10.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:10.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:10.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:10.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:10.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:10.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:10.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:11.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:11.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:11.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:11.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:11.190 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:11.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:11.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:11.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:11.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:11.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:11.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:11.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:11.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:11.576 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:11.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:21:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:21:11] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-09-14T04:21:11.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-14T04:21:11.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:11.700 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:11.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:11.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:11.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:11.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:11.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:11.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:11.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:12.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:12.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:12.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:12.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:12.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:12.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:12.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:12.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:12.382 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:12.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:12.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:12 smithi160 ceph-mon[88407]: pgmap v1208: 89 pgs: 89 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-14T04:21:12.541 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:12.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:12.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:12.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:12 smithi114 ceph-mon[103029]: pgmap v1208: 89 pgs: 89 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-14T04:21:12.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:12 smithi114 ceph-mon[106969]: pgmap v1208: 89 pgs: 89 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-14T04:21:12.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:12.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:12.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:12.812 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:12.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:12.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:12.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:12.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:12.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:12.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:12.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:13.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:13.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:13.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:13.188 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:13.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:13.257 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:13.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:13.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:13.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:13.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:13.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:13.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:13.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:13.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:13.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:13.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:13.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:13.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:13.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:13.825 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:13.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:13.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:13.967 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:13.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:14.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:14.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:14.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:14.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:14.236 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:21:14.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:14.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:14.248 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:14.336 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:14 smithi114 ceph-mon[103029]: pgmap v1209: 89 pgs: 89 active+clean; 1.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 716 op/s 2024-09-14T04:21:14.336 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:14 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3742422009' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:21:14.336 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:14 smithi114 ceph-mon[106969]: pgmap v1209: 89 pgs: 89 active+clean; 1.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 716 op/s 2024-09-14T04:21:14.337 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:14 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3742422009' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:21:14.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:14.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:14.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:14.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:14.525 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:14.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:14.573 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:21:14.573 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:21:14.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:14.656 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:14.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:14.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:14 smithi160 ceph-mon[88407]: pgmap v1209: 89 pgs: 89 active+clean; 1.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 716 op/s 2024-09-14T04:21:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:14 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3742422009' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:21:14.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:14.786 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:14.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:14.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:14.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:14.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:14.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:15.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:15.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:15.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:15.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:15.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:15.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:15.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:15.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:15.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:15.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:15.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:15.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:15.543 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:15 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:15.544 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3742422009' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:21:15.544 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:15 smithi114 ceph-mon[106969]: osdmap e313: 8 total, 8 up, 8 in 2024-09-14T04:21:15.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:15 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:15.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3742422009' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:21:15.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:15 smithi114 ceph-mon[103029]: osdmap e313: 8 total, 8 up, 8 in 2024-09-14T04:21:15.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:15.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:15.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:15 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3742422009' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:21:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:15 smithi160 ceph-mon[88407]: osdmap e313: 8 total, 8 up, 8 in 2024-09-14T04:21:15.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:15.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:15.730 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:15.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:15.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:15.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:15.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:15.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:15.993 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:15.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:16.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:16.126 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:16.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:16.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:16.254 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:16.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:16.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-14T04:21:16.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:16.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:16.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:16.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:16.541 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:16 smithi114 ceph-mon[103029]: pgmap v1211: 89 pgs: 89 active+clean; 845 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 908 op/s 2024-09-14T04:21:16.541 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:16 smithi114 ceph-mon[106969]: pgmap v1211: 89 pgs: 89 active+clean; 845 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 908 op/s 2024-09-14T04:21:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:16.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:16.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:16 smithi160 ceph-mon[88407]: pgmap v1211: 89 pgs: 89 active+clean; 845 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 908 op/s 2024-09-14T04:21:16.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:16.715 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:16.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:16.817 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:16.854 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:16.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:16.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:16.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:16.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:17.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:17.140 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:17.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:17.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:17.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:17.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:17.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:17.413 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:17.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:17.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:17.559 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:17.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:17.661 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:17.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:17.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:17.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:17.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:17.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:17.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:17.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:17.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:18.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:18.088 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:18.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:18.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:18.242 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:18.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:18.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:18.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:18.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:18.488 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:18 smithi114 ceph-mon[103029]: pgmap v1212: 89 pgs: 89 active+clean; 845 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 767 op/s 2024-09-14T04:21:18.488 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:18 smithi114 ceph-mon[106969]: pgmap v1212: 89 pgs: 89 active+clean; 845 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 767 op/s 2024-09-14T04:21:18.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:18.520 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:18.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:18.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:18.622 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:18.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:18.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:18 smithi160 ceph-mon[88407]: pgmap v1212: 89 pgs: 89 active+clean; 845 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 767 op/s 2024-09-14T04:21:18.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:18.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:18.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:18.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:18.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:18.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:18.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:19.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:19.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:19.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:19.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:21:19.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:19.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:19.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:19.204 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:21:19.205 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:21:19.205 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:21:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:21:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:21:19.205 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:21:19.215 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:19.225 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:21:19.225 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:21:19.264 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:21:19.264 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:21:19.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:19.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:21:19.297 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:21:19.336 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:21:19.337 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:21:19.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:19.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:19.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:19.379 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:21:19.379 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:21:19.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:19 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1767169105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:19.385 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1767169105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:19.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:19.472 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:19.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:19.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:19.621 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:19.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:19 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1767169105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:19.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:19.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:19.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:19.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:19.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:19.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:19.980 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:20.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:20.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:20.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:20.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:20.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:20.230 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:20.262 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:20.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:20.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:20.287 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.4 to 1 (10000) 2024-09-14T04:21:20.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:20.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:20.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:20.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:20.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:20.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:20 smithi114 ceph-mon[106969]: pgmap v1213: 89 pgs: 89 active+clean; 694 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-09-14T04:21:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:20 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1601775875' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:21:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:20 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:21:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:20 smithi114 ceph-mon[103029]: pgmap v1213: 89 pgs: 89 active+clean; 694 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-09-14T04:21:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:20 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1601775875' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:21:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:20 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:21:20.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:20.650 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:20.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:21:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:21:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:21:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:20 smithi160 ceph-mon[88407]: pgmap v1213: 89 pgs: 89 active+clean; 694 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 741 op/s 2024-09-14T04:21:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:20 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1601775875' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:21:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:20 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:21:20.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:20.784 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:20.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:20.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:20.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:20.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:21.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:21.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:21.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:21.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:21.172 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-14T04:21:21.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:21:21.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:21.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:21.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:21.348 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:21.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:21.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:21 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:21:21.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:21 smithi114 ceph-mon[103029]: osdmap e314: 8 total, 8 up, 8 in 2024-09-14T04:21:21.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:21 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:21:21.434 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:21 smithi114 ceph-mon[106969]: osdmap e314: 8 total, 8 up, 8 in 2024-09-14T04:21:21.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:21.496 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:21.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:21.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:21.621 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:21.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:21 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:21:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:21 smithi160 ceph-mon[88407]: osdmap e314: 8 total, 8 up, 8 in 2024-09-14T04:21:21.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:21.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:21.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:21.849 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:21.861 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:21:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:21:21] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-09-14T04:21:21.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:21.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:21.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:22.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:22.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:22.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:22.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:22.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:22.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:22.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:22.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:22.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:22.416 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:22.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:22.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:22.520 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:22 smithi114 ceph-mon[103029]: pgmap v1215: 89 pgs: 89 active+clean; 609 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 873 op/s 2024-09-14T04:21:22.520 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:22 smithi114 ceph-mon[106969]: pgmap v1215: 89 pgs: 89 active+clean; 609 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 873 op/s 2024-09-14T04:21:22.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:22.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:22 smithi160 ceph-mon[88407]: pgmap v1215: 89 pgs: 89 active+clean; 609 MiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 873 op/s 2024-09-14T04:21:22.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:22.695 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:22.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:22.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:22.833 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:22.836 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:22.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:22.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:22.870 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:21:22.870 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:21:22.870 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:21:22.870 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:21:22.870 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:21:22.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:22.969 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:22.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:23.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:23.098 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:23.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:23.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:23.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:23.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:23.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:23.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:23.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:23.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:23.489 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:23.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:23.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:23.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:23.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:23.650 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:23.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:23.751 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:23.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:23.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:23.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:23.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:23.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:24.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:24.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:24.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:24.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:24.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:24.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:24.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:24.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:21:24.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:24.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:24.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:24.432 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:24.457 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:24.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:24.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:24.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:24.599 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:24.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:24.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:24.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:24 smithi114 ceph-mon[103029]: pgmap v1216: 89 pgs: 89 active+clean; 581 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 630 op/s 2024-09-14T04:21:24.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:24 smithi114 ceph-mon[106969]: pgmap v1216: 89 pgs: 89 active+clean; 581 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 630 op/s 2024-09-14T04:21:24.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:24 smithi160 ceph-mon[88407]: pgmap v1216: 89 pgs: 89 active+clean; 581 MiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 630 op/s 2024-09-14T04:21:24.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:24.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:24.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:24.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:24.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:24.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:24.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:24.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:24.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:25.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:25.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:25.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:25.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:25.251 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:25.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:25.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:25.401 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:25.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:25.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:25.556 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:25.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:25.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:25.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:25.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:25.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:25.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:25.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:25.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:25.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:25.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:25.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:26.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:26.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:26.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:26.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:26.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:26.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:26.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:26.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:26.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:26.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:26.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:26.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:26.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:26.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:26.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:26 smithi114 ceph-mon[103029]: pgmap v1217: 89 pgs: 89 active+clean; 380 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:26.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:26 smithi114 ceph-mon[106969]: pgmap v1217: 89 pgs: 89 active+clean; 380 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:26.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:26.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:26 smithi160 ceph-mon[88407]: pgmap v1217: 89 pgs: 89 active+clean; 380 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:26.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-14T04:21:26.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:26.767 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:26.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:26.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:26.895 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:26.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:26.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:27.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:27.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:27.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:27.152 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:27.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:27.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:27.276 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:27.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:27.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:27.432 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:27.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:27.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:27.572 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:27.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:27.575 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:27.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:21:27.647 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:21:27.656 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:27.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:27.698 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:27.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:27.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:27.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:27.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:27.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:27.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:27.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:27.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:28.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:28.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:28.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:28.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:28.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:28.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:28.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:28.312 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:21:28.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:28.351 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:28.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:28.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:28.477 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:28.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:28.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:28.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:28.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:28.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:28 smithi114 ceph-mon[106969]: pgmap v1218: 89 pgs: 89 active+clean; 380 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:28 smithi114 ceph-mon[106969]: from='client.56152 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:21:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:28 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2946810147' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:21:28.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:28 smithi114 ceph-mon[103029]: pgmap v1218: 89 pgs: 89 active+clean; 380 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:28 smithi114 ceph-mon[103029]: from='client.56152 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:21:28.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:28 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2946810147' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:21:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:28 smithi160 ceph-mon[88407]: pgmap v1218: 89 pgs: 89 active+clean; 380 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 743 op/s 2024-09-14T04:21:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:28 smithi160 ceph-mon[88407]: from='client.56152 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:21:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:28 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2946810147' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:21:28.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:28.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:28.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:28.789 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1218,"stamp":"2024-09-14T04:21:27.082364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398727141,"num_objects":6102,"num_object_clones":0,"num_object_copies":18306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6102,"num_whiteouts":0,"num_read":293,"num_read_kb":748,"num_write":230090,"num_write_kb":7552124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":595202,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141636,"ondisk_log_size":141636,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8601352,"kb_used_data":1277960,"kb_used_omap":33,"kb_used_meta":7323038,"kb_avail":741359864,"statfs":{"total":767960285184,"available":759152500736,"internally_reserved":0,"allocated":1308631040,"data_stored":1297314394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34628,"internal_metadata":7498791100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,6,9,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-487522304,"num_objects":-7439,"num_object_clones":0,"num_object_copies":-22317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.008698"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":440,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.289115+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:21:20.289115+0000","last_peered":"2024-09-14T04:21:20.289115+0000","last_clean":"2024-09-14T04:21:20.289115+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:21:20.289115+0000","last_undegraded":"2024-09-14T04:21:20.289115+0000","last_fullsized":"2024-09-14T04:21:20.289115+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":313,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.246095+0000","last_change":"2024-09-14T04:13:37.176637+0000","last_active":"2024-09-14T04:21:14.246095+0000","last_peered":"2024-09-14T04:21:14.246095+0000","last_clean":"2024-09-14T04:21:14.246095+0000","last_became_active":"2024-09-14T04:13:37.159759+0000","last_became_peered":"2024-09-14T04:13:37.159759+0000","last_unstale":"2024-09-14T04:21:14.246095+0000","last_undegraded":"2024-09-14T04:21:14.246095+0000","last_fullsized":"2024-09-14T04:21:14.246095+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:56.293937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":279,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235574+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:21:14.235574+0000","last_peered":"2024-09-14T04:21:14.235574+0000","last_clean":"2024-09-14T04:21:14.235574+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:21:14.235574+0000","last_undegraded":"2024-09-14T04:21:14.235574+0000","last_fullsized":"2024-09-14T04:21:14.235574+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":525,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.296526+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:21:20.296526+0000","last_peered":"2024-09-14T04:21:20.296526+0000","last_clean":"2024-09-14T04:21:20.296526+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:21:20.296526+0000","last_undegraded":"2024-09-14T04:21:20.296526+0000","last_fullsized":"2024-09-14T04:21:20.296526+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":510,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290749+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:21:20.290749+0000","last_peered":"2024-09-14T04:21:20.290749+0000","last_clean":"2024-09-14T04:21:20.290749+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:21:20.290749+0000","last_undegraded":"2024-09-14T04:21:20.290749+0000","last_fullsized":"2024-09-14T04:21:20.290749+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":276,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235612+0000","last_change":"2024-09-14T04:14:30.541064+0000","last_active":"2024-09-14T04:21:14.235612+0000","last_peered":"2024-09-14T04:21:14.235612+0000","last_clean":"2024-09-14T04:21:14.235612+0000","last_became_active":"2024-09-14T04:14:30.540755+0000","last_became_peered":"2024-09-14T04:14:30.540755+0000","last_unstale":"2024-09-14T04:21:14.235612+0000","last_undegraded":"2024-09-14T04:21:14.235612+0000","last_fullsized":"2024-09-14T04:21:14.235612+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:37.189820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":253,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.289908+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:21:20.289908+0000","last_peered":"2024-09-14T04:21:20.289908+0000","last_clean":"2024-09-14T04:21:20.289908+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:21:20.289908+0000","last_undegraded":"2024-09-14T04:21:20.289908+0000","last_fullsized":"2024-09-14T04:21:20.289908+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":288,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.239031+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:21:14.239031+0000","last_peered":"2024-09-14T04:21:14.239031+0000","last_clean":"2024-09-14T04:21:14.239031+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:21:14.239031+0000","last_undegraded":"2024-09-14T04:21:14.239031+0000","last_fullsized":"2024-09-14T04:21:14.239031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"314'14565","reported_seq":14609,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:23.357490+0000","last_change":"2024-09-14T04:17:48.104166+0000","last_active":"2024-09-14T04:21:23.357490+0000","last_peered":"2024-09-14T04:21:23.357490+0000","last_clean":"2024-09-14T04:21:23.357490+0000","last_became_active":"2024-09-14T04:17:48.083298+0000","last_became_peered":"2024-09-14T04:17:48.083298+0000","last_unstale":"2024-09-14T04:21:23.357490+0000","last_undegraded":"2024-09-14T04:21:23.357490+0000","last_fullsized":"2024-09-14T04:21:23.357490+0000","mapping_epoch":290,"log_start":"293'5700","ondisk_log_start":"293'5700","created":284,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:53.010206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182784,"num_objects":369,"num_object_clones":0,"num_object_copies":1107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14565,"num_write_kb":477888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":298,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.236287+0000","last_change":"2024-09-14T04:14:30.543098+0000","last_active":"2024-09-14T04:21:14.236287+0000","last_peered":"2024-09-14T04:21:14.236287+0000","last_clean":"2024-09-14T04:21:14.236287+0000","last_became_active":"2024-09-14T04:14:30.542773+0000","last_became_peered":"2024-09-14T04:14:30.542773+0000","last_unstale":"2024-09-14T04:21:14.236287+0000","last_undegraded":"2024-09-14T04:21:14.236287+0000","last_fullsized":"2024-09-14T04:21:14.236287+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:06:27.423486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":262,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235137+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:21:14.235137+0000","last_peered":"2024-09-14T04:21:14.235137+0000","last_clean":"2024-09-14T04:21:14.235137+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:21:14.235137+0000","last_undegraded":"2024-09-14T04:21:14.235137+0000","last_fullsized":"2024-09-14T04:21:14.235137+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"314'14315","reported_seq":14353,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:23.338410+0000","last_change":"2024-09-14T04:17:13.575564+0000","last_active":"2024-09-14T04:21:23.338410+0000","last_peered":"2024-09-14T04:21:23.338410+0000","last_clean":"2024-09-14T04:21:23.338410+0000","last_became_active":"2024-09-14T04:17:13.575303+0000","last_became_peered":"2024-09-14T04:17:13.575303+0000","last_unstale":"2024-09-14T04:21:23.338410+0000","last_undegraded":"2024-09-14T04:21:23.338410+0000","last_fullsized":"2024-09-14T04:21:23.338410+0000","mapping_epoch":284,"log_start":"293'5400","ondisk_log_start":"293'5400","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:02:07.463876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24576000,"num_objects":375,"num_object_clones":0,"num_object_copies":1125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14315,"num_write_kb":470080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":70,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.305649+0000","last_change":"2024-09-14T04:17:48.084067+0000","last_active":"2024-09-14T04:21:20.305649+0000","last_peered":"2024-09-14T04:21:20.305649+0000","last_clean":"2024-09-14T04:21:20.305649+0000","last_became_active":"2024-09-14T04:17:48.083584+0000","last_became_peered":"2024-09-14T04:17:48.083584+0000","last_unstale":"2024-09-14T04:21:20.305649+0000","last_undegraded":"2024-09-14T04:21:20.305649+0000","last_fullsized":"2024-09-14T04:21:20.305649+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:01.274210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":358,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.302585+0000","last_change":"2024-09-14T04:17:48.083353+0000","last_active":"2024-09-14T04:21:20.302585+0000","last_peered":"2024-09-14T04:21:20.302585+0000","last_clean":"2024-09-14T04:21:20.302585+0000","last_became_active":"2024-09-14T04:17:48.083032+0000","last_became_peered":"2024-09-14T04:17:48.083032+0000","last_unstale":"2024-09-14T04:21:20.302585+0000","last_undegraded":"2024-09-14T04:21:20.302585+0000","last_fullsized":"2024-09-14T04:21:20.302585+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:15:25.358824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"314'14335","reported_seq":14405,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:24.378351+0000","last_change":"2024-09-14T04:17:48.088284+0000","last_active":"2024-09-14T04:21:24.378351+0000","last_peered":"2024-09-14T04:21:24.378351+0000","last_clean":"2024-09-14T04:21:24.378351+0000","last_became_active":"2024-09-14T04:17:48.083349+0000","last_became_peered":"2024-09-14T04:17:48.083349+0000","last_unstale":"2024-09-14T04:21:24.378351+0000","last_undegraded":"2024-09-14T04:21:24.378351+0000","last_fullsized":"2024-09-14T04:21:24.378351+0000","mapping_epoch":290,"log_start":"294'6400","ondisk_log_start":"294'6400","created":284,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:56.489946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14335,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":75,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235560+0000","last_change":"2024-09-14T04:14:30.561646+0000","last_active":"2024-09-14T04:21:14.235560+0000","last_peered":"2024-09-14T04:21:14.235560+0000","last_clean":"2024-09-14T04:21:14.235560+0000","last_became_active":"2024-09-14T04:14:30.539174+0000","last_became_peered":"2024-09-14T04:14:30.539174+0000","last_unstale":"2024-09-14T04:21:14.235560+0000","last_undegraded":"2024-09-14T04:21:14.235560+0000","last_fullsized":"2024-09-14T04:21:14.235560+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:02:49.397620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":289,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.238949+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:21:14.238949+0000","last_peered":"2024-09-14T04:21:14.238949+0000","last_clean":"2024-09-14T04:21:14.238949+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:21:14.238949+0000","last_undegraded":"2024-09-14T04:21:14.238949+0000","last_fullsized":"2024-09-14T04:21:14.238949+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"314'14648","reported_seq":14719,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:24.719384+0000","last_change":"2024-09-14T04:17:13.574686+0000","last_active":"2024-09-14T04:21:24.719384+0000","last_peered":"2024-09-14T04:21:24.719384+0000","last_clean":"2024-09-14T04:21:24.719384+0000","last_became_active":"2024-09-14T04:17:13.574396+0000","last_became_peered":"2024-09-14T04:17:13.574396+0000","last_unstale":"2024-09-14T04:21:24.719384+0000","last_undegraded":"2024-09-14T04:21:24.719384+0000","last_fullsized":"2024-09-14T04:21:24.719384+0000","mapping_epoch":284,"log_start":"293'5200","ondisk_log_start":"293'5200","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":9448,"log_dups_size":0,"ondisk_log_size":9448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:06:20.930337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21889024,"num_objects":334,"num_object_clones":0,"num_object_copies":1002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14648,"num_write_kb":479424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":300,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235945+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:21:14.235945+0000","last_peered":"2024-09-14T04:21:14.235945+0000","last_clean":"2024-09-14T04:21:14.235945+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:21:14.235945+0000","last_undegraded":"2024-09-14T04:21:14.235945+0000","last_fullsized":"2024-09-14T04:21:14.235945+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":546,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290257+0000","last_change":"2024-09-14T04:13:37.165995+0000","last_active":"2024-09-14T04:21:20.290257+0000","last_peered":"2024-09-14T04:21:20.290257+0000","last_clean":"2024-09-14T04:21:20.290257+0000","last_became_active":"2024-09-14T04:13:37.165699+0000","last_became_peered":"2024-09-14T04:13:37.165699+0000","last_unstale":"2024-09-14T04:21:20.290257+0000","last_undegraded":"2024-09-14T04:21:20.290257+0000","last_fullsized":"2024-09-14T04:21:20.290257+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:21.792342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.4","version":"314'14122","reported_seq":14160,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:23.332824+0000","last_change":"2024-09-14T04:17:13.572270+0000","last_active":"2024-09-14T04:21:23.332824+0000","last_peered":"2024-09-14T04:21:23.332824+0000","last_clean":"2024-09-14T04:21:23.332824+0000","last_became_active":"2024-09-14T04:17:13.571986+0000","last_became_peered":"2024-09-14T04:17:13.571986+0000","last_unstale":"2024-09-14T04:21:23.332824+0000","last_undegraded":"2024-09-14T04:21:23.332824+0000","last_fullsized":"2024-09-14T04:21:23.332824+0000","mapping_epoch":284,"log_start":"293'5200","ondisk_log_start":"293'5200","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:58:04.909769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":388,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14122,"num_write_kb":464320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":491,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.288972+0000","last_change":"2024-09-14T04:13:37.173224+0000","last_active":"2024-09-14T04:21:20.288972+0000","last_peered":"2024-09-14T04:21:20.288972+0000","last_clean":"2024-09-14T04:21:20.288972+0000","last_became_active":"2024-09-14T04:13:37.158630+0000","last_became_peered":"2024-09-14T04:13:37.158630+0000","last_unstale":"2024-09-14T04:21:20.288972+0000","last_undegraded":"2024-09-14T04:21:20.288972+0000","last_fullsized":"2024-09-14T04:21:20.288972+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:41:25.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":327,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.289262+0000","last_change":"2024-09-14T04:17:48.083765+0000","last_active":"2024-09-14T04:21:20.289262+0000","last_peered":"2024-09-14T04:21:20.289262+0000","last_clean":"2024-09-14T04:21:20.289262+0000","last_became_active":"2024-09-14T04:17:48.083385+0000","last_became_peered":"2024-09-14T04:17:48.083385+0000","last_unstale":"2024-09-14T04:21:20.289262+0000","last_undegraded":"2024-09-14T04:21:20.289262+0000","last_fullsized":"2024-09-14T04:21:20.289262+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:13:49.268399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"314'14431","reported_seq":14469,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:23.309997+0000","last_change":"2024-09-14T04:17:13.572972+0000","last_active":"2024-09-14T04:21:23.309997+0000","last_peered":"2024-09-14T04:21:23.309997+0000","last_clean":"2024-09-14T04:21:23.309997+0000","last_became_active":"2024-09-14T04:17:13.572619+0000","last_became_peered":"2024-09-14T04:17:13.572619+0000","last_unstale":"2024-09-14T04:21:23.309997+0000","last_undegraded":"2024-09-14T04:21:23.309997+0000","last_fullsized":"2024-09-14T04:21:23.309997+0000","mapping_epoch":284,"log_start":"293'5600","ondisk_log_start":"293'5600","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8831,"log_dups_size":0,"ondisk_log_size":8831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:07:11.298450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":423,"num_object_clones":0,"num_object_copies":1269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14431,"num_write_kb":475328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":300,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.239068+0000","last_change":"2024-09-14T04:13:37.166773+0000","last_active":"2024-09-14T04:21:14.239068+0000","last_peered":"2024-09-14T04:21:14.239068+0000","last_clean":"2024-09-14T04:21:14.239068+0000","last_became_active":"2024-09-14T04:13:37.166465+0000","last_became_peered":"2024-09-14T04:13:37.166465+0000","last_unstale":"2024-09-14T04:21:14.239068+0000","last_undegraded":"2024-09-14T04:21:14.239068+0000","last_fullsized":"2024-09-14T04:21:14.239068+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:57:18.786263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":318,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286386+0000","last_change":"2024-09-14T04:17:48.084357+0000","last_active":"2024-09-14T04:21:20.286386+0000","last_peered":"2024-09-14T04:21:20.286386+0000","last_clean":"2024-09-14T04:21:20.286386+0000","last_became_active":"2024-09-14T04:17:48.084034+0000","last_became_peered":"2024-09-14T04:17:48.084034+0000","last_unstale":"2024-09-14T04:21:20.286386+0000","last_undegraded":"2024-09-14T04:21:20.286386+0000","last_fullsized":"2024-09-14T04:21:20.286386+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:42:00.818438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"314'14410","reported_seq":14482,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:23.580215+0000","last_change":"2024-09-14T04:17:13.575249+0000","last_active":"2024-09-14T04:21:23.580215+0000","last_peered":"2024-09-14T04:21:23.580215+0000","last_clean":"2024-09-14T04:21:23.580215+0000","last_became_active":"2024-09-14T04:17:13.574960+0000","last_became_peered":"2024-09-14T04:17:13.574960+0000","last_unstale":"2024-09-14T04:21:23.580215+0000","last_undegraded":"2024-09-14T04:21:23.580215+0000","last_fullsized":"2024-09-14T04:21:23.580215+0000","mapping_epoch":284,"log_start":"294'6000","ondisk_log_start":"294'6000","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8410,"log_dups_size":0,"ondisk_log_size":8410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:31.199701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14410,"num_write_kb":473856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":72,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235538+0000","last_change":"2024-09-14T04:13:37.158807+0000","last_active":"2024-09-14T04:21:14.235538+0000","last_peered":"2024-09-14T04:21:14.235538+0000","last_clean":"2024-09-14T04:21:14.235538+0000","last_became_active":"2024-09-14T04:13:37.158475+0000","last_became_peered":"2024-09-14T04:13:37.158475+0000","last_unstale":"2024-09-14T04:21:14.235538+0000","last_undegraded":"2024-09-14T04:21:14.235538+0000","last_fullsized":"2024-09-14T04:21:14.235538+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:09.005093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":182,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.238901+0000","last_change":"2024-09-14T04:17:48.081945+0000","last_active":"2024-09-14T04:21:14.238901+0000","last_peered":"2024-09-14T04:21:14.238901+0000","last_clean":"2024-09-14T04:21:14.238901+0000","last_became_active":"2024-09-14T04:17:48.081510+0000","last_became_peered":"2024-09-14T04:17:48.081510+0000","last_unstale":"2024-09-14T04:21:14.238901+0000","last_undegraded":"2024-09-14T04:21:14.238901+0000","last_fullsized":"2024-09-14T04:21:14.238901+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:19:48.331487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"314'14394","reported_seq":14466,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:23.594926+0000","last_change":"2024-09-14T04:17:13.572496+0000","last_active":"2024-09-14T04:21:23.594926+0000","last_peered":"2024-09-14T04:21:23.594926+0000","last_clean":"2024-09-14T04:21:23.594926+0000","last_became_active":"2024-09-14T04:17:13.572182+0000","last_became_peered":"2024-09-14T04:17:13.572182+0000","last_unstale":"2024-09-14T04:21:23.594926+0000","last_undegraded":"2024-09-14T04:21:23.594926+0000","last_fullsized":"2024-09-14T04:21:23.594926+0000","mapping_epoch":284,"log_start":"294'6000","ondisk_log_start":"294'6000","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:43:40.019232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26869760,"num_objects":410,"num_object_clones":0,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14394,"num_write_kb":473728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":254,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.293232+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:21:20.293232+0000","last_peered":"2024-09-14T04:21:20.293232+0000","last_clean":"2024-09-14T04:21:20.293232+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:21:20.293232+0000","last_undegraded":"2024-09-14T04:21:20.293232+0000","last_fullsized":"2024-09-14T04:21:20.293232+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":491,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.304442+0000","last_change":"2024-09-14T04:13:37.168484+0000","last_active":"2024-09-14T04:21:20.304442+0000","last_peered":"2024-09-14T04:21:20.304442+0000","last_clean":"2024-09-14T04:21:20.304442+0000","last_became_active":"2024-09-14T04:13:37.168136+0000","last_became_peered":"2024-09-14T04:13:37.168136+0000","last_unstale":"2024-09-14T04:21:20.304442+0000","last_undegraded":"2024-09-14T04:21:20.304442+0000","last_fullsized":"2024-09-14T04:21:20.304442+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:13.681730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"314'14375","reported_seq":14414,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:23.321102+0000","last_change":"2024-09-14T04:17:13.572747+0000","last_active":"2024-09-14T04:21:23.321102+0000","last_peered":"2024-09-14T04:21:23.321102+0000","last_clean":"2024-09-14T04:21:23.321102+0000","last_became_active":"2024-09-14T04:17:13.572451+0000","last_became_peered":"2024-09-14T04:17:13.572451+0000","last_unstale":"2024-09-14T04:21:23.321102+0000","last_undegraded":"2024-09-14T04:21:23.321102+0000","last_fullsized":"2024-09-14T04:21:23.321102+0000","mapping_epoch":284,"log_start":"294'6200","ondisk_log_start":"294'6200","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8175,"log_dups_size":0,"ondisk_log_size":8175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:00:57.163738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476568,"num_objects":405,"num_object_clones":0,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14375,"num_write_kb":472897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":261,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.295361+0000","last_change":"2024-09-14T04:14:30.540124+0000","last_active":"2024-09-14T04:21:20.295361+0000","last_peered":"2024-09-14T04:21:20.295361+0000","last_clean":"2024-09-14T04:21:20.295361+0000","last_became_active":"2024-09-14T04:14:30.539704+0000","last_became_peered":"2024-09-14T04:14:30.539704+0000","last_unstale":"2024-09-14T04:21:20.295361+0000","last_undegraded":"2024-09-14T04:21:20.295361+0000","last_fullsized":"2024-09-14T04:21:20.295361+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:52:13.421617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":293,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235360+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:21:14.235360+0000","last_peered":"2024-09-14T04:21:14.235360+0000","last_clean":"2024-09-14T04:21:14.235360+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:21:14.235360+0000","last_undegraded":"2024-09-14T04:21:14.235360+0000","last_fullsized":"2024-09-14T04:21:14.235360+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":254,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290970+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:21:20.290970+0000","last_peered":"2024-09-14T04:21:20.290970+0000","last_clean":"2024-09-14T04:21:20.290970+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:21:20.290970+0000","last_undegraded":"2024-09-14T04:21:20.290970+0000","last_fullsized":"2024-09-14T04:21:20.290970+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":282,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.239664+0000","last_change":"2024-09-14T04:13:37.166354+0000","last_active":"2024-09-14T04:21:14.239664+0000","last_peered":"2024-09-14T04:21:14.239664+0000","last_clean":"2024-09-14T04:21:14.239664+0000","last_became_active":"2024-09-14T04:13:37.166057+0000","last_became_peered":"2024-09-14T04:13:37.166057+0000","last_unstale":"2024-09-14T04:21:14.239664+0000","last_undegraded":"2024-09-14T04:21:14.239664+0000","last_fullsized":"2024-09-14T04:21:14.239664+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:40.817585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"314'14317","reported_seq":14389,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:24.735387+0000","last_change":"2024-09-14T04:17:13.570675+0000","last_active":"2024-09-14T04:21:24.735387+0000","last_peered":"2024-09-14T04:21:24.735387+0000","last_clean":"2024-09-14T04:21:24.735387+0000","last_became_active":"2024-09-14T04:17:13.569382+0000","last_became_peered":"2024-09-14T04:17:13.569382+0000","last_unstale":"2024-09-14T04:21:24.735387+0000","last_undegraded":"2024-09-14T04:21:24.735387+0000","last_fullsized":"2024-09-14T04:21:24.735387+0000","mapping_epoch":284,"log_start":"292'4900","ondisk_log_start":"292'4900","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":9417,"log_dups_size":0,"ondisk_log_size":9417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:10:19.648738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20250624,"num_objects":309,"num_object_clones":0,"num_object_copies":927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14317,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":204,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.302288+0000","last_change":"2024-09-14T04:14:30.539202+0000","last_active":"2024-09-14T04:21:20.302288+0000","last_peered":"2024-09-14T04:21:20.302288+0000","last_clean":"2024-09-14T04:21:20.302288+0000","last_became_active":"2024-09-14T04:14:30.538884+0000","last_became_peered":"2024-09-14T04:14:30.538884+0000","last_unstale":"2024-09-14T04:21:20.302288+0000","last_undegraded":"2024-09-14T04:21:20.302288+0000","last_fullsized":"2024-09-14T04:21:20.302288+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:39.511239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.289856+0000","last_change":"2024-09-14T04:13:37.165079+0000","last_active":"2024-09-14T04:21:20.289856+0000","last_peered":"2024-09-14T04:21:20.289856+0000","last_clean":"2024-09-14T04:21:20.289856+0000","last_became_active":"2024-09-14T04:13:37.164753+0000","last_became_peered":"2024-09-14T04:13:37.164753+0000","last_unstale":"2024-09-14T04:21:20.289856+0000","last_undegraded":"2024-09-14T04:21:20.289856+0000","last_fullsized":"2024-09-14T04:21:20.289856+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:50:32.658239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":523,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290602+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:21:20.290602+0000","last_peered":"2024-09-14T04:21:20.290602+0000","last_clean":"2024-09-14T04:21:20.290602+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:21:20.290602+0000","last_undegraded":"2024-09-14T04:21:20.290602+0000","last_fullsized":"2024-09-14T04:21:20.290602+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"314'14537","reported_seq":14575,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:23.339202+0000","last_change":"2024-09-14T04:17:13.572985+0000","last_active":"2024-09-14T04:21:23.339202+0000","last_peered":"2024-09-14T04:21:23.339202+0000","last_clean":"2024-09-14T04:21:23.339202+0000","last_became_active":"2024-09-14T04:17:13.572399+0000","last_became_peered":"2024-09-14T04:17:13.572399+0000","last_unstale":"2024-09-14T04:21:23.339202+0000","last_undegraded":"2024-09-14T04:21:23.339202+0000","last_fullsized":"2024-09-14T04:21:23.339202+0000","mapping_epoch":284,"log_start":"293'5700","ondisk_log_start":"293'5700","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8837,"log_dups_size":0,"ondisk_log_size":8837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:58.922138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":421,"num_object_clones":0,"num_object_copies":1263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14537,"num_write_kb":478656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":517,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.288919+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:21:20.288919+0000","last_peered":"2024-09-14T04:21:20.288919+0000","last_clean":"2024-09-14T04:21:20.288919+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:21:20.288919+0000","last_undegraded":"2024-09-14T04:21:20.288919+0000","last_fullsized":"2024-09-14T04:21:20.288919+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":478,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.302481+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:21:20.302481+0000","last_peered":"2024-09-14T04:21:20.302481+0000","last_clean":"2024-09-14T04:21:20.302481+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:21:20.302481+0000","last_undegraded":"2024-09-14T04:21:20.302481+0000","last_fullsized":"2024-09-14T04:21:20.302481+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":248,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.293602+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:21:20.293602+0000","last_peered":"2024-09-14T04:21:20.293602+0000","last_clean":"2024-09-14T04:21:20.293602+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:21:20.293602+0000","last_undegraded":"2024-09-14T04:21:20.293602+0000","last_fullsized":"2024-09-14T04:21:20.293602+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.c","version":"314'14596","reported_seq":14668,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:24.746296+0000","last_change":"2024-09-14T04:17:13.570232+0000","last_active":"2024-09-14T04:21:24.746296+0000","last_peered":"2024-09-14T04:21:24.746296+0000","last_clean":"2024-09-14T04:21:24.746296+0000","last_became_active":"2024-09-14T04:17:13.569953+0000","last_became_peered":"2024-09-14T04:17:13.569953+0000","last_unstale":"2024-09-14T04:21:24.746296+0000","last_undegraded":"2024-09-14T04:21:24.746296+0000","last_fullsized":"2024-09-14T04:21:24.746296+0000","mapping_epoch":284,"log_start":"293'5200","ondisk_log_start":"293'5200","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:09:54.113872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":290,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14596,"num_write_kb":476352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":294,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235312+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:21:14.235312+0000","last_peered":"2024-09-14T04:21:14.235312+0000","last_clean":"2024-09-14T04:21:14.235312+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:21:14.235312+0000","last_undegraded":"2024-09-14T04:21:14.235312+0000","last_fullsized":"2024-09-14T04:21:14.235312+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":273,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290573+0000","last_change":"2024-09-14T04:17:48.083263+0000","last_active":"2024-09-14T04:21:20.290573+0000","last_peered":"2024-09-14T04:21:20.290573+0000","last_clean":"2024-09-14T04:21:20.290573+0000","last_became_active":"2024-09-14T04:17:48.082791+0000","last_became_peered":"2024-09-14T04:17:48.082791+0000","last_unstale":"2024-09-14T04:21:20.290573+0000","last_undegraded":"2024-09-14T04:21:20.290573+0000","last_fullsized":"2024-09-14T04:21:20.290573+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:20:46.928898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":503,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.302402+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:21:20.302402+0000","last_peered":"2024-09-14T04:21:20.302402+0000","last_clean":"2024-09-14T04:21:20.302402+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:21:20.302402+0000","last_undegraded":"2024-09-14T04:21:20.302402+0000","last_fullsized":"2024-09-14T04:21:20.302402+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"314'14316","reported_seq":14388,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:22.717130+0000","last_change":"2024-09-14T04:17:13.575911+0000","last_active":"2024-09-14T04:21:22.717130+0000","last_peered":"2024-09-14T04:21:22.717130+0000","last_clean":"2024-09-14T04:21:22.717130+0000","last_became_active":"2024-09-14T04:17:13.575627+0000","last_became_peered":"2024-09-14T04:17:13.575627+0000","last_unstale":"2024-09-14T04:21:22.717130+0000","last_undegraded":"2024-09-14T04:21:22.717130+0000","last_fullsized":"2024-09-14T04:21:22.717130+0000","mapping_epoch":284,"log_start":"292'4300","ondisk_log_start":"292'4300","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:32:39.359481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28966912,"num_objects":442,"num_object_clones":0,"num_object_copies":1326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14316,"num_write_kb":472256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":317,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235645+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:21:14.235645+0000","last_peered":"2024-09-14T04:21:14.235645+0000","last_clean":"2024-09-14T04:21:14.235645+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:21:14.235645+0000","last_undegraded":"2024-09-14T04:21:14.235645+0000","last_fullsized":"2024-09-14T04:21:14.235645+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":517,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.302482+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:21:20.302482+0000","last_peered":"2024-09-14T04:21:20.302482+0000","last_clean":"2024-09-14T04:21:20.302482+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:21:20.302482+0000","last_undegraded":"2024-09-14T04:21:20.302482+0000","last_fullsized":"2024-09-14T04:21:20.302482+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":481,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.302945+0000","last_change":"2024-09-14T04:13:37.163670+0000","last_active":"2024-09-14T04:21:20.302945+0000","last_peered":"2024-09-14T04:21:20.302945+0000","last_clean":"2024-09-14T04:21:20.302945+0000","last_became_active":"2024-09-14T04:13:37.163234+0000","last_became_peered":"2024-09-14T04:13:37.163234+0000","last_unstale":"2024-09-14T04:21:20.302945+0000","last_undegraded":"2024-09-14T04:21:20.302945+0000","last_fullsized":"2024-09-14T04:21:20.302945+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:53:34.767208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"314'14104","reported_seq":14173,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:24.406218+0000","last_change":"2024-09-14T04:17:13.573945+0000","last_active":"2024-09-14T04:21:24.406218+0000","last_peered":"2024-09-14T04:21:24.406218+0000","last_clean":"2024-09-14T04:21:24.406218+0000","last_became_active":"2024-09-14T04:17:13.573636+0000","last_became_peered":"2024-09-14T04:17:13.573636+0000","last_unstale":"2024-09-14T04:21:24.406218+0000","last_undegraded":"2024-09-14T04:21:24.406218+0000","last_fullsized":"2024-09-14T04:21:24.406218+0000","mapping_epoch":284,"log_start":"294'6200","ondisk_log_start":"294'6200","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":7904,"log_dups_size":0,"ondisk_log_size":7904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:34:37.026824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25821184,"num_objects":394,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14104,"num_write_kb":463936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":478,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.302635+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:21:20.302635+0000","last_peered":"2024-09-14T04:21:20.302635+0000","last_clean":"2024-09-14T04:21:20.302635+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:21:20.302635+0000","last_undegraded":"2024-09-14T04:21:20.302635+0000","last_fullsized":"2024-09-14T04:21:20.302635+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":311,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.246136+0000","last_change":"2024-09-14T04:13:37.162059+0000","last_active":"2024-09-14T04:21:14.246136+0000","last_peered":"2024-09-14T04:21:14.246136+0000","last_clean":"2024-09-14T04:21:14.246136+0000","last_became_active":"2024-09-14T04:13:37.161728+0000","last_became_peered":"2024-09-14T04:13:37.161728+0000","last_unstale":"2024-09-14T04:21:14.246136+0000","last_undegraded":"2024-09-14T04:21:14.246136+0000","last_fullsized":"2024-09-14T04:21:14.246136+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:39:59.671661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":501,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290339+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:21:20.290339+0000","last_peered":"2024-09-14T04:21:20.290339+0000","last_clean":"2024-09-14T04:21:20.290339+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:21:20.290339+0000","last_undegraded":"2024-09-14T04:21:20.290339+0000","last_fullsized":"2024-09-14T04:21:20.290339+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"314'14084","reported_seq":14156,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:24.375159+0000","last_change":"2024-09-14T04:17:13.576627+0000","last_active":"2024-09-14T04:21:24.375159+0000","last_peered":"2024-09-14T04:21:24.375159+0000","last_clean":"2024-09-14T04:21:24.375159+0000","last_became_active":"2024-09-14T04:17:13.576099+0000","last_became_peered":"2024-09-14T04:17:13.576099+0000","last_unstale":"2024-09-14T04:21:24.375159+0000","last_undegraded":"2024-09-14T04:21:24.375159+0000","last_fullsized":"2024-09-14T04:21:24.375159+0000","mapping_epoch":284,"log_start":"294'6100","ondisk_log_start":"294'6100","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":7984,"log_dups_size":0,"ondisk_log_size":7984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:31:19.819415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020096,"num_objects":336,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14084,"num_write_kb":461440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":469,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290031+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:21:20.290031+0000","last_peered":"2024-09-14T04:21:20.290031+0000","last_clean":"2024-09-14T04:21:20.290031+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:21:20.290031+0000","last_undegraded":"2024-09-14T04:21:20.290031+0000","last_fullsized":"2024-09-14T04:21:20.290031+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":491,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.307113+0000","last_change":"2024-09-14T04:13:37.169517+0000","last_active":"2024-09-14T04:21:20.307113+0000","last_peered":"2024-09-14T04:21:20.307113+0000","last_clean":"2024-09-14T04:21:20.307113+0000","last_became_active":"2024-09-14T04:13:37.169027+0000","last_became_peered":"2024-09-14T04:13:37.169027+0000","last_unstale":"2024-09-14T04:21:20.307113+0000","last_undegraded":"2024-09-14T04:21:20.307113+0000","last_fullsized":"2024-09-14T04:21:20.307113+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:49:57.369201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":125,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235055+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:21:14.235055+0000","last_peered":"2024-09-14T04:21:14.235055+0000","last_clean":"2024-09-14T04:21:14.235055+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:21:14.235055+0000","last_undegraded":"2024-09-14T04:21:14.235055+0000","last_fullsized":"2024-09-14T04:21:14.235055+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"314'14329","reported_seq":14400,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:22.735509+0000","last_change":"2024-09-14T04:17:13.577599+0000","last_active":"2024-09-14T04:21:22.735509+0000","last_peered":"2024-09-14T04:21:22.735509+0000","last_clean":"2024-09-14T04:21:22.735509+0000","last_became_active":"2024-09-14T04:17:13.577330+0000","last_became_peered":"2024-09-14T04:17:13.577330+0000","last_unstale":"2024-09-14T04:21:22.735509+0000","last_undegraded":"2024-09-14T04:21:22.735509+0000","last_fullsized":"2024-09-14T04:21:22.735509+0000","mapping_epoch":284,"log_start":"292'4300","ondisk_log_start":"292'4300","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:59:12.058934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29032448,"num_objects":443,"num_object_clones":0,"num_object_copies":1329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14329,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":537,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290833+0000","last_change":"2024-09-14T04:13:37.178106+0000","last_active":"2024-09-14T04:21:20.290833+0000","last_peered":"2024-09-14T04:21:20.290833+0000","last_clean":"2024-09-14T04:21:20.290833+0000","last_became_active":"2024-09-14T04:13:37.167009+0000","last_became_peered":"2024-09-14T04:13:37.167009+0000","last_unstale":"2024-09-14T04:21:20.290833+0000","last_undegraded":"2024-09-14T04:21:20.290833+0000","last_fullsized":"2024-09-14T04:21:20.290833+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:28:37.700664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":292,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.245720+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:21:14.245720+0000","last_peered":"2024-09-14T04:21:14.245720+0000","last_clean":"2024-09-14T04:21:14.245720+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:21:14.245720+0000","last_undegraded":"2024-09-14T04:21:14.245720+0000","last_fullsized":"2024-09-14T04:21:14.245720+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":128,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235160+0000","last_change":"2024-09-14T04:17:48.080621+0000","last_active":"2024-09-14T04:21:14.235160+0000","last_peered":"2024-09-14T04:21:14.235160+0000","last_clean":"2024-09-14T04:21:14.235160+0000","last_became_active":"2024-09-14T04:17:48.080300+0000","last_became_peered":"2024-09-14T04:17:48.080300+0000","last_unstale":"2024-09-14T04:21:14.235160+0000","last_undegraded":"2024-09-14T04:21:14.235160+0000","last_fullsized":"2024-09-14T04:21:14.235160+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:14:47.756124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":72,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235892+0000","last_change":"2024-09-14T04:13:37.162516+0000","last_active":"2024-09-14T04:21:14.235892+0000","last_peered":"2024-09-14T04:21:14.235892+0000","last_clean":"2024-09-14T04:21:14.235892+0000","last_became_active":"2024-09-14T04:13:37.162248+0000","last_became_peered":"2024-09-14T04:13:37.162248+0000","last_unstale":"2024-09-14T04:21:14.235892+0000","last_undegraded":"2024-09-14T04:21:14.235892+0000","last_fullsized":"2024-09-14T04:21:14.235892+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:41.935908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":181,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.245742+0000","last_change":"2024-09-14T04:17:48.080930+0000","last_active":"2024-09-14T04:21:14.245742+0000","last_peered":"2024-09-14T04:21:14.245742+0000","last_clean":"2024-09-14T04:21:14.245742+0000","last_became_active":"2024-09-14T04:17:48.080575+0000","last_became_peered":"2024-09-14T04:17:48.080575+0000","last_unstale":"2024-09-14T04:21:14.245742+0000","last_undegraded":"2024-09-14T04:21:14.245742+0000","last_fullsized":"2024-09-14T04:21:14.245742+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:14:19.265519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":476,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.302821+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:21:20.302821+0000","last_peered":"2024-09-14T04:21:20.302821+0000","last_clean":"2024-09-14T04:21:20.302821+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:21:20.302821+0000","last_undegraded":"2024-09-14T04:21:20.302821+0000","last_fullsized":"2024-09-14T04:21:20.302821+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":129,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.289530+0000","last_change":"2024-09-14T04:14:30.539928+0000","last_active":"2024-09-14T04:21:20.289530+0000","last_peered":"2024-09-14T04:21:20.289530+0000","last_clean":"2024-09-14T04:21:20.289530+0000","last_became_active":"2024-09-14T04:14:30.539572+0000","last_became_peered":"2024-09-14T04:14:30.539572+0000","last_unstale":"2024-09-14T04:21:20.289530+0000","last_undegraded":"2024-09-14T04:21:20.289530+0000","last_fullsized":"2024-09-14T04:21:20.289530+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:55:42.675243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":280,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235724+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:21:14.235724+0000","last_peered":"2024-09-14T04:21:14.235724+0000","last_clean":"2024-09-14T04:21:14.235724+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:21:14.235724+0000","last_undegraded":"2024-09-14T04:21:14.235724+0000","last_fullsized":"2024-09-14T04:21:14.235724+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":428,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.294038+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:21:20.294038+0000","last_peered":"2024-09-14T04:21:20.294038+0000","last_clean":"2024-09-14T04:21:20.294038+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:21:20.294038+0000","last_undegraded":"2024-09-14T04:21:20.294038+0000","last_fullsized":"2024-09-14T04:21:20.294038+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":524,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290220+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:21:20.290220+0000","last_peered":"2024-09-14T04:21:20.290220+0000","last_clean":"2024-09-14T04:21:20.290220+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:21:20.290220+0000","last_undegraded":"2024-09-14T04:21:20.290220+0000","last_fullsized":"2024-09-14T04:21:20.290220+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":440,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.298285+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:21:20.298285+0000","last_peered":"2024-09-14T04:21:20.298285+0000","last_clean":"2024-09-14T04:21:20.298285+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:21:20.298285+0000","last_undegraded":"2024-09-14T04:21:20.298285+0000","last_fullsized":"2024-09-14T04:21:20.298285+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":289,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.245644+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:21:14.245644+0000","last_peered":"2024-09-14T04:21:14.245644+0000","last_clean":"2024-09-14T04:21:14.245644+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:21:14.245644+0000","last_undegraded":"2024-09-14T04:21:14.245644+0000","last_fullsized":"2024-09-14T04:21:14.245644+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":253,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.293133+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:21:20.293133+0000","last_peered":"2024-09-14T04:21:20.293133+0000","last_clean":"2024-09-14T04:21:20.293133+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:21:20.293133+0000","last_undegraded":"2024-09-14T04:21:20.293133+0000","last_fullsized":"2024-09-14T04:21:20.293133+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":527,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290675+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:21:20.290675+0000","last_peered":"2024-09-14T04:21:20.290675+0000","last_clean":"2024-09-14T04:21:20.290675+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:21:20.290675+0000","last_undegraded":"2024-09-14T04:21:20.290675+0000","last_fullsized":"2024-09-14T04:21:20.290675+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":490,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.302660+0000","last_change":"2024-09-14T04:13:37.163535+0000","last_active":"2024-09-14T04:21:20.302660+0000","last_peered":"2024-09-14T04:21:20.302660+0000","last_clean":"2024-09-14T04:21:20.302660+0000","last_became_active":"2024-09-14T04:13:37.163164+0000","last_became_peered":"2024-09-14T04:13:37.163164+0000","last_unstale":"2024-09-14T04:21:20.302660+0000","last_undegraded":"2024-09-14T04:21:20.302660+0000","last_fullsized":"2024-09-14T04:21:20.302660+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:13:25.324074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":260,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.289409+0000","last_change":"2024-09-14T04:13:37.162587+0000","last_active":"2024-09-14T04:21:20.289409+0000","last_peered":"2024-09-14T04:21:20.289409+0000","last_clean":"2024-09-14T04:21:20.289409+0000","last_became_active":"2024-09-14T04:13:37.162219+0000","last_became_peered":"2024-09-14T04:13:37.162219+0000","last_unstale":"2024-09-14T04:21:20.289409+0000","last_undegraded":"2024-09-14T04:21:20.289409+0000","last_fullsized":"2024-09-14T04:21:20.289409+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:52:47.043189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":72,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235752+0000","last_change":"2024-09-14T04:13:37.164992+0000","last_active":"2024-09-14T04:21:14.235752+0000","last_peered":"2024-09-14T04:21:14.235752+0000","last_clean":"2024-09-14T04:21:14.235752+0000","last_became_active":"2024-09-14T04:13:37.164654+0000","last_became_peered":"2024-09-14T04:13:37.164654+0000","last_unstale":"2024-09-14T04:21:14.235752+0000","last_undegraded":"2024-09-14T04:21:14.235752+0000","last_fullsized":"2024-09-14T04:21:14.235752+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:49.248470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":519,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286123+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:21:20.286123+0000","last_peered":"2024-09-14T04:21:20.286123+0000","last_clean":"2024-09-14T04:21:20.286123+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:21:20.286123+0000","last_undegraded":"2024-09-14T04:21:20.286123+0000","last_fullsized":"2024-09-14T04:21:20.286123+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":516,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.290419+0000","last_change":"2024-09-14T04:13:37.166866+0000","last_active":"2024-09-14T04:21:20.290419+0000","last_peered":"2024-09-14T04:21:20.290419+0000","last_clean":"2024-09-14T04:21:20.290419+0000","last_became_active":"2024-09-14T04:13:37.166453+0000","last_became_peered":"2024-09-14T04:13:37.166453+0000","last_unstale":"2024-09-14T04:21:20.290419+0000","last_undegraded":"2024-09-14T04:21:20.290419+0000","last_fullsized":"2024-09-14T04:21:20.290419+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:34:59.229028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":344,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.288893+0000","last_change":"2024-09-14T04:17:48.085379+0000","last_active":"2024-09-14T04:21:20.288893+0000","last_peered":"2024-09-14T04:21:20.288893+0000","last_clean":"2024-09-14T04:21:20.288893+0000","last_became_active":"2024-09-14T04:17:48.084944+0000","last_became_peered":"2024-09-14T04:17:48.084944+0000","last_unstale":"2024-09-14T04:21:20.288893+0000","last_undegraded":"2024-09-14T04:21:20.288893+0000","last_fullsized":"2024-09-14T04:21:20.288893+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:43:26.386934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":310,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235875+0000","last_change":"2024-09-14T04:13:37.174369+0000","last_active":"2024-09-14T04:21:14.235875+0000","last_peered":"2024-09-14T04:21:14.235875+0000","last_clean":"2024-09-14T04:21:14.235875+0000","last_became_active":"2024-09-14T04:13:37.165108+0000","last_became_peered":"2024-09-14T04:13:37.165108+0000","last_unstale":"2024-09-14T04:21:14.235875+0000","last_undegraded":"2024-09-14T04:21:14.235875+0000","last_fullsized":"2024-09-14T04:21:14.235875+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:43.400463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":344,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.291097+0000","last_change":"2024-09-14T04:17:48.081862+0000","last_active":"2024-09-14T04:21:20.291097+0000","last_peered":"2024-09-14T04:21:20.291097+0000","last_clean":"2024-09-14T04:21:20.291097+0000","last_became_active":"2024-09-14T04:17:48.081095+0000","last_became_peered":"2024-09-14T04:17:48.081095+0000","last_unstale":"2024-09-14T04:21:20.291097+0000","last_undegraded":"2024-09-14T04:21:20.291097+0000","last_fullsized":"2024-09-14T04:21:20.291097+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:04:07.654057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":293,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235646+0000","last_change":"2024-09-14T04:13:37.168004+0000","last_active":"2024-09-14T04:21:14.235646+0000","last_peered":"2024-09-14T04:21:14.235646+0000","last_clean":"2024-09-14T04:21:14.235646+0000","last_became_active":"2024-09-14T04:13:37.159695+0000","last_became_peered":"2024-09-14T04:13:37.159695+0000","last_unstale":"2024-09-14T04:21:14.235646+0000","last_undegraded":"2024-09-14T04:21:14.235646+0000","last_fullsized":"2024-09-14T04:21:14.235646+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:18:40.779932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":253,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.293943+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:21:20.293943+0000","last_peered":"2024-09-14T04:21:20.293943+0000","last_clean":"2024-09-14T04:21:20.293943+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:21:20.293943+0000","last_undegraded":"2024-09-14T04:21:20.293943+0000","last_fullsized":"2024-09-14T04:21:20.293943+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":254,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.289692+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:21:20.289692+0000","last_peered":"2024-09-14T04:21:20.289692+0000","last_clean":"2024-09-14T04:21:20.289692+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:21:20.289692+0000","last_undegraded":"2024-09-14T04:21:20.289692+0000","last_fullsized":"2024-09-14T04:21:20.289692+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":72,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-14T04:21:14.235443+0000","last_change":"2024-09-14T04:13:37.165840+0000","last_active":"2024-09-14T04:21:14.235443+0000","last_peered":"2024-09-14T04:21:14.235443+0000","last_clean":"2024-09-14T04:21:14.235443+0000","last_became_active":"2024-09-14T04:13:37.165508+0000","last_became_peered":"2024-09-14T04:13:37.165508+0000","last_unstale":"2024-09-14T04:21:14.235443+0000","last_undegraded":"2024-09-14T04:21:14.235443+0000","last_fullsized":"2024-09-14T04:21:14.235443+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:24:02.783651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":515,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.289856+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:21:20.289856+0000","last_peered":"2024-09-14T04:21:20.289856+0000","last_clean":"2024-09-14T04:21:20.289856+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:21:20.289856+0000","last_undegraded":"2024-09-14T04:21:20.289856+0000","last_fullsized":"2024-09-14T04:21:20.289856+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":398131224,"num_objects":6076,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229878,"num_write_kb":7550465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1226911744,"data_stored":1226899528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141478,"ondisk_log_size":141478,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4834,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430692,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1380640,"kb_used_data":185048,"kb_used_omap":1,"kb_used_meta":1195582,"kb_avail":92364512,"statfs":{"total":95995035648,"available":94581260288,"internally_reserved":0,"allocated":189489152,"data_stored":188054621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1224276312},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626921,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1104848,"kb_used_data":195016,"kb_used_omap":1,"kb_used_meta":909822,"kb_avail":92640304,"statfs":{"total":95995035648,"available":94863671296,"internally_reserved":0,"allocated":199696384,"data_stored":198277771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931658072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823149,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1248256,"kb_used_data":54584,"kb_used_omap":1,"kb_used_meta":1193662,"kb_avail":92496896,"statfs":{"total":95995035648,"available":94716821504,"internally_reserved":0,"allocated":55894016,"data_stored":54486754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1222310232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917231,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425680,"kb_used_data":91528,"kb_used_omap":6,"kb_used_meta":334137,"kb_avail":93319472,"statfs":{"total":95995035648,"available":95559139328,"internally_reserved":0,"allocated":93724672,"data_stored":92305572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":342157311},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552530,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1063424,"kb_used_data":189304,"kb_used_omap":4,"kb_used_meta":874107,"kb_avail":92681728,"statfs":{"total":95995035648,"available":94906089472,"internally_reserved":0,"allocated":193847296,"data_stored":192440115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":895086013},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435571,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1112128,"kb_used_data":283960,"kb_used_omap":6,"kb_used_meta":827897,"kb_avail":92633024,"statfs":{"total":95995035648,"available":94856216576,"internally_reserved":0,"allocated":290775040,"data_stored":289366935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":847766989},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828041,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1051652,"kb_used_data":138300,"kb_used_omap":5,"kb_used_meta":913338,"kb_avail":92693500,"statfs":{"total":95995035648,"available":94918144000,"internally_reserved":0,"allocated":141619200,"data_stored":140207876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":935258351},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":80,"seq":343597384029,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1214724,"kb_used_data":140220,"kb_used_omap":5,"kb_used_meta":1074490,"kb_avail":92530428,"statfs":{"total":95995035648,"available":94751158272,"internally_reserved":0,"allocated":143585280,"data_stored":142174750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":1100277820},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125894656,"data_stored":125894656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129765376,"data_stored":129761304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284823552,"data_stored":284819480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":186253312,"data_stored":186253312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":82903040,"data_stored":82903040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":51118080,"data_stored":51118080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":185139200,"data_stored":185139200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":181014528,"data_stored":181010456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:21:28.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:21:28.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:28.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:28.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:28.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:29.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:29.072 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:29.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:29.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:29.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:29.226 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:29.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:29.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:29.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:29.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:29.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:29 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:21:29.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:29 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:21:29.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:29 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2946810147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:21:29.455 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:29 smithi114 ceph-mon[103029]: osdmap e315: 8 total, 8 up, 8 in 2024-09-14T04:21:29.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:29 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:21:29.455 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:29 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:21:29.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:29 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2946810147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:21:29.456 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:29 smithi114 ceph-mon[106969]: osdmap e315: 8 total, 8 up, 8 in 2024-09-14T04:21:29.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:29.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:29.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:29.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:29.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:29.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:29 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:21:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:29 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:21:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:29 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2946810147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:21:29.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:29 smithi160 ceph-mon[88407]: osdmap e315: 8 total, 8 up, 8 in 2024-09-14T04:21:29.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:29.759 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:29.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:29.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:29.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:29.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:29.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:29.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:30.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:30.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:30.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:30.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:30.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:30.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:30.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:30.332 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:30.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:30.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:30.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:30.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:30.558 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:21:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:21:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:21:30.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:30 smithi160 ceph-mon[88407]: pgmap v1220: 89 pgs: 89 active+clean; 237 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 675 op/s 2024-09-14T04:21:30.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:30.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:30.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:30 smithi114 ceph-mon[106969]: pgmap v1220: 89 pgs: 89 active+clean; 237 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 675 op/s 2024-09-14T04:21:30.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:30 smithi114 ceph-mon[103029]: pgmap v1220: 89 pgs: 89 active+clean; 237 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 675 op/s 2024-09-14T04:21:30.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:30.738 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:30.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:30.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:30.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:30.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:30.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:31.020 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:31.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:31.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:31.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:31.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:31.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:31.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:31.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:31.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:31.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:31.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:31.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:21:31.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:31.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:31.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:31.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:31.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:31.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:31.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:31.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:31.859 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:21:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:21:31] "GET /metrics HTTP/1.1" 200 36565 "" "Prometheus/2.51.0" 2024-09-14T04:21:31.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:31.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:31.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:31.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:31.972 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 117976 objects 2024-09-14T04:21:31.972 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :156.557 2024-09-14T04:21:32.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:32.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:32.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:32.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:32.201 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:32.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:32.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:32.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:32.326 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:32.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:32.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:32.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:32.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:32.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:32.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:32.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:32.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:32 smithi114 ceph-mon[103029]: pgmap v1221: 89 pgs: 89 active+clean; 151 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-09-14T04:21:32.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:32 smithi114 ceph-mon[106969]: pgmap v1221: 89 pgs: 89 active+clean; 151 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-09-14T04:21:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:32 smithi160 ceph-mon[88407]: pgmap v1221: 89 pgs: 89 active+clean; 151 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 731 op/s 2024-09-14T04:21:32.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:32.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:32.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:32.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:32.821 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:32.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:32.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:32.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:32.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:33.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:33.068 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:33.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:33.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:33.194 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:33.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:33.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:33.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:33.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:33.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:33.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:33.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:33.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:33.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:33.573 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:33.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:33.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:33.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:33.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:33.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:33.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:33.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:33.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:33.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:33.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:33.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:34.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:34.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:34.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:34.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:34.210 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:34.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:34.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:34.338 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:34.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:34.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:34.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:34.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:34.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:34.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:34.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:34.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:34.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:34 smithi114 ceph-mon[106969]: pgmap v1222: 89 pgs: 89 active+clean; 125 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 729 op/s 2024-09-14T04:21:34.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:34 smithi114 ceph-mon[103029]: pgmap v1222: 89 pgs: 89 active+clean; 125 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 729 op/s 2024-09-14T04:21:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:34 smithi160 ceph-mon[88407]: pgmap v1222: 89 pgs: 89 active+clean; 125 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 729 op/s 2024-09-14T04:21:34.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:34.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:34.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:34.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:34.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:34.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:34.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:34.976 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:34.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:35.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:35.102 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:35.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:35.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:35.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:35.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:35.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:35.350 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:35.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:35.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:35.474 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:35.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:35.494 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-09-14T04:21:35.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-09-14T04:21:35.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:35.579 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:35.592 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:35.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:35.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:35.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:35.732 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:35.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:35.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:35.859 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:35.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:35.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:35.982 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:35.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:36.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:36.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:36.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:36.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:36.236 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:36.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:36.337 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:36.364 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:36.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:36.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:36.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:36.489 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:36.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:36.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:36.599 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:36 smithi114 ceph-mon[106969]: pgmap v1223: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 607 op/s 2024-09-14T04:21:36.599 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:36 smithi114 ceph-mon[103029]: pgmap v1223: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 607 op/s 2024-09-14T04:21:36.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:36.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:36.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:36 smithi160 ceph-mon[88407]: pgmap v1223: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 607 op/s 2024-09-14T04:21:36.715 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:36.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:36.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:36.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:36.868 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:36.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:36.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:36.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:36.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:36.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:37.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:37.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:37.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:37.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:37.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:37.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:37.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:37.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:37.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:37.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:37.496 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:37.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:21:37.507 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:21:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:21:37.524 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:21:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:21:37.540 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:21:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:21:37.556 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:21:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:21:37.573 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:21:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:21:37.588 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:21:37.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:21:37.604 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:21:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:21:37.620 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:21:37.623 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:37.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:37.725 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:37.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:37.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:37.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:37.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:37.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:37.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:38.003 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:38.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:38.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:38.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:38.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:38.229 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:38.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:38.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:38.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:38.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:38.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:38.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:38.503 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:38.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:38.541 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:38 smithi114 ceph-mon[106969]: pgmap v1224: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 607 op/s 2024-09-14T04:21:38.541 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2690401899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:38.541 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:38.541 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:38.542 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:38 smithi114 ceph-mon[103029]: pgmap v1224: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 607 op/s 2024-09-14T04:21:38.542 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2690401899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:38.542 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:38.542 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:38.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:38.629 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:38.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:38 smithi160 ceph-mon[88407]: pgmap v1224: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 607 op/s 2024-09-14T04:21:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2690401899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:38.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:38.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:38.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:38.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:38.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:38.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:38.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:39.004 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:39.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:39.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:39.131 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:39.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:39.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:39.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:39.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:39.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:39.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:39.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:39.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:39.507 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:39.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:39.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:39.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:39.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:39.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:39.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:39.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:39.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:39.881 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:39.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:39.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:40.009 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:40.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:40.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:40.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:40.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:40.217 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:21:40.217 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:21:40.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:40.262 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:40.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:40.358 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:21:40.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:40.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:40 smithi114 ceph-mon[103029]: pgmap v1225: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 521 op/s 2024-09-14T04:21:40.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2802965546' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:21:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:40 smithi114 ceph-mon[106969]: pgmap v1225: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 521 op/s 2024-09-14T04:21:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2802965546' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:21:40.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:40.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:40.432 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:40.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:40.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:40.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:40.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:21:40.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:40.638 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:40.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:40 smithi160 ceph-mon[88407]: pgmap v1225: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 521 op/s 2024-09-14T04:21:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2802965546' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:21:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:21:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:21:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:21:40.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:40.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:40.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:40.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:40.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:40.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:40.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:41.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:41.015 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:41.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:41.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:41.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:41.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:41.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:41.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:41.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:41.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:41.394 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:41.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:41.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:41.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:41.521 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:41.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:41.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:41 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:41.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:41 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2802965546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:21:41.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:41 smithi114 ceph-mon[103029]: osdmap e316: 8 total, 8 up, 8 in 2024-09-14T04:21:41.529 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:21:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:21:41] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-14T04:21:41.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:41 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:41.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:41 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2802965546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:21:41.530 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:41 smithi114 ceph-mon[106969]: osdmap e316: 8 total, 8 up, 8 in 2024-09-14T04:21:41.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:41.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:41.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:41 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:41 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2802965546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:21:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:41 smithi160 ceph-mon[88407]: osdmap e316: 8 total, 8 up, 8 in 2024-09-14T04:21:41.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:41.771 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:41.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:41.872 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:41.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:41.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:42.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:42.027 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:42.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:42.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:42.154 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:42.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:42.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:42.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:42.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:42.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:42 smithi114 ceph-mon[106969]: pgmap v1227: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 241 op/s 2024-09-14T04:21:42.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:42 smithi114 ceph-mon[103029]: pgmap v1227: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 241 op/s 2024-09-14T04:21:42.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:42.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:42.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:42.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:42.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:42.527 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:42.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:42.553 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:21:42.555 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:21:42.628 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:42.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:42.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:42 smithi160 ceph-mon[88407]: pgmap v1227: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 241 op/s 2024-09-14T04:21:42.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:42.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:42.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:42.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:42.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:42.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:43.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:43.035 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:43.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:43.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:43.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:43.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:43.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:43.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:43.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:43.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:43.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:43.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:43.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:43.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:43.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:43.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:43.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:43 smithi114 ceph-mon[103029]: from='client.36711 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:21:43.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:43 smithi114 ceph-mon[106969]: from='client.36711 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:21:43.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:43 smithi160 ceph-mon[88407]: from='client.36711 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:21:43.665 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:43.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:43.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:43.794 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:43.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:43.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:43.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:21:43.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:43.948 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1227,"stamp":"2024-09-14T04:21:41.086761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":293,"num_read_kb":748,"num_write":236166,"num_write_kb":7552124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":595202,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141612,"ondisk_log_size":141612,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7376860,"kb_used_data":31516,"kb_used_omap":33,"kb_used_meta":7345246,"kb_avail":742584356,"statfs":{"total":767960285184,"available":760406380544,"internally_reserved":0,"allocated":32272384,"data_stored":20936098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34628,"internal_metadata":7521532092},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-158203928,"num_objects":-2415,"num_object_clones":0,"num_object_copies":-7245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2415,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003068"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":442,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327733+0000","last_change":"2024-09-14T03:58:45.811678+0000","last_active":"2024-09-14T04:21:28.327733+0000","last_peered":"2024-09-14T04:21:28.327733+0000","last_clean":"2024-09-14T04:21:28.327733+0000","last_became_active":"2024-09-14T03:58:45.810654+0000","last_became_peered":"2024-09-14T03:58:45.810654+0000","last_unstale":"2024-09-14T04:21:28.327733+0000","last_undegraded":"2024-09-14T04:21:28.327733+0000","last_fullsized":"2024-09-14T04:21:28.327733+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:23:45.649469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":314,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.288635+0000","last_change":"2024-09-14T04:13:37.176637+0000","last_active":"2024-09-14T04:21:20.288635+0000","last_peered":"2024-09-14T04:21:20.288635+0000","last_clean":"2024-09-14T04:21:20.288635+0000","last_became_active":"2024-09-14T04:13:37.159759+0000","last_became_peered":"2024-09-14T04:13:37.159759+0000","last_unstale":"2024-09-14T04:21:20.288635+0000","last_undegraded":"2024-09-14T04:21:20.288635+0000","last_fullsized":"2024-09-14T04:21:20.288635+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:56.293937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":280,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286351+0000","last_change":"2024-09-14T03:58:45.788755+0000","last_active":"2024-09-14T04:21:20.286351+0000","last_peered":"2024-09-14T04:21:20.286351+0000","last_clean":"2024-09-14T04:21:20.286351+0000","last_became_active":"2024-09-14T03:58:45.788464+0000","last_became_peered":"2024-09-14T03:58:45.788464+0000","last_unstale":"2024-09-14T04:21:20.286351+0000","last_undegraded":"2024-09-14T04:21:20.286351+0000","last_fullsized":"2024-09-14T04:21:20.286351+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:28:31.225976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":527,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.329628+0000","last_change":"2024-09-14T04:07:44.424089+0000","last_active":"2024-09-14T04:21:28.329628+0000","last_peered":"2024-09-14T04:21:28.329628+0000","last_clean":"2024-09-14T04:21:28.329628+0000","last_became_active":"2024-09-14T04:07:44.423767+0000","last_became_peered":"2024-09-14T04:07:44.423767+0000","last_unstale":"2024-09-14T04:21:28.329628+0000","last_undegraded":"2024-09-14T04:21:28.329628+0000","last_fullsized":"2024-09-14T04:21:28.329628+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:54.809780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":513,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.331841+0000","last_change":"2024-09-14T03:58:45.780819+0000","last_active":"2024-09-14T04:21:28.331841+0000","last_peered":"2024-09-14T04:21:28.331841+0000","last_clean":"2024-09-14T04:21:28.331841+0000","last_became_active":"2024-09-14T03:58:45.780208+0000","last_became_peered":"2024-09-14T03:58:45.780208+0000","last_unstale":"2024-09-14T04:21:28.331841+0000","last_undegraded":"2024-09-14T04:21:28.331841+0000","last_fullsized":"2024-09-14T04:21:28.331841+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:01.434575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":277,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.281390+0000","last_change":"2024-09-14T04:14:30.541064+0000","last_active":"2024-09-14T04:21:20.281390+0000","last_peered":"2024-09-14T04:21:20.281390+0000","last_clean":"2024-09-14T04:21:20.281390+0000","last_became_active":"2024-09-14T04:14:30.540755+0000","last_became_peered":"2024-09-14T04:14:30.540755+0000","last_unstale":"2024-09-14T04:21:20.281390+0000","last_undegraded":"2024-09-14T04:21:20.281390+0000","last_fullsized":"2024-09-14T04:21:20.281390+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:37.189820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":255,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.326187+0000","last_change":"2024-09-14T04:07:44.424840+0000","last_active":"2024-09-14T04:21:28.326187+0000","last_peered":"2024-09-14T04:21:28.326187+0000","last_clean":"2024-09-14T04:21:28.326187+0000","last_became_active":"2024-09-14T04:07:44.424519+0000","last_became_peered":"2024-09-14T04:07:44.424519+0000","last_unstale":"2024-09-14T04:21:28.326187+0000","last_undegraded":"2024-09-14T04:21:28.326187+0000","last_fullsized":"2024-09-14T04:21:28.326187+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:30.201688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":289,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.285373+0000","last_change":"2024-09-14T03:52:28.194155+0000","last_active":"2024-09-14T04:21:20.285373+0000","last_peered":"2024-09-14T04:21:20.285373+0000","last_clean":"2024-09-14T04:21:20.285373+0000","last_became_active":"2024-09-14T03:52:28.193667+0000","last_became_peered":"2024-09-14T03:52:28.193667+0000","last_unstale":"2024-09-14T04:21:20.285373+0000","last_undegraded":"2024-09-14T04:21:20.285373+0000","last_fullsized":"2024-09-14T04:21:20.285373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:32.956044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"315'14934","reported_seq":14980,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.941451+0000","last_change":"2024-09-14T04:17:48.104166+0000","last_active":"2024-09-14T04:21:31.941451+0000","last_peered":"2024-09-14T04:21:31.941451+0000","last_clean":"2024-09-14T04:21:31.941451+0000","last_became_active":"2024-09-14T04:17:48.083298+0000","last_became_peered":"2024-09-14T04:17:48.083298+0000","last_unstale":"2024-09-14T04:21:31.941451+0000","last_undegraded":"2024-09-14T04:21:31.941451+0000","last_fullsized":"2024-09-14T04:21:31.941451+0000","mapping_epoch":290,"log_start":"294'6100","ondisk_log_start":"294'6100","created":284,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8834,"log_dups_size":0,"ondisk_log_size":8834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:53.010206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14934,"num_write_kb":477888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":299,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.281723+0000","last_change":"2024-09-14T04:14:30.543098+0000","last_active":"2024-09-14T04:21:20.281723+0000","last_peered":"2024-09-14T04:21:20.281723+0000","last_clean":"2024-09-14T04:21:20.281723+0000","last_became_active":"2024-09-14T04:14:30.542773+0000","last_became_peered":"2024-09-14T04:14:30.542773+0000","last_unstale":"2024-09-14T04:21:20.281723+0000","last_undegraded":"2024-09-14T04:21:20.281723+0000","last_fullsized":"2024-09-14T04:21:20.281723+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:06:27.423486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":263,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286144+0000","last_change":"2024-09-14T03:43:11.910490+0000","last_active":"2024-09-14T04:21:20.286144+0000","last_peered":"2024-09-14T04:21:20.286144+0000","last_clean":"2024-09-14T04:21:20.286144+0000","last_became_active":"2024-09-14T03:43:11.910196+0000","last_became_peered":"2024-09-14T03:43:11.910196+0000","last_unstale":"2024-09-14T04:21:20.286144+0000","last_undegraded":"2024-09-14T04:21:20.286144+0000","last_fullsized":"2024-09-14T04:21:20.286144+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:05.306696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"315'14690","reported_seq":14729,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.965703+0000","last_change":"2024-09-14T04:17:13.575564+0000","last_active":"2024-09-14T04:21:31.965703+0000","last_peered":"2024-09-14T04:21:31.965703+0000","last_clean":"2024-09-14T04:21:31.965703+0000","last_became_active":"2024-09-14T04:17:13.575303+0000","last_became_peered":"2024-09-14T04:17:13.575303+0000","last_unstale":"2024-09-14T04:21:31.965703+0000","last_undegraded":"2024-09-14T04:21:31.965703+0000","last_fullsized":"2024-09-14T04:21:31.965703+0000","mapping_epoch":284,"log_start":"293'5800","ondisk_log_start":"293'5800","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:02:07.463876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14690,"num_write_kb":470080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":72,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.344309+0000","last_change":"2024-09-14T04:17:48.084067+0000","last_active":"2024-09-14T04:21:28.344309+0000","last_peered":"2024-09-14T04:21:28.344309+0000","last_clean":"2024-09-14T04:21:28.344309+0000","last_became_active":"2024-09-14T04:17:48.083584+0000","last_became_peered":"2024-09-14T04:17:48.083584+0000","last_unstale":"2024-09-14T04:21:28.344309+0000","last_undegraded":"2024-09-14T04:21:28.344309+0000","last_fullsized":"2024-09-14T04:21:28.344309+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:01.274210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":360,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.330995+0000","last_change":"2024-09-14T04:17:48.083353+0000","last_active":"2024-09-14T04:21:28.330995+0000","last_peered":"2024-09-14T04:21:28.330995+0000","last_clean":"2024-09-14T04:21:28.330995+0000","last_became_active":"2024-09-14T04:17:48.083032+0000","last_became_peered":"2024-09-14T04:17:48.083032+0000","last_unstale":"2024-09-14T04:21:28.330995+0000","last_undegraded":"2024-09-14T04:21:28.330995+0000","last_fullsized":"2024-09-14T04:21:28.330995+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:15:25.358824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"315'14674","reported_seq":14746,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.843160+0000","last_change":"2024-09-14T04:17:48.088284+0000","last_active":"2024-09-14T04:21:31.843160+0000","last_peered":"2024-09-14T04:21:31.843160+0000","last_clean":"2024-09-14T04:21:31.843160+0000","last_became_active":"2024-09-14T04:17:48.083349+0000","last_became_peered":"2024-09-14T04:17:48.083349+0000","last_unstale":"2024-09-14T04:21:31.843160+0000","last_undegraded":"2024-09-14T04:21:31.843160+0000","last_fullsized":"2024-09-14T04:21:31.843160+0000","mapping_epoch":290,"log_start":"295'6700","ondisk_log_start":"295'6700","created":284,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":7974,"log_dups_size":0,"ondisk_log_size":7974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:56:56.489946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14674,"num_write_kb":469568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":76,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286341+0000","last_change":"2024-09-14T04:14:30.561646+0000","last_active":"2024-09-14T04:21:20.286341+0000","last_peered":"2024-09-14T04:21:20.286341+0000","last_clean":"2024-09-14T04:21:20.286341+0000","last_became_active":"2024-09-14T04:14:30.539174+0000","last_became_peered":"2024-09-14T04:14:30.539174+0000","last_unstale":"2024-09-14T04:21:20.286341+0000","last_undegraded":"2024-09-14T04:21:20.286341+0000","last_fullsized":"2024-09-14T04:21:20.286341+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:02:49.397620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":290,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.282567+0000","last_change":"2024-09-14T03:55:58.252674+0000","last_active":"2024-09-14T04:21:20.282567+0000","last_peered":"2024-09-14T04:21:20.282567+0000","last_clean":"2024-09-14T04:21:20.282567+0000","last_became_active":"2024-09-14T03:55:58.252088+0000","last_became_peered":"2024-09-14T03:55:58.252088+0000","last_unstale":"2024-09-14T04:21:20.282567+0000","last_undegraded":"2024-09-14T04:21:20.282567+0000","last_fullsized":"2024-09-14T04:21:20.282567+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:37.591958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"315'14982","reported_seq":15057,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.839282+0000","last_change":"2024-09-14T04:17:13.574686+0000","last_active":"2024-09-14T04:21:31.839282+0000","last_peered":"2024-09-14T04:21:31.839282+0000","last_clean":"2024-09-14T04:21:31.839282+0000","last_became_active":"2024-09-14T04:17:13.574396+0000","last_became_peered":"2024-09-14T04:17:13.574396+0000","last_unstale":"2024-09-14T04:21:31.839282+0000","last_undegraded":"2024-09-14T04:21:31.839282+0000","last_fullsized":"2024-09-14T04:21:31.839282+0000","mapping_epoch":284,"log_start":"293'5600","ondisk_log_start":"293'5600","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:06:20.930337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14982,"num_write_kb":479424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":301,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286456+0000","last_change":"2024-09-14T04:07:44.335399+0000","last_active":"2024-09-14T04:21:20.286456+0000","last_peered":"2024-09-14T04:21:20.286456+0000","last_clean":"2024-09-14T04:21:20.286456+0000","last_became_active":"2024-09-14T04:07:44.335024+0000","last_became_peered":"2024-09-14T04:07:44.335024+0000","last_unstale":"2024-09-14T04:21:20.286456+0000","last_undegraded":"2024-09-14T04:21:20.286456+0000","last_fullsized":"2024-09-14T04:21:20.286456+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:03.340254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":548,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327245+0000","last_change":"2024-09-14T04:13:37.165995+0000","last_active":"2024-09-14T04:21:28.327245+0000","last_peered":"2024-09-14T04:21:28.327245+0000","last_clean":"2024-09-14T04:21:28.327245+0000","last_became_active":"2024-09-14T04:13:37.165699+0000","last_became_peered":"2024-09-14T04:13:37.165699+0000","last_unstale":"2024-09-14T04:21:28.327245+0000","last_undegraded":"2024-09-14T04:21:28.327245+0000","last_fullsized":"2024-09-14T04:21:28.327245+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:46:21.792342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.4","version":"315'14510","reported_seq":14549,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.960248+0000","last_change":"2024-09-14T04:17:13.572270+0000","last_active":"2024-09-14T04:21:31.960248+0000","last_peered":"2024-09-14T04:21:31.960248+0000","last_clean":"2024-09-14T04:21:31.960248+0000","last_became_active":"2024-09-14T04:17:13.571986+0000","last_became_peered":"2024-09-14T04:17:13.571986+0000","last_unstale":"2024-09-14T04:21:31.960248+0000","last_undegraded":"2024-09-14T04:21:31.960248+0000","last_fullsized":"2024-09-14T04:21:31.960248+0000","mapping_epoch":284,"log_start":"293'5600","ondisk_log_start":"293'5600","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:58:04.909769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14510,"num_write_kb":464320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":493,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.326730+0000","last_change":"2024-09-14T04:13:37.173224+0000","last_active":"2024-09-14T04:21:28.326730+0000","last_peered":"2024-09-14T04:21:28.326730+0000","last_clean":"2024-09-14T04:21:28.326730+0000","last_became_active":"2024-09-14T04:13:37.158630+0000","last_became_peered":"2024-09-14T04:13:37.158630+0000","last_unstale":"2024-09-14T04:21:28.326730+0000","last_undegraded":"2024-09-14T04:21:28.326730+0000","last_fullsized":"2024-09-14T04:21:28.326730+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:41:25.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":329,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327880+0000","last_change":"2024-09-14T04:17:48.083765+0000","last_active":"2024-09-14T04:21:28.327880+0000","last_peered":"2024-09-14T04:21:28.327880+0000","last_clean":"2024-09-14T04:21:28.327880+0000","last_became_active":"2024-09-14T04:17:48.083385+0000","last_became_peered":"2024-09-14T04:17:48.083385+0000","last_unstale":"2024-09-14T04:21:28.327880+0000","last_undegraded":"2024-09-14T04:21:28.327880+0000","last_fullsized":"2024-09-14T04:21:28.327880+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:13:49.268399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.5","version":"315'14854","reported_seq":14893,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.961733+0000","last_change":"2024-09-14T04:17:13.572972+0000","last_active":"2024-09-14T04:21:31.961733+0000","last_peered":"2024-09-14T04:21:31.961733+0000","last_clean":"2024-09-14T04:21:31.961733+0000","last_became_active":"2024-09-14T04:17:13.572619+0000","last_became_peered":"2024-09-14T04:17:13.572619+0000","last_unstale":"2024-09-14T04:21:31.961733+0000","last_undegraded":"2024-09-14T04:21:31.961733+0000","last_fullsized":"2024-09-14T04:21:31.961733+0000","mapping_epoch":284,"log_start":"294'6000","ondisk_log_start":"294'6000","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8854,"log_dups_size":0,"ondisk_log_size":8854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:07:11.298450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14854,"num_write_kb":475328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":301,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.282699+0000","last_change":"2024-09-14T04:13:37.166773+0000","last_active":"2024-09-14T04:21:20.282699+0000","last_peered":"2024-09-14T04:21:20.282699+0000","last_clean":"2024-09-14T04:21:20.282699+0000","last_became_active":"2024-09-14T04:13:37.166465+0000","last_became_peered":"2024-09-14T04:13:37.166465+0000","last_unstale":"2024-09-14T04:21:20.282699+0000","last_undegraded":"2024-09-14T04:21:20.282699+0000","last_fullsized":"2024-09-14T04:21:20.282699+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:57:18.786263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":320,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.329906+0000","last_change":"2024-09-14T04:17:48.084357+0000","last_active":"2024-09-14T04:21:28.329906+0000","last_peered":"2024-09-14T04:21:28.329906+0000","last_clean":"2024-09-14T04:21:28.329906+0000","last_became_active":"2024-09-14T04:17:48.084034+0000","last_became_peered":"2024-09-14T04:17:48.084034+0000","last_unstale":"2024-09-14T04:21:28.329906+0000","last_undegraded":"2024-09-14T04:21:28.329906+0000","last_fullsized":"2024-09-14T04:21:28.329906+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:42:00.818438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"315'14808","reported_seq":14883,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.967881+0000","last_change":"2024-09-14T04:17:13.575249+0000","last_active":"2024-09-14T04:21:31.967881+0000","last_peered":"2024-09-14T04:21:31.967881+0000","last_clean":"2024-09-14T04:21:31.967881+0000","last_became_active":"2024-09-14T04:17:13.574960+0000","last_became_peered":"2024-09-14T04:17:13.574960+0000","last_unstale":"2024-09-14T04:21:31.967881+0000","last_undegraded":"2024-09-14T04:21:31.967881+0000","last_fullsized":"2024-09-14T04:21:31.967881+0000","mapping_epoch":284,"log_start":"294'6400","ondisk_log_start":"294'6400","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8408,"log_dups_size":0,"ondisk_log_size":8408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:31.199701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14808,"num_write_kb":473856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":73,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.281356+0000","last_change":"2024-09-14T04:13:37.158807+0000","last_active":"2024-09-14T04:21:20.281356+0000","last_peered":"2024-09-14T04:21:20.281356+0000","last_clean":"2024-09-14T04:21:20.281356+0000","last_became_active":"2024-09-14T04:13:37.158475+0000","last_became_peered":"2024-09-14T04:13:37.158475+0000","last_unstale":"2024-09-14T04:21:20.281356+0000","last_undegraded":"2024-09-14T04:21:20.281356+0000","last_fullsized":"2024-09-14T04:21:20.281356+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:36:09.005093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":183,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.282517+0000","last_change":"2024-09-14T04:17:48.081945+0000","last_active":"2024-09-14T04:21:20.282517+0000","last_peered":"2024-09-14T04:21:20.282517+0000","last_clean":"2024-09-14T04:21:20.282517+0000","last_became_active":"2024-09-14T04:17:48.081510+0000","last_became_peered":"2024-09-14T04:17:48.081510+0000","last_unstale":"2024-09-14T04:21:20.282517+0000","last_undegraded":"2024-09-14T04:21:20.282517+0000","last_fullsized":"2024-09-14T04:21:20.282517+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:19:48.331487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"315'14804","reported_seq":14879,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.907637+0000","last_change":"2024-09-14T04:17:13.572496+0000","last_active":"2024-09-14T04:21:31.907637+0000","last_peered":"2024-09-14T04:21:31.907637+0000","last_clean":"2024-09-14T04:21:31.907637+0000","last_became_active":"2024-09-14T04:17:13.572182+0000","last_became_peered":"2024-09-14T04:17:13.572182+0000","last_unstale":"2024-09-14T04:21:31.907637+0000","last_undegraded":"2024-09-14T04:21:31.907637+0000","last_fullsized":"2024-09-14T04:21:31.907637+0000","mapping_epoch":284,"log_start":"294'6400","ondisk_log_start":"294'6400","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8404,"log_dups_size":0,"ondisk_log_size":8404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:43:40.019232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14804,"num_write_kb":473728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":256,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.325847+0000","last_change":"2024-09-14T04:07:44.424093+0000","last_active":"2024-09-14T04:21:28.325847+0000","last_peered":"2024-09-14T04:21:28.325847+0000","last_clean":"2024-09-14T04:21:28.325847+0000","last_became_active":"2024-09-14T04:07:44.423773+0000","last_became_peered":"2024-09-14T04:07:44.423773+0000","last_unstale":"2024-09-14T04:21:28.325847+0000","last_undegraded":"2024-09-14T04:21:28.325847+0000","last_fullsized":"2024-09-14T04:21:28.325847+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:40:22.631832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":493,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.333256+0000","last_change":"2024-09-14T04:13:37.168484+0000","last_active":"2024-09-14T04:21:28.333256+0000","last_peered":"2024-09-14T04:21:28.333256+0000","last_clean":"2024-09-14T04:21:28.333256+0000","last_became_active":"2024-09-14T04:13:37.168136+0000","last_became_peered":"2024-09-14T04:13:37.168136+0000","last_unstale":"2024-09-14T04:21:28.333256+0000","last_undegraded":"2024-09-14T04:21:28.333256+0000","last_fullsized":"2024-09-14T04:21:28.333256+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:13.681730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"315'14780","reported_seq":14820,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.961805+0000","last_change":"2024-09-14T04:17:13.572747+0000","last_active":"2024-09-14T04:21:31.961805+0000","last_peered":"2024-09-14T04:21:31.961805+0000","last_clean":"2024-09-14T04:21:31.961805+0000","last_became_active":"2024-09-14T04:17:13.572451+0000","last_became_peered":"2024-09-14T04:17:13.572451+0000","last_unstale":"2024-09-14T04:21:31.961805+0000","last_undegraded":"2024-09-14T04:21:31.961805+0000","last_fullsized":"2024-09-14T04:21:31.961805+0000","mapping_epoch":284,"log_start":"295'6600","ondisk_log_start":"295'6600","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:00:57.163738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14780,"num_write_kb":472897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":263,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327547+0000","last_change":"2024-09-14T04:14:30.540124+0000","last_active":"2024-09-14T04:21:28.327547+0000","last_peered":"2024-09-14T04:21:28.327547+0000","last_clean":"2024-09-14T04:21:28.327547+0000","last_became_active":"2024-09-14T04:14:30.539704+0000","last_became_peered":"2024-09-14T04:14:30.539704+0000","last_unstale":"2024-09-14T04:21:28.327547+0000","last_undegraded":"2024-09-14T04:21:28.327547+0000","last_fullsized":"2024-09-14T04:21:28.327547+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:52:13.421617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":294,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286290+0000","last_change":"2024-09-14T03:55:58.248438+0000","last_active":"2024-09-14T04:21:20.286290+0000","last_peered":"2024-09-14T04:21:20.286290+0000","last_clean":"2024-09-14T04:21:20.286290+0000","last_became_active":"2024-09-14T03:55:58.248064+0000","last_became_peered":"2024-09-14T03:55:58.248064+0000","last_unstale":"2024-09-14T04:21:20.286290+0000","last_undegraded":"2024-09-14T04:21:20.286290+0000","last_fullsized":"2024-09-14T04:21:20.286290+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:35.728417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031400699999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":256,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.326404+0000","last_change":"2024-09-14T04:07:44.333868+0000","last_active":"2024-09-14T04:21:28.326404+0000","last_peered":"2024-09-14T04:21:28.326404+0000","last_clean":"2024-09-14T04:21:28.326404+0000","last_became_active":"2024-09-14T04:07:44.333555+0000","last_became_peered":"2024-09-14T04:07:44.333555+0000","last_unstale":"2024-09-14T04:21:28.326404+0000","last_undegraded":"2024-09-14T04:21:28.326404+0000","last_fullsized":"2024-09-14T04:21:28.326404+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:38:34.980558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":283,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.282893+0000","last_change":"2024-09-14T04:13:37.166354+0000","last_active":"2024-09-14T04:21:20.282893+0000","last_peered":"2024-09-14T04:21:20.282893+0000","last_clean":"2024-09-14T04:21:20.282893+0000","last_became_active":"2024-09-14T04:13:37.166057+0000","last_became_peered":"2024-09-14T04:13:37.166057+0000","last_unstale":"2024-09-14T04:21:20.282893+0000","last_undegraded":"2024-09-14T04:21:20.282893+0000","last_fullsized":"2024-09-14T04:21:20.282893+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:40.817585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"315'14626","reported_seq":14700,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.895384+0000","last_change":"2024-09-14T04:17:13.570675+0000","last_active":"2024-09-14T04:21:31.895384+0000","last_peered":"2024-09-14T04:21:31.895384+0000","last_clean":"2024-09-14T04:21:31.895384+0000","last_became_active":"2024-09-14T04:17:13.569382+0000","last_became_peered":"2024-09-14T04:17:13.569382+0000","last_unstale":"2024-09-14T04:21:31.895384+0000","last_undegraded":"2024-09-14T04:21:31.895384+0000","last_fullsized":"2024-09-14T04:21:31.895384+0000","mapping_epoch":284,"log_start":"293'5200","ondisk_log_start":"293'5200","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":9426,"log_dups_size":0,"ondisk_log_size":9426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:10:19.648738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14626,"num_write_kb":468032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":206,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.331380+0000","last_change":"2024-09-14T04:14:30.539202+0000","last_active":"2024-09-14T04:21:28.331380+0000","last_peered":"2024-09-14T04:21:28.331380+0000","last_clean":"2024-09-14T04:21:28.331380+0000","last_became_active":"2024-09-14T04:14:30.538884+0000","last_became_peered":"2024-09-14T04:14:30.538884+0000","last_unstale":"2024-09-14T04:21:28.331380+0000","last_undegraded":"2024-09-14T04:21:28.331380+0000","last_fullsized":"2024-09-14T04:21:28.331380+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:50:39.511239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":543,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327283+0000","last_change":"2024-09-14T04:13:37.165079+0000","last_active":"2024-09-14T04:21:28.327283+0000","last_peered":"2024-09-14T04:21:28.327283+0000","last_clean":"2024-09-14T04:21:28.327283+0000","last_became_active":"2024-09-14T04:13:37.164753+0000","last_became_peered":"2024-09-14T04:13:37.164753+0000","last_unstale":"2024-09-14T04:21:28.327283+0000","last_undegraded":"2024-09-14T04:21:28.327283+0000","last_fullsized":"2024-09-14T04:21:28.327283+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:50:32.658239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":525,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.331427+0000","last_change":"2024-09-14T03:55:58.254297+0000","last_active":"2024-09-14T04:21:28.331427+0000","last_peered":"2024-09-14T04:21:28.331427+0000","last_clean":"2024-09-14T04:21:28.331427+0000","last_became_active":"2024-09-14T03:55:58.254013+0000","last_became_peered":"2024-09-14T03:55:58.254013+0000","last_unstale":"2024-09-14T04:21:28.331427+0000","last_undegraded":"2024-09-14T04:21:28.331427+0000","last_fullsized":"2024-09-14T04:21:28.331427+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:26.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"315'14958","reported_seq":14997,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.863049+0000","last_change":"2024-09-14T04:17:13.572985+0000","last_active":"2024-09-14T04:21:31.863049+0000","last_peered":"2024-09-14T04:21:31.863049+0000","last_clean":"2024-09-14T04:21:31.863049+0000","last_became_active":"2024-09-14T04:17:13.572399+0000","last_became_peered":"2024-09-14T04:17:13.572399+0000","last_unstale":"2024-09-14T04:21:31.863049+0000","last_undegraded":"2024-09-14T04:21:31.863049+0000","last_fullsized":"2024-09-14T04:21:31.863049+0000","mapping_epoch":284,"log_start":"294'6100","ondisk_log_start":"294'6100","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":8858,"log_dups_size":0,"ondisk_log_size":8858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:30:58.922138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14958,"num_write_kb":478656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.329013+0000","last_change":"2024-09-14T04:07:44.337933+0000","last_active":"2024-09-14T04:21:28.329013+0000","last_peered":"2024-09-14T04:21:28.329013+0000","last_clean":"2024-09-14T04:21:28.329013+0000","last_became_active":"2024-09-14T04:07:44.337608+0000","last_became_peered":"2024-09-14T04:07:44.337608+0000","last_unstale":"2024-09-14T04:21:28.329013+0000","last_undegraded":"2024-09-14T04:21:28.329013+0000","last_fullsized":"2024-09-14T04:21:28.329013+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:21:56.114497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":480,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.331789+0000","last_change":"2024-09-14T03:55:58.289209+0000","last_active":"2024-09-14T04:21:28.331789+0000","last_peered":"2024-09-14T04:21:28.331789+0000","last_clean":"2024-09-14T04:21:28.331789+0000","last_became_active":"2024-09-14T03:55:58.288904+0000","last_became_peered":"2024-09-14T03:55:58.288904+0000","last_unstale":"2024-09-14T04:21:28.331789+0000","last_undegraded":"2024-09-14T04:21:28.331789+0000","last_fullsized":"2024-09-14T04:21:28.331789+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.018097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":250,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.332583+0000","last_change":"2024-09-14T04:07:54.360476+0000","last_active":"2024-09-14T04:21:28.332583+0000","last_peered":"2024-09-14T04:21:28.332583+0000","last_clean":"2024-09-14T04:21:28.332583+0000","last_became_active":"2024-09-14T04:07:54.360167+0000","last_became_peered":"2024-09-14T04:07:54.360167+0000","last_unstale":"2024-09-14T04:21:28.332583+0000","last_undegraded":"2024-09-14T04:21:28.332583+0000","last_fullsized":"2024-09-14T04:21:28.332583+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:48:00.746179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.c","version":"315'14886","reported_seq":14960,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.948689+0000","last_change":"2024-09-14T04:17:13.570232+0000","last_active":"2024-09-14T04:21:31.948689+0000","last_peered":"2024-09-14T04:21:31.948689+0000","last_clean":"2024-09-14T04:21:31.948689+0000","last_became_active":"2024-09-14T04:17:13.569953+0000","last_became_peered":"2024-09-14T04:17:13.569953+0000","last_unstale":"2024-09-14T04:21:31.948689+0000","last_undegraded":"2024-09-14T04:21:31.948689+0000","last_fullsized":"2024-09-14T04:21:31.948689+0000","mapping_epoch":284,"log_start":"293'5500","ondisk_log_start":"293'5500","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:09:54.113872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14886,"num_write_kb":476352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":295,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286522+0000","last_change":"2024-09-14T03:54:09.088210+0000","last_active":"2024-09-14T04:21:20.286522+0000","last_peered":"2024-09-14T04:21:20.286522+0000","last_clean":"2024-09-14T04:21:20.286522+0000","last_became_active":"2024-09-14T03:54:09.087868+0000","last_became_peered":"2024-09-14T03:54:09.087868+0000","last_unstale":"2024-09-14T04:21:20.286522+0000","last_undegraded":"2024-09-14T04:21:20.286522+0000","last_fullsized":"2024-09-14T04:21:20.286522+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:11:03.745792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067113700000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":275,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327616+0000","last_change":"2024-09-14T04:17:48.083263+0000","last_active":"2024-09-14T04:21:28.327616+0000","last_peered":"2024-09-14T04:21:28.327616+0000","last_clean":"2024-09-14T04:21:28.327616+0000","last_became_active":"2024-09-14T04:17:48.082791+0000","last_became_peered":"2024-09-14T04:17:48.082791+0000","last_unstale":"2024-09-14T04:21:28.327616+0000","last_undegraded":"2024-09-14T04:21:28.327616+0000","last_fullsized":"2024-09-14T04:21:28.327616+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:20:46.928898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":505,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.344130+0000","last_change":"2024-09-14T04:07:44.342023+0000","last_active":"2024-09-14T04:21:28.344130+0000","last_peered":"2024-09-14T04:21:28.344130+0000","last_clean":"2024-09-14T04:21:28.344130+0000","last_became_active":"2024-09-14T04:07:44.341597+0000","last_became_peered":"2024-09-14T04:07:44.341597+0000","last_unstale":"2024-09-14T04:21:28.344130+0000","last_undegraded":"2024-09-14T04:21:28.344130+0000","last_fullsized":"2024-09-14T04:21:28.344130+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:26:34.088686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"315'14758","reported_seq":14833,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.971646+0000","last_change":"2024-09-14T04:17:13.575911+0000","last_active":"2024-09-14T04:21:31.971646+0000","last_peered":"2024-09-14T04:21:31.971646+0000","last_clean":"2024-09-14T04:21:31.971646+0000","last_became_active":"2024-09-14T04:17:13.575627+0000","last_became_peered":"2024-09-14T04:17:13.575627+0000","last_unstale":"2024-09-14T04:21:31.971646+0000","last_undegraded":"2024-09-14T04:21:31.971646+0000","last_fullsized":"2024-09-14T04:21:31.971646+0000","mapping_epoch":284,"log_start":"292'4700","ondisk_log_start":"292'4700","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:32:39.359481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14758,"num_write_kb":472256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":318,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286433+0000","last_change":"2024-09-14T04:07:44.334791+0000","last_active":"2024-09-14T04:21:20.286433+0000","last_peered":"2024-09-14T04:21:20.286433+0000","last_clean":"2024-09-14T04:21:20.286433+0000","last_became_active":"2024-09-14T04:07:44.334398+0000","last_became_peered":"2024-09-14T04:07:44.334398+0000","last_unstale":"2024-09-14T04:21:20.286433+0000","last_undegraded":"2024-09-14T04:21:20.286433+0000","last_fullsized":"2024-09-14T04:21:20.286433+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:49.350915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.344246+0000","last_change":"2024-09-14T03:54:09.094609+0000","last_active":"2024-09-14T04:21:28.344246+0000","last_peered":"2024-09-14T04:21:28.344246+0000","last_clean":"2024-09-14T04:21:28.344246+0000","last_became_active":"2024-09-14T03:54:09.091316+0000","last_became_peered":"2024-09-14T03:54:09.091316+0000","last_unstale":"2024-09-14T04:21:28.344246+0000","last_undegraded":"2024-09-14T04:21:28.344246+0000","last_fullsized":"2024-09-14T04:21:28.344246+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:34:52.490353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":483,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.332421+0000","last_change":"2024-09-14T04:13:37.163670+0000","last_active":"2024-09-14T04:21:28.332421+0000","last_peered":"2024-09-14T04:21:28.332421+0000","last_clean":"2024-09-14T04:21:28.332421+0000","last_became_active":"2024-09-14T04:13:37.163234+0000","last_became_peered":"2024-09-14T04:13:37.163234+0000","last_unstale":"2024-09-14T04:21:28.332421+0000","last_undegraded":"2024-09-14T04:21:28.332421+0000","last_fullsized":"2024-09-14T04:21:28.332421+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:53:34.767208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"315'14498","reported_seq":14572,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.962112+0000","last_change":"2024-09-14T04:17:13.573945+0000","last_active":"2024-09-14T04:21:31.962112+0000","last_peered":"2024-09-14T04:21:31.962112+0000","last_clean":"2024-09-14T04:21:31.962112+0000","last_became_active":"2024-09-14T04:17:13.573636+0000","last_became_peered":"2024-09-14T04:17:13.573636+0000","last_unstale":"2024-09-14T04:21:31.962112+0000","last_undegraded":"2024-09-14T04:21:31.962112+0000","last_fullsized":"2024-09-14T04:21:31.962112+0000","mapping_epoch":284,"log_start":"295'6600","ondisk_log_start":"295'6600","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":7898,"log_dups_size":0,"ondisk_log_size":7898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:34:37.026824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14498,"num_write_kb":463936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":480,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.344384+0000","last_change":"2024-09-14T03:55:58.248479+0000","last_active":"2024-09-14T04:21:28.344384+0000","last_peered":"2024-09-14T04:21:28.344384+0000","last_clean":"2024-09-14T04:21:28.344384+0000","last_became_active":"2024-09-14T03:55:58.248195+0000","last_became_peered":"2024-09-14T03:55:58.248195+0000","last_unstale":"2024-09-14T04:21:28.344384+0000","last_undegraded":"2024-09-14T04:21:28.344384+0000","last_fullsized":"2024-09-14T04:21:28.344384+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:14.649754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":312,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.288970+0000","last_change":"2024-09-14T04:13:37.162059+0000","last_active":"2024-09-14T04:21:20.288970+0000","last_peered":"2024-09-14T04:21:20.288970+0000","last_clean":"2024-09-14T04:21:20.288970+0000","last_became_active":"2024-09-14T04:13:37.161728+0000","last_became_peered":"2024-09-14T04:13:37.161728+0000","last_unstale":"2024-09-14T04:21:20.288970+0000","last_undegraded":"2024-09-14T04:21:20.288970+0000","last_fullsized":"2024-09-14T04:21:20.288970+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:39:59.671661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":504,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327400+0000","last_change":"2024-09-14T03:52:28.177947+0000","last_active":"2024-09-14T04:21:28.327400+0000","last_peered":"2024-09-14T04:21:28.327400+0000","last_clean":"2024-09-14T04:21:28.327400+0000","last_became_active":"2024-09-14T03:52:28.175184+0000","last_became_peered":"2024-09-14T03:52:28.175184+0000","last_unstale":"2024-09-14T04:21:28.327400+0000","last_undegraded":"2024-09-14T04:21:28.327400+0000","last_fullsized":"2024-09-14T04:21:28.327400+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:18.976570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"315'14420","reported_seq":14495,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.956048+0000","last_change":"2024-09-14T04:17:13.576627+0000","last_active":"2024-09-14T04:21:31.956048+0000","last_peered":"2024-09-14T04:21:31.956048+0000","last_clean":"2024-09-14T04:21:31.956048+0000","last_became_active":"2024-09-14T04:17:13.576099+0000","last_became_peered":"2024-09-14T04:17:13.576099+0000","last_unstale":"2024-09-14T04:21:31.956048+0000","last_undegraded":"2024-09-14T04:21:31.956048+0000","last_fullsized":"2024-09-14T04:21:31.956048+0000","mapping_epoch":284,"log_start":"295'6500","ondisk_log_start":"295'6500","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":7920,"log_dups_size":0,"ondisk_log_size":7920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:31:19.819415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14420,"num_write_kb":461440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":471,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.326308+0000","last_change":"2024-09-14T03:58:45.795613+0000","last_active":"2024-09-14T04:21:28.326308+0000","last_peered":"2024-09-14T04:21:28.326308+0000","last_clean":"2024-09-14T04:21:28.326308+0000","last_became_active":"2024-09-14T03:58:45.792968+0000","last_became_peered":"2024-09-14T03:58:45.792968+0000","last_unstale":"2024-09-14T04:21:28.326308+0000","last_undegraded":"2024-09-14T04:21:28.326308+0000","last_fullsized":"2024-09-14T04:21:28.326308+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:21:41.403279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":493,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.344753+0000","last_change":"2024-09-14T04:13:37.169517+0000","last_active":"2024-09-14T04:21:28.344753+0000","last_peered":"2024-09-14T04:21:28.344753+0000","last_clean":"2024-09-14T04:21:28.344753+0000","last_became_active":"2024-09-14T04:13:37.169027+0000","last_became_peered":"2024-09-14T04:13:37.169027+0000","last_unstale":"2024-09-14T04:21:28.344753+0000","last_undegraded":"2024-09-14T04:21:28.344753+0000","last_fullsized":"2024-09-14T04:21:28.344753+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:49:57.369201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":126,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286067+0000","last_change":"2024-09-14T04:07:54.361978+0000","last_active":"2024-09-14T04:21:20.286067+0000","last_peered":"2024-09-14T04:21:20.286067+0000","last_clean":"2024-09-14T04:21:20.286067+0000","last_became_active":"2024-09-14T04:07:54.361643+0000","last_became_peered":"2024-09-14T04:07:54.361643+0000","last_unstale":"2024-09-14T04:21:20.286067+0000","last_undegraded":"2024-09-14T04:21:20.286067+0000","last_fullsized":"2024-09-14T04:21:20.286067+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:08:06.449840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.9","version":"315'14772","reported_seq":14847,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:31.950448+0000","last_change":"2024-09-14T04:17:13.577599+0000","last_active":"2024-09-14T04:21:31.950448+0000","last_peered":"2024-09-14T04:21:31.950448+0000","last_clean":"2024-09-14T04:21:31.950448+0000","last_became_active":"2024-09-14T04:17:13.577330+0000","last_became_peered":"2024-09-14T04:17:13.577330+0000","last_unstale":"2024-09-14T04:21:31.950448+0000","last_undegraded":"2024-09-14T04:21:31.950448+0000","last_fullsized":"2024-09-14T04:21:31.950448+0000","mapping_epoch":284,"log_start":"292'4700","ondisk_log_start":"292'4700","created":284,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:17:12.542831+0000","last_clean_scrub_stamp":"2024-09-14T04:17:12.542831+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:59:12.058934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14772,"num_write_kb":472704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":539,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.331899+0000","last_change":"2024-09-14T04:13:37.178106+0000","last_active":"2024-09-14T04:21:28.331899+0000","last_peered":"2024-09-14T04:21:28.331899+0000","last_clean":"2024-09-14T04:21:28.331899+0000","last_became_active":"2024-09-14T04:13:37.167009+0000","last_became_peered":"2024-09-14T04:13:37.167009+0000","last_unstale":"2024-09-14T04:21:28.331899+0000","last_undegraded":"2024-09-14T04:21:28.331899+0000","last_fullsized":"2024-09-14T04:21:28.331899+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:28:37.700664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":293,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.282825+0000","last_change":"2024-09-14T03:55:58.252066+0000","last_active":"2024-09-14T04:21:20.282825+0000","last_peered":"2024-09-14T04:21:20.282825+0000","last_clean":"2024-09-14T04:21:20.282825+0000","last_became_active":"2024-09-14T03:55:58.251681+0000","last_became_peered":"2024-09-14T03:55:58.251681+0000","last_unstale":"2024-09-14T04:21:20.282825+0000","last_undegraded":"2024-09-14T04:21:20.282825+0000","last_fullsized":"2024-09-14T04:21:20.282825+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:05:18.182776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":129,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286084+0000","last_change":"2024-09-14T04:17:48.080621+0000","last_active":"2024-09-14T04:21:20.286084+0000","last_peered":"2024-09-14T04:21:20.286084+0000","last_clean":"2024-09-14T04:21:20.286084+0000","last_became_active":"2024-09-14T04:17:48.080300+0000","last_became_peered":"2024-09-14T04:17:48.080300+0000","last_unstale":"2024-09-14T04:21:20.286084+0000","last_undegraded":"2024-09-14T04:21:20.286084+0000","last_fullsized":"2024-09-14T04:21:20.286084+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:14:47.756124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":73,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.281640+0000","last_change":"2024-09-14T04:13:37.162516+0000","last_active":"2024-09-14T04:21:20.281640+0000","last_peered":"2024-09-14T04:21:20.281640+0000","last_clean":"2024-09-14T04:21:20.281640+0000","last_became_active":"2024-09-14T04:13:37.162248+0000","last_became_peered":"2024-09-14T04:13:37.162248+0000","last_unstale":"2024-09-14T04:21:20.281640+0000","last_undegraded":"2024-09-14T04:21:20.281640+0000","last_fullsized":"2024-09-14T04:21:20.281640+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:41.935908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":182,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.288637+0000","last_change":"2024-09-14T04:17:48.080930+0000","last_active":"2024-09-14T04:21:20.288637+0000","last_peered":"2024-09-14T04:21:20.288637+0000","last_clean":"2024-09-14T04:21:20.288637+0000","last_became_active":"2024-09-14T04:17:48.080575+0000","last_became_peered":"2024-09-14T04:17:48.080575+0000","last_unstale":"2024-09-14T04:21:20.288637+0000","last_undegraded":"2024-09-14T04:21:20.288637+0000","last_fullsized":"2024-09-14T04:21:20.288637+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:14:19.265519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":478,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.332229+0000","last_change":"2024-09-14T03:55:58.250488+0000","last_active":"2024-09-14T04:21:28.332229+0000","last_peered":"2024-09-14T04:21:28.332229+0000","last_clean":"2024-09-14T04:21:28.332229+0000","last_became_active":"2024-09-14T03:55:58.250208+0000","last_became_peered":"2024-09-14T03:55:58.250208+0000","last_unstale":"2024-09-14T04:21:28.332229+0000","last_undegraded":"2024-09-14T04:21:28.332229+0000","last_fullsized":"2024-09-14T04:21:28.332229+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:08.944456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":131,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.331194+0000","last_change":"2024-09-14T04:14:30.539928+0000","last_active":"2024-09-14T04:21:28.331194+0000","last_peered":"2024-09-14T04:21:28.331194+0000","last_clean":"2024-09-14T04:21:28.331194+0000","last_became_active":"2024-09-14T04:14:30.539572+0000","last_became_peered":"2024-09-14T04:14:30.539572+0000","last_unstale":"2024-09-14T04:21:28.331194+0000","last_undegraded":"2024-09-14T04:21:28.331194+0000","last_fullsized":"2024-09-14T04:21:28.331194+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:55:42.675243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286521+0000","last_change":"2024-09-14T03:54:09.082843+0000","last_active":"2024-09-14T04:21:20.286521+0000","last_peered":"2024-09-14T04:21:20.286521+0000","last_clean":"2024-09-14T04:21:20.286521+0000","last_became_active":"2024-09-14T03:54:09.082440+0000","last_became_peered":"2024-09-14T03:54:09.082440+0000","last_unstale":"2024-09-14T04:21:20.286521+0000","last_undegraded":"2024-09-14T04:21:20.286521+0000","last_fullsized":"2024-09-14T04:21:20.286521+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:33.171568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":430,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327116+0000","last_change":"2024-09-14T03:58:45.791168+0000","last_active":"2024-09-14T04:21:28.327116+0000","last_peered":"2024-09-14T04:21:28.327116+0000","last_clean":"2024-09-14T04:21:28.327116+0000","last_became_active":"2024-09-14T03:58:45.789766+0000","last_became_peered":"2024-09-14T03:58:45.789766+0000","last_unstale":"2024-09-14T04:21:28.327116+0000","last_undegraded":"2024-09-14T04:21:28.327116+0000","last_fullsized":"2024-09-14T04:21:28.327116+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:34.496354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":527,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327873+0000","last_change":"2024-09-14T04:07:44.334336+0000","last_active":"2024-09-14T04:21:28.327873+0000","last_peered":"2024-09-14T04:21:28.327873+0000","last_clean":"2024-09-14T04:21:28.327873+0000","last_became_active":"2024-09-14T04:07:44.334006+0000","last_became_peered":"2024-09-14T04:07:44.334006+0000","last_unstale":"2024-09-14T04:21:28.327873+0000","last_undegraded":"2024-09-14T04:21:28.327873+0000","last_fullsized":"2024-09-14T04:21:28.327873+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:56:20.968184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":442,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.325932+0000","last_change":"2024-09-14T03:58:45.822337+0000","last_active":"2024-09-14T04:21:28.325932+0000","last_peered":"2024-09-14T04:21:28.325932+0000","last_clean":"2024-09-14T04:21:28.325932+0000","last_became_active":"2024-09-14T03:58:45.821612+0000","last_became_peered":"2024-09-14T03:58:45.821612+0000","last_unstale":"2024-09-14T04:21:28.325932+0000","last_undegraded":"2024-09-14T04:21:28.325932+0000","last_fullsized":"2024-09-14T04:21:28.325932+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:47.771256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":290,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.282748+0000","last_change":"2024-09-14T03:55:58.252540+0000","last_active":"2024-09-14T04:21:20.282748+0000","last_peered":"2024-09-14T04:21:20.282748+0000","last_clean":"2024-09-14T04:21:20.282748+0000","last_became_active":"2024-09-14T03:55:58.251743+0000","last_became_peered":"2024-09-14T03:55:58.251743+0000","last_unstale":"2024-09-14T04:21:20.282748+0000","last_undegraded":"2024-09-14T04:21:20.282748+0000","last_fullsized":"2024-09-14T04:21:20.282748+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:33:23.443009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":255,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.326988+0000","last_change":"2024-09-14T04:07:44.339429+0000","last_active":"2024-09-14T04:21:28.326988+0000","last_peered":"2024-09-14T04:21:28.326988+0000","last_clean":"2024-09-14T04:21:28.326988+0000","last_became_active":"2024-09-14T04:07:44.338382+0000","last_became_peered":"2024-09-14T04:07:44.338382+0000","last_unstale":"2024-09-14T04:21:28.326988+0000","last_undegraded":"2024-09-14T04:21:28.326988+0000","last_fullsized":"2024-09-14T04:21:28.326988+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:59.231310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":529,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.331703+0000","last_change":"2024-09-14T04:07:44.424047+0000","last_active":"2024-09-14T04:21:28.331703+0000","last_peered":"2024-09-14T04:21:28.331703+0000","last_clean":"2024-09-14T04:21:28.331703+0000","last_became_active":"2024-09-14T04:07:44.423705+0000","last_became_peered":"2024-09-14T04:07:44.423705+0000","last_unstale":"2024-09-14T04:21:28.331703+0000","last_undegraded":"2024-09-14T04:21:28.331703+0000","last_fullsized":"2024-09-14T04:21:28.331703+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:26:11.540619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":492,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.331077+0000","last_change":"2024-09-14T04:13:37.163535+0000","last_active":"2024-09-14T04:21:28.331077+0000","last_peered":"2024-09-14T04:21:28.331077+0000","last_clean":"2024-09-14T04:21:28.331077+0000","last_became_active":"2024-09-14T04:13:37.163164+0000","last_became_peered":"2024-09-14T04:13:37.163164+0000","last_unstale":"2024-09-14T04:21:28.331077+0000","last_undegraded":"2024-09-14T04:21:28.331077+0000","last_fullsized":"2024-09-14T04:21:28.331077+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:13:25.324074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":262,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.331060+0000","last_change":"2024-09-14T04:13:37.162587+0000","last_active":"2024-09-14T04:21:28.331060+0000","last_peered":"2024-09-14T04:21:28.331060+0000","last_clean":"2024-09-14T04:21:28.331060+0000","last_became_active":"2024-09-14T04:13:37.162219+0000","last_became_peered":"2024-09-14T04:13:37.162219+0000","last_unstale":"2024-09-14T04:21:28.331060+0000","last_undegraded":"2024-09-14T04:21:28.331060+0000","last_fullsized":"2024-09-14T04:21:28.331060+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:52:47.043189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":73,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.281489+0000","last_change":"2024-09-14T04:13:37.164992+0000","last_active":"2024-09-14T04:21:20.281489+0000","last_peered":"2024-09-14T04:21:20.281489+0000","last_clean":"2024-09-14T04:21:20.281489+0000","last_became_active":"2024-09-14T04:13:37.164654+0000","last_became_peered":"2024-09-14T04:13:37.164654+0000","last_unstale":"2024-09-14T04:21:20.281489+0000","last_undegraded":"2024-09-14T04:21:20.281489+0000","last_fullsized":"2024-09-14T04:21:20.281489+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:49.248470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":521,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.330104+0000","last_change":"2024-09-14T04:07:44.338452+0000","last_active":"2024-09-14T04:21:28.330104+0000","last_peered":"2024-09-14T04:21:28.330104+0000","last_clean":"2024-09-14T04:21:28.330104+0000","last_became_active":"2024-09-14T04:07:44.338148+0000","last_became_peered":"2024-09-14T04:07:44.338148+0000","last_unstale":"2024-09-14T04:21:28.330104+0000","last_undegraded":"2024-09-14T04:21:28.330104+0000","last_fullsized":"2024-09-14T04:21:28.330104+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:17:20.658215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":518,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327456+0000","last_change":"2024-09-14T04:13:37.166866+0000","last_active":"2024-09-14T04:21:28.327456+0000","last_peered":"2024-09-14T04:21:28.327456+0000","last_clean":"2024-09-14T04:21:28.327456+0000","last_became_active":"2024-09-14T04:13:37.166453+0000","last_became_peered":"2024-09-14T04:13:37.166453+0000","last_unstale":"2024-09-14T04:21:28.327456+0000","last_undegraded":"2024-09-14T04:21:28.327456+0000","last_fullsized":"2024-09-14T04:21:28.327456+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:34:59.229028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":346,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.326656+0000","last_change":"2024-09-14T04:17:48.085379+0000","last_active":"2024-09-14T04:21:28.326656+0000","last_peered":"2024-09-14T04:21:28.326656+0000","last_clean":"2024-09-14T04:21:28.326656+0000","last_became_active":"2024-09-14T04:17:48.084944+0000","last_became_peered":"2024-09-14T04:17:48.084944+0000","last_unstale":"2024-09-14T04:21:28.326656+0000","last_undegraded":"2024-09-14T04:21:28.326656+0000","last_fullsized":"2024-09-14T04:21:28.326656+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:43:26.386934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":311,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286381+0000","last_change":"2024-09-14T04:13:37.174369+0000","last_active":"2024-09-14T04:21:20.286381+0000","last_peered":"2024-09-14T04:21:20.286381+0000","last_clean":"2024-09-14T04:21:20.286381+0000","last_became_active":"2024-09-14T04:13:37.165108+0000","last_became_peered":"2024-09-14T04:13:37.165108+0000","last_unstale":"2024-09-14T04:21:20.286381+0000","last_undegraded":"2024-09-14T04:21:20.286381+0000","last_fullsized":"2024-09-14T04:21:20.286381+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:43.400463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":346,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.326528+0000","last_change":"2024-09-14T04:17:48.081862+0000","last_active":"2024-09-14T04:21:28.326528+0000","last_peered":"2024-09-14T04:21:28.326528+0000","last_clean":"2024-09-14T04:21:28.326528+0000","last_became_active":"2024-09-14T04:17:48.081095+0000","last_became_peered":"2024-09-14T04:17:48.081095+0000","last_unstale":"2024-09-14T04:21:28.326528+0000","last_undegraded":"2024-09-14T04:21:28.326528+0000","last_fullsized":"2024-09-14T04:21:28.326528+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:04:07.654057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":294,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.286435+0000","last_change":"2024-09-14T04:13:37.168004+0000","last_active":"2024-09-14T04:21:20.286435+0000","last_peered":"2024-09-14T04:21:20.286435+0000","last_clean":"2024-09-14T04:21:20.286435+0000","last_became_active":"2024-09-14T04:13:37.159695+0000","last_became_peered":"2024-09-14T04:13:37.159695+0000","last_unstale":"2024-09-14T04:21:20.286435+0000","last_undegraded":"2024-09-14T04:21:20.286435+0000","last_fullsized":"2024-09-14T04:21:20.286435+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:18:40.779932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":255,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327035+0000","last_change":"2024-09-14T04:07:44.341005+0000","last_active":"2024-09-14T04:21:28.327035+0000","last_peered":"2024-09-14T04:21:28.327035+0000","last_clean":"2024-09-14T04:21:28.327035+0000","last_became_active":"2024-09-14T04:07:44.340388+0000","last_became_peered":"2024-09-14T04:07:44.340388+0000","last_unstale":"2024-09-14T04:21:28.327035+0000","last_undegraded":"2024-09-14T04:21:28.327035+0000","last_fullsized":"2024-09-14T04:21:28.327035+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:51:33.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":256,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.325988+0000","last_change":"2024-09-14T04:07:44.335509+0000","last_active":"2024-09-14T04:21:28.325988+0000","last_peered":"2024-09-14T04:21:28.325988+0000","last_clean":"2024-09-14T04:21:28.325988+0000","last_became_active":"2024-09-14T04:07:44.335157+0000","last_became_peered":"2024-09-14T04:07:44.335157+0000","last_unstale":"2024-09-14T04:21:28.325988+0000","last_undegraded":"2024-09-14T04:21:28.325988+0000","last_fullsized":"2024-09-14T04:21:28.325988+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:54.946259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":73,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-14T04:21:20.285733+0000","last_change":"2024-09-14T04:13:37.165840+0000","last_active":"2024-09-14T04:21:20.285733+0000","last_peered":"2024-09-14T04:21:20.285733+0000","last_clean":"2024-09-14T04:21:20.285733+0000","last_became_active":"2024-09-14T04:13:37.165508+0000","last_became_peered":"2024-09-14T04:13:37.165508+0000","last_unstale":"2024-09-14T04:21:20.285733+0000","last_undegraded":"2024-09-14T04:21:20.285733+0000","last_fullsized":"2024-09-14T04:21:20.285733+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:24:02.783651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":518,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-14T04:21:28.327589+0000","last_change":"2024-09-14T03:55:58.254976+0000","last_active":"2024-09-14T04:21:28.327589+0000","last_peered":"2024-09-14T04:21:28.327589+0000","last_clean":"2024-09-14T04:21:28.327589+0000","last_became_active":"2024-09-14T03:55:58.254417+0000","last_became_peered":"2024-09-14T03:55:58.254417+0000","last_unstale":"2024-09-14T04:21:28.327589+0000","last_undegraded":"2024-09-14T04:21:28.327589+0000","last_fullsized":"2024-09-14T04:21:28.327589+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:05:39.897902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":235954,"num_write_kb":7550465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141454,"ondisk_log_size":141454,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":4834,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430695,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1195752,"kb_used_data":4320,"kb_used_omap":1,"kb_used_meta":1191422,"kb_avail":92549400,"statfs":{"total":95995035648,"available":94770585600,"internally_reserved":0,"allocated":4423680,"data_stored":2989239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1220016472},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626924,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905820,"kb_used_data":4308,"kb_used_omap":1,"kb_used_meta":901502,"kb_avail":92839332,"statfs":{"total":95995035648,"available":95067475968,"internally_reserved":0,"allocated":4411392,"data_stored":2988797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923138392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823152,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1201548,"kb_used_data":3716,"kb_used_omap":1,"kb_used_meta":1197822,"kb_avail":92543604,"statfs":{"total":95995035648,"available":94764650496,"internally_reserved":0,"allocated":3805184,"data_stored":2393940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1226570072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917234,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":346204,"kb_used_data":3732,"kb_used_omap":6,"kb_used_meta":342457,"kb_avail":93398948,"statfs":{"total":95995035648,"available":95640522752,"internally_reserved":0,"allocated":3821568,"data_stored":2398486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":350676991},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552533,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":869516,"kb_used_data":3716,"kb_used_omap":4,"kb_used_meta":865787,"kb_avail":92875636,"statfs":{"total":95995035648,"available":95104651264,"internally_reserved":0,"allocated":3805184,"data_stored":2394021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":886566333},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435574,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845512,"kb_used_data":3712,"kb_used_omap":6,"kb_used_meta":841785,"kb_avail":92899640,"statfs":{"total":95995035648,"available":95129231360,"internally_reserved":0,"allocated":3801088,"data_stored":2393073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":861988301},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828044,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":925388,"kb_used_data":3716,"kb_used_omap":5,"kb_used_meta":921658,"kb_avail":92819764,"statfs":{"total":95995035648,"available":95047438336,"internally_reserved":0,"allocated":3805184,"data_stored":2393950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":943778031},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597384032,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1087120,"kb_used_data":4296,"kb_used_omap":5,"kb_used_meta":1082810,"kb_avail":92658032,"statfs":{"total":95995035648,"available":94881824768,"internally_reserved":0,"allocated":4399104,"data_stored":2984592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":1108797500},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:21:43.955 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-14T04:21:43.956 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:21:43.956 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:21:43.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd out 5 2024-09-14T04:21:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:43.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:43.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:44.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:44.085 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:44.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:44.186 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:44.212 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:44.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:44.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:44.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:44.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:44.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:44.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:44.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:44.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:44.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:44.596 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:44.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:44.615 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:44 smithi114 ceph-mon[103029]: pgmap v1228: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 199 op/s 2024-09-14T04:21:44.615 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:44 smithi114 ceph-mon[106969]: pgmap v1228: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 199 op/s 2024-09-14T04:21:44.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-14T04:21:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:44 smithi160 ceph-mon[88407]: pgmap v1228: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 199 op/s 2024-09-14T04:21:44.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:44.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:44.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:44.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:44.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:44.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:44.883 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:44.894 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:44.900 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:44.959 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:44.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:44.981 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:44.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:45.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:45.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:45.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:45.230 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:45.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:45.299 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:45.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:45.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:45.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:45.380 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_7' removed 2024-09-14T04:21:45.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:45.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:45.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:45.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:45.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:45.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:45.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:45.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:45 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3842278313' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:21:45.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:45 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:21:45.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:45 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3842278313' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:21:45.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:45 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:21:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:45 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3842278313' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:21:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:45 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:21:45.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:45.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:45.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:45.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:45.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:45.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:45.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:45.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:45.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:45.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:46.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:46.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:46.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:46.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:46.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:46.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:46.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:46.344 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:46.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:46 smithi114 ceph-mon[103029]: pgmap v1229: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:46 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:21:46.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:46 smithi114 ceph-mon[103029]: osdmap e317: 8 total, 8 up, 8 in 2024-09-14T04:21:46.420 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:46 smithi114 ceph-mon[106969]: pgmap v1229: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:46.420 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:46 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:21:46.420 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:46 smithi114 ceph-mon[106969]: osdmap e317: 8 total, 8 up, 8 in 2024-09-14T04:21:46.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:46.466 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:46.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:46.568 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:46.589 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:46.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:46 smithi160 ceph-mon[88407]: pgmap v1229: 89 pgs: 89 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:46 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:21:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:46 smithi160 ceph-mon[88407]: osdmap e317: 8 total, 8 up, 8 in 2024-09-14T04:21:46.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:46.715 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:46.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:46.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:46.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:46.840 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:46.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:46.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:46.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:46.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:47.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:47.089 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:47.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:47.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:47.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:47.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:47.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:47.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:47.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:47.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:47.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:47.475 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:47.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:47.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:47.604 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:47.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:47.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:47.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:47.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:47.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:47.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:47.852 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:47.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:47.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:47.978 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:47.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:47.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:48.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:48.102 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:48.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:48.162 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-14T04:21:48.163 INFO:tasks.radosbench:Beginning radosbench... 2024-09-14T04:21:48.163 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_8 2024-09-14T04:21:48.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_8 16 2024-09-14T04:21:48.204 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:48.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:48.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:48.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:48.353 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:48.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:48.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:48.384 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:21:48.396 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:48 smithi114 ceph-mon[106969]: pgmap v1231: 73 pgs: 73 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:48.396 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:48 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1904001112' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:21:48.396 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:48 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:21:48.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:48 smithi114 ceph-mon[103029]: pgmap v1231: 73 pgs: 73 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:48.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:48 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1904001112' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:21:48.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:48 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:21:48.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:48.481 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:48.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:48.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:48.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:48.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:48 smithi160 ceph-mon[88407]: pgmap v1231: 73 pgs: 73 active+clean; 582 KiB data, 7.0 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:48 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1904001112' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:21:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:48 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:21:48.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:48.733 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:48.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:48.834 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:48.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:48.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:48.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:48.988 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:48.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:49.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:49.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:49.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:49.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:49.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:49.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:49.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:49.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:49.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:49.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:49.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:49.498 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:49.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:49.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:49.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:49 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:49.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:49 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:21:49.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:49 smithi114 ceph-mon[103029]: osdmap e318: 8 total, 8 up, 8 in 2024-09-14T04:21:49.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:49 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:49.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:49 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:21:49.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:49 smithi114 ceph-mon[106969]: osdmap e318: 8 total, 8 up, 8 in 2024-09-14T04:21:49.623 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:49.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:49.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:49 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:21:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:49 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:21:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:49 smithi160 ceph-mon[88407]: osdmap e318: 8 total, 8 up, 8 in 2024-09-14T04:21:49.725 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:49.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:49.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:49.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:49.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:49.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:49.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:50.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:50.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:50.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:50.128 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:50.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:50.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:50.252 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:50.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:50.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:50.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:50.379 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:50.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:50.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:50.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:50.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:50.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:50 smithi114 ceph-mon[103029]: pgmap v1233: 73 pgs: 73 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:50.540 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:50 smithi114 ceph-mon[106969]: pgmap v1233: 73 pgs: 73 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:50.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:50.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:50.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:50 smithi160 ceph-mon[88407]: pgmap v1233: 73 pgs: 73 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:21:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:21:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:21:50.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:50.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:50.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:50.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:50.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:50.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:50.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:51.015 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:51.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:51.108 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:21:51.035+0000 7f3162f23640 -1 mon.a@0(leader).osd e318 definitely_dead 0 2024-09-14T04:21:51.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:51.142 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:51.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:51.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:51.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:51.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:51.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:21:51.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:51.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:51.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:51.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:51.397 INFO:teuthology.orchestra.run.smithi114.stderr:marked out osd.5. 2024-09-14T04:21:51.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:51.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:51.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:51.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:51.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:51.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:51 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/703415842' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-14T04:21:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:51 smithi160 ceph-mon[88407]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-14T04:21:51.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:51.766 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:51 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/703415842' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-14T04:21:51.766 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:51 smithi114 ceph-mon[106969]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-14T04:21:51.767 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:21:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:21:51] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-14T04:21:51.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/703415842' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-14T04:21:51.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:51 smithi114 ceph-mon[103029]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-14T04:21:51.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:51.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:51.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:51.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:51.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:51.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:51.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:52.027 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:52.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:52.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:52.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:52.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:52.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:52.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:52.273 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:52.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:52.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:52.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:52.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:52.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:52.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:52.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:52.522 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:52.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:52.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:52 smithi114 ceph-mon[103029]: pgmap v1234: 73 pgs: 73 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:52 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/703415842' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-14T04:21:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:52 smithi114 ceph-mon[103029]: osdmap e319: 8 total, 8 up, 7 in 2024-09-14T04:21:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:52 smithi114 ceph-mon[106969]: pgmap v1234: 73 pgs: 73 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:52 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/703415842' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-14T04:21:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:52 smithi114 ceph-mon[106969]: osdmap e319: 8 total, 8 up, 7 in 2024-09-14T04:21:52.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:52.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:52.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:52.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:52 smithi160 ceph-mon[88407]: pgmap v1234: 73 pgs: 73 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:21:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:52 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/703415842' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-14T04:21:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:52 smithi160 ceph-mon[88407]: osdmap e319: 8 total, 8 up, 7 in 2024-09-14T04:21:52.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:52.774 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:52.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:52.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:52.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:52.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:52.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:52.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:53.022 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:53.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:53.083 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:21:53.083 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:21:53.083 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:21:53.083 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:21:53.083 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:21:53.083 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:21:53.091 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:53.092 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:21:53.092 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:21:53.117 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:21:53.118 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:21:53.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:53.134 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:21:53.135 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:21:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:53.147 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:53.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:53.149 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:21:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:21:53.166 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:21:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:21:53.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:53.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:53.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:53.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:53.398 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:53.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:53.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:53.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:53.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:53.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:53.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:53 smithi114 ceph-mon[106969]: osdmap e320: 8 total, 8 up, 7 in 2024-09-14T04:21:53.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:53.632 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:53.633 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3145579566' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:53.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:53 smithi114 ceph-mon[103029]: osdmap e320: 8 total, 8 up, 7 in 2024-09-14T04:21:53.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:53.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:53.633 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3145579566' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:53.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:53.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:53 smithi160 ceph-mon[88407]: osdmap e320: 8 total, 8 up, 7 in 2024-09-14T04:21:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:21:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:21:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3145579566' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:21:53.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:53.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:53.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:53.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:53.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:53.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:53.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:54.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:54.037 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:54.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:54.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:54.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:54.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:54.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:54.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:54.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:54.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:54.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:54.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:54.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:54.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:54.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:54.558 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:54 smithi114 ceph-mon[106969]: pgmap v1237: 73 pgs: 2 peering, 71 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:21:54.558 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:54 smithi114 ceph-mon[106969]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-14T04:21:54.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:54 smithi114 ceph-mon[103029]: pgmap v1237: 73 pgs: 2 peering, 71 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:21:54.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:54 smithi114 ceph-mon[103029]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-14T04:21:54.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:54 smithi160 ceph-mon[88407]: pgmap v1237: 73 pgs: 2 peering, 71 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:21:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:54 smithi160 ceph-mon[88407]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-14T04:21:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:54.669 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:54.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:54.759 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:54.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:54.779 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:21:54.779 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:21:54.779 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-14T04:21:54.780 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:21:54.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:54.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:54.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:54.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:54.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:54.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:55.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:55.044 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:55.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:55.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:55.168 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:55.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:55.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:55.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:55.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:55.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:55.418 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:55.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:55.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:55.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:55.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:55.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:55.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:55.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:55.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:55.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:55.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:55.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-14T04:21:55.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:55.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:55.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:56.035 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:56.061 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:56.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:56.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:56.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:56.187 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:56.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:56.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:56.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:56.312 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:56.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:56.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:56.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:56.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:56.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:56.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:56.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:56.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:21:56.662 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:56 smithi114 ceph-mon[103029]: pgmap v1238: 73 pgs: 2 peering, 71 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 98 B/s, 1 objects/s recovering 2024-09-14T04:21:56.662 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:56 smithi114 ceph-mon[106969]: pgmap v1238: 73 pgs: 2 peering, 71 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 98 B/s, 1 objects/s recovering 2024-09-14T04:21:56.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:56.694 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:56.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:56.796 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:56.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:56.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:56 smithi160 ceph-mon[88407]: pgmap v1238: 73 pgs: 2 peering, 71 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 98 B/s, 1 objects/s recovering 2024-09-14T04:21:56.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:56.945 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:56.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:56.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:57.047 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:21:57.073 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:57.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:21:57.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:57.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:57.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:57.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:57.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:57.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:57.416 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_8' created 2024-09-14T04:21:57.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:57.453 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:57.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:57.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:57.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:57.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:57.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:57.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:57.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:57.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:57.782 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:57.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:57.819 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 4] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:21:57.820 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:21:57.820 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-14T04:21:57.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd in 5 2024-09-14T04:21:57.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:57.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:57.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:57.848 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:57 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2283049262' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-14T04:21:57.849 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:57 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2283049262' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-14T04:21:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:57 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2283049262' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-14T04:21:57.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:57.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:57.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:58.054 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:58.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:58.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:58.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:58.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:58.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:58.309 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:58.332 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:58.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:58.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:58.460 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:58.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:58.550 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:58 smithi114 ceph-mon[103029]: pgmap v1239: 73 pgs: 73 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 82 B/s, 0 objects/s recovering 2024-09-14T04:21:58.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:58 smithi114 ceph-mon[103029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-14T04:21:58.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:58 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2283049262' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-09-14T04:21:58.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:58 smithi114 ceph-mon[103029]: osdmap e321: 8 total, 8 up, 7 in 2024-09-14T04:21:58.551 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:58 smithi114 ceph-mon[106969]: pgmap v1239: 73 pgs: 73 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 82 B/s, 0 objects/s recovering 2024-09-14T04:21:58.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:58 smithi114 ceph-mon[106969]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-14T04:21:58.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:58 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2283049262' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-09-14T04:21:58.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:58 smithi114 ceph-mon[106969]: osdmap e321: 8 total, 8 up, 7 in 2024-09-14T04:21:58.562 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:58.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_8 rados --yes-i-really-mean-it || true 2024-09-14T04:21:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:58.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:58.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:58.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:58.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:58.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:58.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:58.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:58.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:58.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:58 smithi160 ceph-mon[88407]: pgmap v1239: 73 pgs: 73 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 82 B/s, 0 objects/s recovering 2024-09-14T04:21:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:58 smithi160 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-14T04:21:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:58 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2283049262' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-09-14T04:21:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:58 smithi160 ceph-mon[88407]: osdmap e321: 8 total, 8 up, 7 in 2024-09-14T04:21:58.909 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:58.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:21:58.952 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:21:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:58.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:58.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:21:59.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:21:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:59.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:21:59.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:21:59.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:59.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:59.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:21:59.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:21:59.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:21:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:59.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:21:59.432 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:21:59.438 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:21:59.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:59 smithi114 ceph-mon[106969]: osdmap e322: 8 total, 8 up, 7 in 2024-09-14T04:21:59.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:59 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3531440227' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:21:59.461 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:21:59 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:21:59.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:59 smithi114 ceph-mon[103029]: osdmap e322: 8 total, 8 up, 7 in 2024-09-14T04:21:59.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:59 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3531440227' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:21:59.462 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:21:59 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:21:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:59.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:59.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:21:59.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:59.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:59.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:59.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:59.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:59.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:21:59.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:21:59.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:59.840 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:59.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:21:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:59 smithi160 ceph-mon[88407]: osdmap e322: 8 total, 8 up, 7 in 2024-09-14T04:21:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:59 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3531440227' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:21:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:21:59 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:21:59.941 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:21:59.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:21:59.966 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:21:59.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:00.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:00.091 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:00.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:00.192 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:00.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:00.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:00.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:00.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:00.341 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:00.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:00.443 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:00.467 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:00.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:00.569 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:00.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:00.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:22:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:22:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:22:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:00 smithi160 ceph-mon[88407]: pgmap v1242: 89 pgs: 6 creating+peering, 8 unknown, 75 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 98 B/s, 1 objects/s recovering 2024-09-14T04:22:00.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:00 smithi160 ceph-mon[88407]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:22:00.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:00 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:00.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:00 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:22:00.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:00 smithi160 ceph-mon[88407]: osdmap e323: 8 total, 8 up, 7 in 2024-09-14T04:22:00.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:00.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:00.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:00.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:00.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:00.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:00 smithi114 ceph-mon[103029]: pgmap v1242: 89 pgs: 6 creating+peering, 8 unknown, 75 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 98 B/s, 1 objects/s recovering 2024-09-14T04:22:00.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:00 smithi114 ceph-mon[103029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:22:00.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:00 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:00.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:00 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:22:00.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:00 smithi114 ceph-mon[103029]: osdmap e323: 8 total, 8 up, 7 in 2024-09-14T04:22:00.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:00 smithi114 ceph-mon[106969]: pgmap v1242: 89 pgs: 6 creating+peering, 8 unknown, 75 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 98 B/s, 1 objects/s recovering 2024-09-14T04:22:00.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:00 smithi114 ceph-mon[106969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:22:00.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:00 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:00.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:00 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:22:00.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:00 smithi114 ceph-mon[106969]: osdmap e323: 8 total, 8 up, 7 in 2024-09-14T04:22:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:00.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:00.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:00.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:00.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:00.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:01.036 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:01.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:01.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:01.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:01.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:01.226 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:01.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:01.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:01.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:01.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:01.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:01.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:01.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:01.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:01.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:01.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:01.705 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:01.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:01.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:01.782 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:22:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:22:01] "GET /metrics HTTP/1.1" 200 34157 "" "Prometheus/2.51.0" 2024-09-14T04:22:01.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:01.852 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:01.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:01.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:01.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:01.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:02.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:02.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:02.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:02.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:02.229 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:02.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:02.331 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:02.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:02.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:02.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:02.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:02.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:02.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:02.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:02.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:02.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:02.739 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:02.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:02.760 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:02 smithi114 ceph-mon[103029]: pgmap v1244: 89 pgs: 6 creating+peering, 3 unknown, 80 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-14T04:22:02.761 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:02 smithi114 ceph-mon[106969]: pgmap v1244: 89 pgs: 6 creating+peering, 3 unknown, 80 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-14T04:22:02.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:02.862 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:02.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:02 smithi160 ceph-mon[88407]: pgmap v1244: 89 pgs: 6 creating+peering, 3 unknown, 80 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-14T04:22:02.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:02.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:02.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:02.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:03.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:03.114 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:03.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:03.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:03.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:03.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:03.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:03.365 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:03.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:03.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:03.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:03.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:03.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:03.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:03.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:03.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:03.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:03.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:03.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:03.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:22:03.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:03.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:03.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:03.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:03.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:03.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:04.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:04.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:04.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:04.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:04.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:04.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:04.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:04.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:04.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:04.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:04.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:04.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:04.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:04.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:04.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:04.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:04.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:04.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:04.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:04.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:04.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:04.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:04.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:04.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:04 smithi114 ceph-mon[103029]: pgmap v1245: 89 pgs: 4 creating+peering, 85 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:22:04.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:04 smithi114 ceph-mon[106969]: pgmap v1245: 89 pgs: 4 creating+peering, 85 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:22:04.872 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:04.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:04 smithi160 ceph-mon[88407]: pgmap v1245: 89 pgs: 4 creating+peering, 85 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:22:04.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:04.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:04.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:05.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:05.120 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:05.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:05.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:05.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:05.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:05.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:05.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:05.367 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:05.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:05.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:05.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:05.492 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:05.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:05.593 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:05.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:05.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:05.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:05.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:05.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:05.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:05.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:05.786 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:05 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1481566795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:05.786 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:05 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1481566795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:05.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:05.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:05.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:05.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:22:05.884 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:22:05.884 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:22:05.884 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:22:05.885 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:22:05.885 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:22:05.885 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:22:05.885 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:22:05.885 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:22:05.885 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:22:05.885 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:22:05.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:22:05.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:22:05.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:22:05.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:22:05.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:22:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:05 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1481566795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:05.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:05.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:05.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:05.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:06.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:06.120 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:06.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:06.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:06.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:06.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:06.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:06.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:06.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:06.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:06.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:06.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:06.499 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:06.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:06.565 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:06 smithi114 ceph-mon[106969]: pgmap v1246: 89 pgs: 89 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:22:06.565 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:06 smithi114 ceph-mon[103029]: pgmap v1246: 89 pgs: 89 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:22:06.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:06.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:06.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:06.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:06.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:06.754 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:06.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:06.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:06.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:06 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:22:06.710+0000 7f3162f23640 -1 mon.a@0(leader).osd e323 definitely_dead 0 2024-09-14T04:22:06.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:06.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:06 smithi160 ceph-mon[88407]: pgmap v1246: 89 pgs: 89 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:22:06.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:07.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:07.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:07.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:07.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:07.123 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:07.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:07.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:07.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:07.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:07.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:07.318 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:22:07.319 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:22:07.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:07.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:07.370 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:07.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:07.450 INFO:teuthology.orchestra.run.smithi114.stderr:marked in osd.5. 2024-09-14T04:22:07.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:07.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:07.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:07.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:07.595 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:07.616 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:07.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:07.696 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:07 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3995861803' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-14T04:22:07.697 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:07 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-14T04:22:07.697 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:22:07.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:07.740 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:07.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:07.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:07.815 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:07 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3995861803' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-14T04:22:07.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:07 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-14T04:22:07.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:22:07.816 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:07 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3995861803' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-14T04:22:07.816 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:07 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-14T04:22:07.816 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:22:07.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:07.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:07.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:07.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:07.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:07.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:07.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:07.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:08.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:08.112 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:08.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:08.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:08.220 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-14T04:22:08.239 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:08.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:08.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:08.363 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:08.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:08.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:08.478 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:08.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:08.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:08.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:08.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:08.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:08.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:08.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:08.738 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:08.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:08.751 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:08 smithi160 ceph-mon[88407]: pgmap v1247: 89 pgs: 89 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:22:08.751 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:08 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-14T04:22:08.751 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:08 smithi160 ceph-mon[88407]: osdmap e324: 8 total, 8 up, 8 in 2024-09-14T04:22:08.751 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:08.751 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:08.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:08 smithi114 ceph-mon[106969]: pgmap v1247: 89 pgs: 89 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:22:08.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:08 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-14T04:22:08.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:08 smithi114 ceph-mon[106969]: osdmap e324: 8 total, 8 up, 8 in 2024-09-14T04:22:08.783 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:08.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:08.784 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:08 smithi114 ceph-mon[103029]: pgmap v1247: 89 pgs: 89 active+clean; 582 KiB data, 5.9 GiB used, 620 GiB / 626 GiB avail 2024-09-14T04:22:08.784 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:08 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-14T04:22:08.784 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:08 smithi114 ceph-mon[103029]: osdmap e324: 8 total, 8 up, 8 in 2024-09-14T04:22:08.784 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:08.784 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:08.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:08.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:08.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:08.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:08.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:09.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:09.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:22:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:09.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:09.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:09.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:09.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-14T04:22:09.234 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:09.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:09.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:09.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:09.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:09.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:09.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:09.462 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_8' 2024-09-14T04:22:09.482 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:09.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:09.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:09.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:09.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:09.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:09.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:09.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:09.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:09 smithi114 ceph-mon[103029]: osdmap e325: 8 total, 8 up, 8 in 2024-09-14T04:22:09.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:09 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1498392177' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:22:09.803 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:22:09.804 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:09 smithi114 ceph-mon[106969]: osdmap e325: 8 total, 8 up, 8 in 2024-09-14T04:22:09.804 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:09 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1498392177' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:22:09.804 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:22:09.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:09.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:09 smithi160 ceph-mon[88407]: osdmap e325: 8 total, 8 up, 8 in 2024-09-14T04:22:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:09 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1498392177' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:22:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:22:09.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:09.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:09.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:10.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:10.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:10.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:10.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:10.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:10.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:10.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:10.346 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:10.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:10.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:10.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:10.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:10.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:10.593 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:10.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:22:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:22:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:22:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:10 smithi160 ceph-mon[88407]: pgmap v1250: 89 pgs: 20 peering, 69 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:22:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:10 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:22:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:10 smithi160 ceph-mon[88407]: osdmap e326: 8 total, 8 up, 8 in 2024-09-14T04:22:10.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:10.716 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:10.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:10.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:10.839 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:10.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:10.866 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:10 smithi114 ceph-mon[106969]: pgmap v1250: 89 pgs: 20 peering, 69 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:22:10.867 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:10 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:22:10.867 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:10 smithi114 ceph-mon[106969]: osdmap e326: 8 total, 8 up, 8 in 2024-09-14T04:22:10.868 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:10 smithi114 ceph-mon[103029]: pgmap v1250: 89 pgs: 20 peering, 69 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail 2024-09-14T04:22:10.868 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:10 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:22:10.868 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:10 smithi114 ceph-mon[103029]: osdmap e326: 8 total, 8 up, 8 in 2024-09-14T04:22:10.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:10.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:10.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:11.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:11.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:11.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:11.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:11.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:11.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:11.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:11.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:11.334 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:11.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:11.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:11.426 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:11.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:11.441 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:11.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:11.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:11.552 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:22:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:22:11] "GET /metrics HTTP/1.1" 200 36503 "" "Prometheus/2.51.0" 2024-09-14T04:22:11.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:11.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:11.585 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:11.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:11.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:11.707 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:11.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:11.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:11.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:11.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:11.892 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_8 bench 90 write' 2024-09-14T04:22:11.894 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-14T04:22:11.894 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-14T04:22:11.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:11.943 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-14T04:22:11.943 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-14T04:22:11.943 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_288325 2024-09-14T04:22:11.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:22:11.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-14T04:22:11.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:11.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:11.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:12.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:12.115 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:12.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:12.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:12.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:12.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:12.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:12.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:12.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:12.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:12.472 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:22:12.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:12.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:12.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:12.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:12.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:12.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:12.726 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:12 smithi160 ceph-mon[88407]: pgmap v1252: 89 pgs: 20 peering, 69 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 80 B/s, 0 objects/s recovering 2024-09-14T04:22:12.726 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:12 smithi160 ceph-mon[88407]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:22:12.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:12 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2566448687' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:12.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:12 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:12.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.727 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[103029]: pgmap v1252: 89 pgs: 20 peering, 69 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 80 B/s, 0 objects/s recovering 2024-09-14T04:22:12.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[103029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:22:12.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2566448687' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:12.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:12.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.759 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.760 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[106969]: pgmap v1252: 89 pgs: 20 peering, 69 active+clean; 582 KiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 80 B/s, 0 objects/s recovering 2024-09-14T04:22:12.760 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.760 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.760 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[106969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:22:12.760 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2566448687' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:12.760 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:12.760 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.760 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:12.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:12.809 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:12.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:12.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:12.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:12.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:12.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:12.943 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 3758 3742 233.862 233.875 0.00616746 0.00426201 2024-09-14T04:22:13.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:13.078 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:13.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:13.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:13.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:13.201 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:13.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:13.221 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:22:13.221 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:22:13.224 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-14T04:22:13.224 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-09-14T04:22:13.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 0 2024-09-14T04:22:13.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:13.343 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:13.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:13.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:13.483 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:13.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:13.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:13.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:13 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:22:13.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:13 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:22:13.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:13 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:22:13.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:13 smithi114 ceph-mon[103029]: osdmap e327: 8 total, 8 up, 8 in 2024-09-14T04:22:13.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:13 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:22:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:13 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:22:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:13 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:22:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:13 smithi114 ceph-mon[106969]: osdmap e327: 8 total, 8 up, 8 in 2024-09-14T04:22:13.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:13.623 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:13.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:13.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:13 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:22:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:13 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:22:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:13 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:22:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:13 smithi160 ceph-mon[88407]: osdmap e327: 8 total, 8 up, 8 in 2024-09-14T04:22:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:13.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:13.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:13.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:13.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:13.905 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:13.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:13.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:13.943 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 6986 6970 217.798 201.75 0.00345588 0.00458185 2024-09-14T04:22:14.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:14.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:14.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:14.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:14.185 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:14.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:14.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:14.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:14.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:14.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:14.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:14.475 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:14.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:14.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:14.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:14 smithi160 ceph-mon[88407]: pgmap v1254: 89 pgs: 18 peering, 71 active+clean; 27 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 4.7 MiB/s wr, 74 op/s; 85 B/s, 0 objects/s recovering 2024-09-14T04:22:14.610 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:14.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:14.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:14 smithi114 ceph-mon[106969]: pgmap v1254: 89 pgs: 18 peering, 71 active+clean; 27 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 4.7 MiB/s wr, 74 op/s; 85 B/s, 0 objects/s recovering 2024-09-14T04:22:14.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:14 smithi114 ceph-mon[103029]: pgmap v1254: 89 pgs: 18 peering, 71 active+clean; 27 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 4.7 MiB/s wr, 74 op/s; 85 B/s, 0 objects/s recovering 2024-09-14T04:22:14.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:14.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:14.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:14.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:14.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:14.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:14.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 10400 10384 216.318 213.375 0.00445922 0.00461647 2024-09-14T04:22:14.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:15.028 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:15.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:15.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:15.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:15.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:15.271 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:15.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:15.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:15.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:15.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:15.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:15.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:15.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:15.576 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:15.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:15.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:15.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:15.712 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:15.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:15.814 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:15.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:15.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:15.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 13569 13553 211.75 198.062 0.00375147 0.00471551 2024-09-14T04:22:15.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:15.985 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:15.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:16.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:16.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:16.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:16.141 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:16.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:16.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:16.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:16.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:16.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:16.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:16.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:16.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:16.537 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:16.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:16.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:16 smithi114 ceph-mon[103029]: pgmap v1255: 89 pgs: 89 active+clean; 287 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 43 MiB/s wr, 692 op/s; 99 B/s, 1 objects/s recovering 2024-09-14T04:22:16.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:16 smithi114 ceph-mon[106969]: pgmap v1255: 89 pgs: 89 active+clean; 287 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 43 MiB/s wr, 692 op/s; 99 B/s, 1 objects/s recovering 2024-09-14T04:22:16.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:16.677 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:16.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:16.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:16.812 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:22:16.812 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:22:16.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:16.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:16 smithi160 ceph-mon[88407]: pgmap v1255: 89 pgs: 89 active+clean; 287 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 43 MiB/s wr, 692 op/s; 99 B/s, 1 objects/s recovering 2024-09-14T04:22:16.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:16.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 16406 16390 204.859 177.312 0.00391815 0.00487549 2024-09-14T04:22:16.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:16.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:16.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:17.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:17.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:17.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:17.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:17.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:17.239 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:17.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:17.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:17.373 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:17.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:17.475 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:17.511 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:17.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:17.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:17.649 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:17.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:17.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:22:17.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:17.825 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:17.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:17.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:17.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 18874 18858 196.422 154.25 0.00208616 0.0050843 2024-09-14T04:22:17.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:17.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:18.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:18.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:18.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:18.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:18.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:18.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:18.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:18.343 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:22:18.343 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:22:18.367 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:18.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:18.375 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:22:18.375 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:22:18.437 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:22:18.437 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:22:18.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:18.470 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:22:18.470 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:22:18.498 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:22:18.498 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:22:18.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:18 smithi114 ceph-mon[103029]: pgmap v1256: 89 pgs: 89 active+clean; 412 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 51 MiB/s wr, 824 op/s; 82 B/s, 1 objects/s recovering 2024-09-14T04:22:18.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:18 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2780949158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:18.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:18 smithi114 ceph-mon[106969]: pgmap v1256: 89 pgs: 89 active+clean; 412 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 51 MiB/s wr, 824 op/s; 82 B/s, 1 objects/s recovering 2024-09-14T04:22:18.499 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:18 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2780949158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:18.503 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:18.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:18.528 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:22:18.528 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:22:18.557 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:22:18.557 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:22:18.587 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:22:18.587 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:22:18.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:18.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:18.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:18.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:18.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:18.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:18.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:18.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:18 smithi160 ceph-mon[88407]: pgmap v1256: 89 pgs: 89 active+clean; 412 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 51 MiB/s wr, 824 op/s; 82 B/s, 1 objects/s recovering 2024-09-14T04:22:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:18 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2780949158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:18.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:18.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:18.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 21228 21212 189.378 147.125 0.00420183 0.00527041 2024-09-14T04:22:19.020 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:19.057 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:19.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:19.158 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:19.192 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:19.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:19.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:19.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:19.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:19.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:19.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:19.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:19.569 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:19.585 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-14T04:22:19.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:19.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:19.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:19.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:19.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:19.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:19.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:19.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:19.886 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:19.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:19.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 23579 23563 184.071 146.938 0.00754978 0.00542772 2024-09-14T04:22:19.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:20.020 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:20.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:20.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:20.112 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:22:20.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:20.161 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:20.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:20.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:20.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:20.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:20.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:20.431 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:20.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:20.471 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:22:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:22:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:22:20.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:20.569 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:20.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:20.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:20.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:20.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:20.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:20.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:20.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:20.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:20.858 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:20 smithi114 ceph-mon[103029]: pgmap v1257: 89 pgs: 89 active+clean; 1.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 68 B/s, 1 objects/s recovering 2024-09-14T04:22:20.858 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:20 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:20.858 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:20 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:20.858 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:20 smithi114 ceph-mon[106969]: pgmap v1257: 89 pgs: 89 active+clean; 1.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 68 B/s, 1 objects/s recovering 2024-09-14T04:22:20.858 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:20 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:20.859 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:20 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:20.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:20 smithi160 ceph-mon[88407]: pgmap v1257: 89 pgs: 89 active+clean; 1.1 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 68 B/s, 1 objects/s recovering 2024-09-14T04:22:20.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:20 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:20.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:20 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:20.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:20.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 26061 26045 180.854 155.125 0.00354996 0.00552298 2024-09-14T04:22:20.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:20.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:21.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:21.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:21.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:21.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:21.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:21.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:21.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:21.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:21.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:21.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:21.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:21.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:21.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:21.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:21.558 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:22:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:22:21] "GET /metrics HTTP/1.1" 200 36503 "" "Prometheus/2.51.0" 2024-09-14T04:22:21.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:21.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:21.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:21.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:21.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:21.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:21.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:21.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:21.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:21.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 28442 28426 177.648 148.812 0.00596869 0.00562365 2024-09-14T04:22:22.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:22.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:22.090 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:22.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:22.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:22.150 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:22:22.150 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:28815 13.b 13:d7679388:::benchmark_data_smithi114_288325_object28814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.136872+0000", 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.011132176000000001, 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011165947000000001, 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 28815 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.151 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136872+0000", 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136872+0000", 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136870+0000", 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136909+0000", 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8532000000000003e-05 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136921+0000", 2024-09-14T04:22:22.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2294e-05 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136928+0000", 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7710000000000001e-06 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.138072+0000", 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001144127 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.138190+0000", 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118267 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,3", 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.138319+0000", 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129488 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.142411+0000", 2024-09-14T04:22:22.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040912199999999996 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:28819 13.b 13:db6b3879:::benchmark_data_smithi114_288325_object28818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.136953+0000", 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.011051119, 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011165959, 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.156 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:22:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 28819 2024-09-14T04:22:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.157 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136953+0000", 2024-09-14T04:22:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136953+0000", 2024-09-14T04:22:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136952+0000", 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136978+0000", 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5853000000000001e-05 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136985+0000", 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0899999999999999e-06 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.136989+0000", 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0600000000000001e-06 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.139527+0000", 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025375100000000002 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.139635+0000", 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108198 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,3", 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.139772+0000", 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000137653 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.143941+0000", 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041684490000000003 2024-09-14T04:22:22.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:28830 13.b 13:d3bc50df:::benchmark_data_smithi114_288325_object28829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.144883+0000", 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0031202529999999999, 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033088219999999999, 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:22.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 28830 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.144883+0000", 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.144883+0000", 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.144882+0000", 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.144919+0000", 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6900999999999998e-05 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.144936+0000", 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7558e-05 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.144943+0000", 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3799999999999999e-06 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145096+0000", 2024-09-14T04:22:22.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015371699999999999 2024-09-14T04:22:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145230+0000", 2024-09-14T04:22:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000133249 2024-09-14T04:22:22.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,3", 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145364+0000", 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013394000000000001 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:28831 13.9 13:9672ff71:::benchmark_data_smithi114_288325_object28830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-14T04:22:22.167 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.144957+0000", 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0030464810000000002, 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033034980000000002, 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 28831 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.144957+0000", 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.144957+0000", 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.144957+0000", 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.144996+0000", 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9517000000000001e-05 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145005+0000", 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8729999999999996e-06 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145010+0000", 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8439999999999997e-06 2024-09-14T04:22:22.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145157+0000", 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014701100000000001 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145339+0000", 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000181669 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,2", 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145452+0000", 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113538 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.148240+0000", 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0027881590000000001 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:28832 13.9 13:97cd315d:::benchmark_data_smithi114_288325_object28831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-14T04:22:22.172 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.145022+0000", 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0029815549999999999, 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033126969999999999, 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 28832 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145022+0000", 2024-09-14T04:22:22.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145022+0000", 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145022+0000", 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145058+0000", 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6578000000000001e-05 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145069+0000", 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.06e-05 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145074+0000", 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7219999999999996e-06 2024-09-14T04:22:22.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146856+0000", 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017815299999999999 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.147009+0000", 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015346500000000001 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,2", 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.147144+0000", 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134568 2024-09-14T04:22:22.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:28834 13.9 13:9af727b9:::benchmark_data_smithi114_288325_object28833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.145080+0000", 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.002923733, 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003330782, 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:22.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 28834 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145080+0000", 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145080+0000", 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145079+0000", 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145108+0000", 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8708e-05 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145116+0000", 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9920000000000007e-06 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145120+0000", 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3150000000000002e-06 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.148265+0000", 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031445930000000002 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.148365+0000", 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9897999999999998e-05 2024-09-14T04:22:22.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:28837 13.9 13.57296599 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.145234+0000", 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0027696579999999999, 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0032412000000000001, 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:22:22.181 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 28837 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145234+0000", 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145234+0000", 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145233+0000", 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145307+0000", 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3291e-05 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145316+0000", 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1980000000000007e-06 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.145320+0000", 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.459e-06 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.46760.0:28815 13.b e327/324)", 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.146576+0000", 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.001427981, 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0019456320000000001, 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146576+0000", 2024-09-14T04:22:22.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146576+0000", 2024-09-14T04:22:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146575+0000", 2024-09-14T04:22:22.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146581+0000", 2024-09-14T04:22:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8100000000000003e-06 2024-09-14T04:22:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146591+0000", 2024-09-14T04:22:22.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0257e-05 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146596+0000", 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3870000000000003e-06 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.46760.0:28815 13.b e327/324)", 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.146615+0000", 2024-09-14T04:22:22.188 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0013885169999999999, 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0019459880000000001, 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146615+0000", 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146615+0000", 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146615+0000", 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146618+0000", 2024-09-14T04:22:22.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2990000000000001e-06 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146624+0000", 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4760000000000003e-06 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146628+0000", 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.67e-06 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.191 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.46760.0:28819 13.b e327/324)", 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.146752+0000", 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0012517380000000001, 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0018494060000000001, 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146752+0000", 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146752+0000", 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146751+0000", 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:22.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146755+0000", 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8700000000000002e-06 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146763+0000", 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2400000000000001e-06 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.146767+0000", 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0690000000000003e-06 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.46760.0:28819 13.b e327/324)", 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.147370+0000", 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00063390099999999997, 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001271728, 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.195 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.147370+0000", 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.147370+0000", 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.147369+0000", 2024-09-14T04:22:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.147374+0000", 2024-09-14T04:22:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.933e-06 2024-09-14T04:22:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.147386+0000", 2024-09-14T04:22:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1853e-05 2024-09-14T04:22:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.147391+0000", 2024-09-14T04:22:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1780000000000002e-06 2024-09-14T04:22:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.46760.0:28830 13.b e327/324)", 2024-09-14T04:22:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:22.148496+0000", 2024-09-14T04:22:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 4294967295.9995079, 2024-09-14T04:22:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018647299999999999, 2024-09-14T04:22:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:22:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.199 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.148496+0000", 2024-09-14T04:22:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.148496+0000", 2024-09-14T04:22:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:22.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.148495+0000", 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.148498+0000", 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2619999999999999e-06 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:22.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.148508+0000", 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2709999999999998e-06 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:22.148513+0000", 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4999999999999999e-06 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:22:22.202 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 12 2024-09-14T04:22:22.203 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:22:22.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:22.244 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:22.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:22.346 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:22.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:22.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:22.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:22.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:22.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:22.572 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.0 primary-affinity to 0 (802) 2024-09-14T04:22:22.620 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:22.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:22.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:22.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:22.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:22.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:22.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:22 smithi114 ceph-mon[103029]: pgmap v1258: 89 pgs: 89 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 8 B/s, 0 objects/s recovering 2024-09-14T04:22:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:22 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/183611872' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-14T04:22:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:22 smithi114 ceph-mon[106969]: pgmap v1258: 89 pgs: 89 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 8 B/s, 0 objects/s recovering 2024-09-14T04:22:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:22 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/183611872' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-14T04:22:22.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:22.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:22:22.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:22.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:22 smithi160 ceph-mon[88407]: pgmap v1258: 89 pgs: 89 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 8 B/s, 0 objects/s recovering 2024-09-14T04:22:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:22 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/183611872' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-14T04:22:22.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 29928 29912 169.941 92.875 0.00152853 0.00571854 2024-09-14T04:22:23.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:23.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:23.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:23.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:23.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:23.149 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:23.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:23.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:23.276 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:23.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:23.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:23.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:23.401 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:23.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:23.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:23.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:23.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:23.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:23.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:23.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:23.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:23.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:23.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/183611872' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-14T04:22:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[106969]: osdmap e328: 8 total, 8 up, 8 in 2024-09-14T04:22:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:23.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:23.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:23.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[106969]: osdmap e329: 8 total, 8 up, 8 in 2024-09-14T04:22:23.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/183611872' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-14T04:22:23.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[103029]: osdmap e328: 8 total, 8 up, 8 in 2024-09-14T04:22:23.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:23.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:23.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:23.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:23 smithi114 ceph-mon[103029]: osdmap e329: 8 total, 8 up, 8 in 2024-09-14T04:22:23.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:23.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:23.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:23.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:23.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:23 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/183611872' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-14T04:22:23.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:23 smithi160 ceph-mon[88407]: osdmap e328: 8 total, 8 up, 8 in 2024-09-14T04:22:23.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:23.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:23.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:23.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:23 smithi160 ceph-mon[88407]: osdmap e329: 8 total, 8 up, 8 in 2024-09-14T04:22:23.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 30419 30403 158.336 30.6875 0.00635045 0.00630992 2024-09-14T04:22:24.010 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:24.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:24.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:24.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:24.179 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:24.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:24.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:24.313 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:24.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:24.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:24.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:24.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:24.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:24.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:24.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:24.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:24.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:24.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:24.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:24.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:24.818 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:24.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:24.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:24.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:24 smithi114 ceph-mon[103029]: pgmap v1260: 89 pgs: 89 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 138 MiB/s wr, 2.20k op/s; 8 B/s, 0 objects/s recovering 2024-09-14T04:22:24.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:24 smithi114 ceph-mon[106969]: pgmap v1260: 89 pgs: 89 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 138 MiB/s wr, 2.20k op/s; 8 B/s, 0 objects/s recovering 2024-09-14T04:22:24.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:24 smithi160 ceph-mon[88407]: pgmap v1260: 89 pgs: 89 active+clean; 1.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 138 MiB/s wr, 2.20k op/s; 8 B/s, 0 objects/s recovering 2024-09-14T04:22:24.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:24.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 31722 31706 152.42 81.4375 0.00513098 0.00655532 2024-09-14T04:22:24.950 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:24.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:25.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:25.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:25.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:25.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:25.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:25.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:25.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:25.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:25.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:25.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:25.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:25.498 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:25.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:25.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:25.625 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:25.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:25.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:25.760 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:25.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:25.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:25.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:25 smithi114 ceph-mon[103029]: Health check failed: Reduced data availability: 3 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-14T04:22:25.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:25 smithi114 ceph-mon[106969]: Health check failed: Reduced data availability: 3 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-14T04:22:25.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:25.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:25.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:25 smithi160 ceph-mon[88407]: Health check failed: Reduced data availability: 3 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-14T04:22:25.944 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 33881 33865 151.171 134.938 0.00708241 0.00660945 2024-09-14T04:22:25.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:26.030 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:26.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:26.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:26.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-14T04:22:26.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:26.163 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:26.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:26.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:26.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:26.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:26.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:26.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:26.445 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:26.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:26.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:26.484 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:22:26.547 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:26.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:26.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:26.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:26.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:26.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:26.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:26.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:26.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:26.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:26.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:26.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:26 smithi160 ceph-mon[88407]: pgmap v1262: 89 pgs: 3 activating, 5 peering, 81 active+clean; 1.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 182 MiB/s wr, 2.90k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-14T04:22:26.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 35490 35474 147.796 100.562 0.00194201 0.00674056 2024-09-14T04:22:26.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:26.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:27.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:27.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:27.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:26 smithi114 ceph-mon[103029]: pgmap v1262: 89 pgs: 3 activating, 5 peering, 81 active+clean; 1.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 182 MiB/s wr, 2.90k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-14T04:22:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:26 smithi114 ceph-mon[106969]: pgmap v1262: 89 pgs: 3 activating, 5 peering, 81 active+clean; 1.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 182 MiB/s wr, 2.90k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-14T04:22:27.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:27.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:27.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:27.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:27.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:27.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:27.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:27.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:27.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:27.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:27.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:27.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:27.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:27.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:27.680 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:27.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:27.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:27.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:27.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:27.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:27.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:27.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 15 37453 37438 146.23 122.75 0.0050663 0.00683325 2024-09-14T04:22:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:27.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:27.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:28.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:28.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:28.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:28.204 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:28.239 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:28.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:28.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:28.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:28.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:28.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:28.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:28.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:28.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:28.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:28.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:28.690 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:22:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:22:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:22:28.691 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:22:28.691 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:22:28.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:28.775 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:28.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:28.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:28 smithi114 ceph-mon[106969]: pgmap v1263: 89 pgs: 3 activating, 5 peering, 81 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.50k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-14T04:22:28.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:28 smithi114 ceph-mon[103029]: pgmap v1263: 89 pgs: 3 activating, 5 peering, 81 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.50k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-14T04:22:28.877 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:28 smithi160 ceph-mon[88407]: pgmap v1263: 89 pgs: 3 activating, 5 peering, 81 active+clean; 1.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.50k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-14T04:22:28.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:28.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:28.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:28.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 39396 39380 144.768 121.375 0.0104026 0.00690193 2024-09-14T04:22:29.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:29.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:29.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:29.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:29.195 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:29.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:29.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:29.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:29.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:29.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:29.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-14T04:22:29.523 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:29.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:29.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:29.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:29.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:29.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:29.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:29.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:29.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:29.932 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:22:29.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 41280 41264 143.266 117.75 0.00882317 0.00697553 2024-09-14T04:22:29.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:29.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:29.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:29 smithi114 ceph-mon[106969]: pgmap v1264: 89 pgs: 89 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-14T04:22:29.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:29 smithi114 ceph-mon[106969]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-09-14T04:22:29.994 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:29 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:22:29.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:29 smithi114 ceph-mon[103029]: pgmap v1264: 89 pgs: 89 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-14T04:22:29.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:29 smithi114 ceph-mon[103029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-09-14T04:22:29.995 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:29 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:22:30.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:30.071 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:29 smithi160 ceph-mon[88407]: pgmap v1264: 89 pgs: 89 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s; 16 KiB/s, 0 objects/s recovering 2024-09-14T04:22:30.071 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:29 smithi160 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 2 pgs peering) 2024-09-14T04:22:30.071 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:29 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:22:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:30.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:30.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:30.230 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:30.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:30.278 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:30.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:30.348 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:22:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:22:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:22:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:30.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:30.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:30.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:30.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:30.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:30.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:30.655 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:30.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:30.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:30.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:30.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:30.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:30.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:30.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:30.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:30.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 15 43110 43095 141.748 114.438 0.00438435 0.00704959 2024-09-14T04:22:31.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:31.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:31.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:31.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:31.230 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:31.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:31.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:31.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:31.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:31.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:31.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:31.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:31.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:31.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:31.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:31.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:31.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:31.791 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:22:31] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-09-14T04:22:31.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:31.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:31.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:31.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:31.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:31.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:31.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:22:31.945090+0000 min lat: 0.00116686 max lat: 1.11607 avg lat: 0.00716358 2024-09-14T04:22:31.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:22:31.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 44656 44640 139.488 96.5625 0.00841205 0.00716358 2024-09-14T04:22:32.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:32.071 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:32.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:32.098 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:32.099 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:32.099 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:31 smithi160 ceph-mon[88407]: pgmap v1265: 89 pgs: 89 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s; 15 KiB/s, 0 objects/s recovering 2024-09-14T04:22:32.099 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:22:32.099 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:22:32.099 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:32.099 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:31 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1436678698' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:22:32.099 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:31 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:22:32.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:32.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:32.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[106969]: pgmap v1265: 89 pgs: 89 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s; 15 KiB/s, 0 objects/s recovering 2024-09-14T04:22:32.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:22:32.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:22:32.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:32.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1436678698' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:22:32.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:22:32.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:32.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:32.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[103029]: pgmap v1265: 89 pgs: 89 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s; 15 KiB/s, 0 objects/s recovering 2024-09-14T04:22:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:22:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:22:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1436678698' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:22:32.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:31 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:22:32.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:32.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:32.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:32.309 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:32.348 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:32.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:32.391 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:22:32.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:32.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:32.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:32.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:32.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:32.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:32.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:32.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:32.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:32.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:32.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:32.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:32.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:32.882 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:32.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:32.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 46199 46183 137.438 96.4375 0.00821123 0.00726969 2024-09-14T04:22:32.985 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:33.023 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:33.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:33.088 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:32 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:33.089 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:32 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:22:33.089 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:32 smithi160 ceph-mon[88407]: osdmap e330: 8 total, 8 up, 8 in 2024-09-14T04:22:33.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:32 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:33.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:32 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:22:33.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:32 smithi114 ceph-mon[106969]: osdmap e330: 8 total, 8 up, 8 in 2024-09-14T04:22:33.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:32 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:33.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:32 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:22:33.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:32 smithi114 ceph-mon[103029]: osdmap e330: 8 total, 8 up, 8 in 2024-09-14T04:22:33.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:33.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:33.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:33.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:33.297 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:33.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:33.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:33.431 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:33.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:33.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:33.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:33.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:33.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:33.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:33.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:33.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:33.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:33.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:33.924 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:33.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 47434 47418 134.699 77.1875 0.00247459 0.0073985 2024-09-14T04:22:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:33.946 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:33.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:34.047 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:34.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:34.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:34.092 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:22:34.092 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:22:34.092 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 6, 7, 4, 5] 2024-09-14T04:22:34.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd out 2 2024-09-14T04:22:34.115 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:33 smithi114 ceph-mon[103029]: pgmap v1267: 89 pgs: 89 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-14T04:22:34.115 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:33 smithi114 ceph-mon[106969]: pgmap v1267: 89 pgs: 89 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-14T04:22:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:33 smithi160 ceph-mon[88407]: pgmap v1267: 89 pgs: 89 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 13 KiB/s, 0 objects/s recovering 2024-09-14T04:22:34.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:34.193 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:34.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:34.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:34.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:34.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:34.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:34.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:34.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:34.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:34.482 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:22:34.482 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:22:34.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:22:34.482 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:22:34.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80150 11.7 11:e234d760:::benchmark_data_smithi114_223345_object80149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.155199+0000", 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 537.46536611500005, 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1129935500000001, 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80150 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.483 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.155199+0000", 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.155199+0000", 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.155198+0000", 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.155242+0000", 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4668e-05 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.155254+0000", 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1875000000000001e-05 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.155264+0000", 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8199999999999992e-06 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.155332+0000", 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8211000000000001e-05 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.155345+0000", 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411e-05 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.152731+0000", 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99738613399999998 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.152739+0000", 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9109999999999993e-06 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.253166+0000", 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1004272930000001 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.253324+0000", 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015751799999999999 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.253428+0000", 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104039 2024-09-14T04:22:34.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.256389+0000", 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002961436 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.256398+0000", 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0499999999999997e-06 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.266180+0000", 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0097820719999999993 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.266269+0000", 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8541000000000005e-05 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.268193+0000", 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001924165 2024-09-14T04:22:34.490 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80187 11.17 11:e871a484:::benchmark_data_smithi114_223345_object80186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.211073+0000", 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 537.40949229800003, 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0608456319999999, 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.491 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80187 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.211073+0000", 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.211073+0000", 2024-09-14T04:22:34.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.211071+0000", 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.211121+0000", 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9109e-05 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.211132+0000", 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1418e-05 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.211138+0000", 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0050000000000001e-06 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.211187+0000", 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8717000000000001e-05 2024-09-14T04:22:34.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.211195+0000", 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9119999999999998e-06 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.152693+0000", 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94149886299999996 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.152705+0000", 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1403000000000001e-05 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.263875+0000", 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.111169799 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.264048+0000", 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017289599999999999 2024-09-14T04:22:34.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.264236+0000", 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018839799999999999 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.270056+0000", 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0058202590000000004 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.270097+0000", 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0496e-05 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.270131+0000", 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4733000000000001e-05 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.270188+0000", 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6674e-05 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.271919+0000", 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001730699 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80190 11.7 11:e53d92eb:::benchmark_data_smithi114_223345_object80189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.215844+0000", 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 537.40472178200002, 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0559982799999998, 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.499 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80190 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.215844+0000", 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.215844+0000", 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.215843+0000", 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.215891+0000", 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8358e-05 2024-09-14T04:22:34.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.215902+0000", 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0461e-05 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.215907+0000", 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6710000000000004e-06 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.215951+0000", 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4076000000000003e-05 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.215958+0000", 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4609999999999996e-06 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.152760+0000", 2024-09-14T04:22:34.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93680184 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.152767+0000", 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.187e-06 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.256501+0000", 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.103734274 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.256706+0000", 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020503700000000001 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.256898+0000", 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000191838 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.265958+0000", 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090601499999999995 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.270036+0000", 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00407771 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.270086+0000", 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9867999999999998e-05 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.270147+0000", 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0813000000000002e-05 2024-09-14T04:22:34.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.271842+0000", 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0016953599999999999 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:80196 11.17 11:ed6733a6:::benchmark_data_smithi114_223345_object80195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:13:36.219077+0000", 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 537.40148852699997, 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.062139449, 2024-09-14T04:22:34.507 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 80196 2024-09-14T04:22:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.508 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.219077+0000", 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.219077+0000", 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.219076+0000", 2024-09-14T04:22:34.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.219142+0000", 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5592999999999994e-05 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.219155+0000", 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3858e-05 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.219161+0000", 2024-09-14T04:22:34.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7799999999999997e-06 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.219207+0000", 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5917000000000001e-05 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:36.219217+0000", 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0322e-05 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.153937+0000", 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93471917400000004 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:37.153946+0000", 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2420000000000006e-06 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.271969+0000", 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1180227650000001 2024-09-14T04:22:34.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.272136+0000", 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016776200000000001 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,4", 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.272329+0000", 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019298899999999999 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.276278+0000", 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039489490000000002 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.276288+0000", 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2369999999999998e-06 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.279155+0000", 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028672860000000001 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.279236+0000", 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0679000000000005e-05 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:13:38.281216+0000", 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00198088 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127218 11.5 11:a349dbb9:::benchmark_data_smithi114_223345_object15281:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.544492+0000", 2024-09-14T04:22:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 484.07607318499998, 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0149885890000001, 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127218 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544492+0000", 2024-09-14T04:22:34.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544492+0000", 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544491+0000", 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544499+0000", 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0609999999999996e-06 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544512+0000", 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3176999999999999e-05 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544519+0000", 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8600000000000004e-06 2024-09-14T04:22:34.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544836+0000", 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031726900000000001 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544845+0000", 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4280000000000002e-06 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.546840+0000", 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.001994389 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for missing object", 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.546936+0000", 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5784000000000003e-05 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559409+0000", 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012473507999999999 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559481+0000", 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1860999999999995e-05 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127221 11.15 11:ad8d8645:::benchmark_data_smithi114_223345_object15284:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.544529+0000", 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 484.07603617199999, 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0193685690000001, 2024-09-14T04:22:34.521 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127221 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544529+0000", 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544529+0000", 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544529+0000", 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544531+0000", 2024-09-14T04:22:34.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5399999999999998e-06 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544538+0000", 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.071e-06 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544541+0000", 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6969999999999999e-06 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544869+0000", 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00032764900000000001 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.544876+0000", 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8179999999999998e-06 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.546654+0000", 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.001777913 2024-09-14T04:22:34.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.546756+0000", 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000102815 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 6,7", 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.546837+0000", 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0723000000000006e-05 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555280+0000", 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0084428850000000007 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563748+0000", 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0084676709999999995 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563778+0000", 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0250999999999999e-05 2024-09-14T04:22:34.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563862+0000", 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3740999999999997e-05 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563898+0000", 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6189000000000002e-05 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.528 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127225 11.15 11:ada36830:::benchmark_data_smithi114_223345_object15288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.552330+0000", 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 484.06823572399998, 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0116689679999999, 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.529 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127225 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.552330+0000", 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.552330+0000", 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.552329+0000", 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.552334+0000", 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2009999999999998e-06 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.552344+0000", 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0370999999999999e-05 2024-09-14T04:22:34.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.552350+0000", 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0299999999999999e-06 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.552405+0000", 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4379000000000002e-05 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.552416+0000", 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0849000000000001e-05 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.548809+0000", 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99639354400000002 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.548960+0000", 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015071099999999999 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 6,7", 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.549084+0000", 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000123934 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557234+0000", 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0081498379999999995 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563930+0000", 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0066963450000000002 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563945+0000", 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5132e-05 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563969+0000", 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4243000000000002e-05 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563999+0000", 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9283999999999999e-05 2024-09-14T04:22:34.535 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127228 11.15 11:af725008:::benchmark_data_smithi114_223345_object15291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.554297+0000", 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 484.066268111, 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0099667619999999, 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.536 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127228 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.554297+0000", 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.554297+0000", 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.554296+0000", 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.554303+0000", 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3300000000000001e-06 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.554317+0000", 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3508e-05 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.554322+0000", 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2390000000000002e-06 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.554367+0000", 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5454999999999998e-05 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.554375+0000", 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.994e-06 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551102+0000", 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99672618999999996 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551216+0000", 2024-09-14T04:22:34.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000114929 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 6,7", 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551328+0000", 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111931 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557239+0000", 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059104509999999997 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564030+0000", 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0067907389999999996 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564188+0000", 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000158264 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564230+0000", 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2601000000000001e-05 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564264+0000", 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3691e-05 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127245 11.15 11:a8d3f769:::benchmark_data_smithi114_223345_object15308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.577311+0000", 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 484.04325481299998, 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98713753000000004, 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.543 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127245 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.577311+0000", 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.577311+0000", 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.577309+0000", 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.577316+0000", 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1770000000000001e-06 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.577332+0000", 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5274e-05 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.577338+0000", 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9850000000000003e-06 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.577381+0000", 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3649000000000002e-05 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.577388+0000", 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0260000000000002e-06 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553263+0000", 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97587479499999996 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553357+0000", 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3638e-05 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 6,7", 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553480+0000", 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012281099999999999 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559377+0000", 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0058978290000000003 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564302+0000", 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049250860000000004 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564385+0000", 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2025000000000002e-05 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564419+0000", 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4348999999999999e-05 2024-09-14T04:22:34.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564448+0000", 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9337999999999999e-05 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127258 11.15 11:ace13f4e:::benchmark_data_smithi114_223345_object15321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.604259+0000", 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 484.01630625400003, 2024-09-14T04:22:34.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96031109800000003, 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127258 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.604259+0000", 2024-09-14T04:22:34.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.604259+0000", 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.604259+0000", 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.604262+0000", 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.101e-06 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.604269+0000", 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3900000000000004e-06 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.604273+0000", 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3220000000000001e-06 2024-09-14T04:22:34.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.604330+0000", 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7012000000000002e-05 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.604338+0000", 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1669999999999999e-06 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555310+0000", 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95097249900000003 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555409+0000", 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8449999999999994e-05 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 6,7", 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555508+0000", 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9752000000000003e-05 2024-09-14T04:22:34.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559476+0000", 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039674030000000004 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564479+0000", 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.005003286 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564506+0000", 2024-09-14T04:22:34.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6516000000000002e-05 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564541+0000", 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5098999999999997e-05 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564570+0000", 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9632e-05 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.557 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127261 11.15 11:ab409929:::benchmark_data_smithi114_223345_object15324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.608724+0000", 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 484.01184148999999, 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.95799352699999996, 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.558 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127261 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.608724+0000", 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.608724+0000", 2024-09-14T04:22:34.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.608722+0000", 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.608733+0000", 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0689999999999999e-05 2024-09-14T04:22:34.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.608748+0000", 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5119e-05 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.608757+0000", 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8049999999999996e-06 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.608819+0000", 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1597e-05 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.608831+0000", 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2779000000000001e-05 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557273+0000", 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94844136300000004 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557381+0000", 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108188 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 6,7", 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557501+0000", 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000120547 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559480+0000", 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0019789410000000001 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564601+0000", 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0051207589999999999 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.566605+0000", 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020043830000000002 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.566680+0000", 2024-09-14T04:22:34.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4121000000000004e-05 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.566718+0000", 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7926999999999998e-05 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127272 11.15 11:aabaa43b:::benchmark_data_smithi114_223345_object15335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.634188+0000", 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 483.98637708899997, 2024-09-14T04:22:34.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.93265323200000005, 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127272 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.634188+0000", 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.634188+0000", 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.634187+0000", 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.634196+0000", 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5540000000000003e-06 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.634215+0000", 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8830000000000001e-05 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.634224+0000", 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8349999999999993e-06 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.634291+0000", 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7282999999999999e-05 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.634300+0000", 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6130000000000004e-06 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559505+0000", 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.92520461799999998 2024-09-14T04:22:34.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559599+0000", 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4928999999999999e-05 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 6,7", 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559699+0000", 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9692999999999996e-05 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564308+0000", 2024-09-14T04:22:34.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0046092850000000003 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.566577+0000", 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022690000000000002 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.566745+0000", 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016778 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.566803+0000", 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7476999999999999e-05 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.566842+0000", 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8926e-05 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127277 11.15 11:ac5e1832:::benchmark_data_smithi114_223345_object15340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.648369+0000", 2024-09-14T04:22:34.572 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 483.97219642599998, 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91893564599999999, 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127277 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.648369+0000", 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.648369+0000", 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.648368+0000", 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.648374+0000", 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1210000000000003e-06 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.648386+0000", 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2388000000000001e-05 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.648391+0000", 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9599999999999999e-06 2024-09-14T04:22:34.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.648445+0000", 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3708000000000001e-05 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.648455+0000", 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0159e-05 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.561501+0000", 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91304573600000005 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.561607+0000", 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106638 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 6,7", 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.561708+0000", 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100828 2024-09-14T04:22:34.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.564341+0000", 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026323200000000001 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.566869+0000", 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025284259999999999 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.567217+0000", 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034821599999999999 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.567273+0000", 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5980999999999998e-05 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.567305+0000", 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1438e-05 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:31611 13.9 13:927299fd:::benchmark_data_smithi114_288325_object31610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:24.685215+0000", 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 8.9353503580000009, 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.244197795, 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 31611 2024-09-14T04:22:34.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.685215+0000", 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.685215+0000", 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.685215+0000", 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.685245+0000", 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9995e-05 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.685251+0000", 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2169999999999999e-06 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.685254+0000", 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4690000000000002e-06 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.685418+0000", 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016333600000000001 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.685538+0000", 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011993 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,2", 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.685800+0000", 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026246800000000002 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.688292+0000", 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0024917569999999998 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.692775+0000", 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0044835279999999996 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.929331+0000", 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.236555924 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.929378+0000", 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6804000000000001e-05 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.929413+0000", 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4888999999999997e-05 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:31615 13.9 13:99be310c:::benchmark_data_smithi114_288325_object31614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:24.687605+0000", 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 8.9329602359999996, 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24199491000000001, 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.586 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 31615 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.687605+0000", 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.687605+0000", 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.687604+0000", 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.688336+0000", 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00073187999999999997 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.688347+0000", 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0447e-05 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.688352+0000", 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1819999999999997e-06 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.688501+0000", 2024-09-14T04:22:34.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014913700000000001 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.688617+0000", 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116173 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,2", 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.688734+0000", 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117337 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.692844+0000", 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041097590000000002 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.693474+0000", 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062984100000000002 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.929522+0000", 2024-09-14T04:22:34.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.23604770899999999 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.929565+0000", 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2772999999999999e-05 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.929600+0000", 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5605000000000002e-05 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:31635 13.9 13:9d9b863c:::benchmark_data_smithi114_288325_object31634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:24.697490+0000", 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 8.9230753580000002, 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.232266097, 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 31635 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.593 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.697490+0000", 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.697490+0000", 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.697489+0000", 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.697536+0000", 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6375000000000002e-05 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.697546+0000", 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0740999999999999e-05 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.697552+0000", 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6570000000000002e-06 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.697639+0000", 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7274000000000005e-05 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.697753+0000", 2024-09-14T04:22:34.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113467 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,2", 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.697921+0000", 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016859700000000001 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.699603+0000", 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0016820209999999999 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.700978+0000", 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0013745420000000001 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.929641+0000", 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.228662959 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.929716+0000", 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5290000000000006e-05 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.929756+0000", 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0030000000000001e-05 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:31648 13.9 13:9c56082b:::benchmark_data_smithi114_288325_object31647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:24.703225+0000", 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 8.9173408559999991, 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.229476028, 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.599 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 31648 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.703225+0000", 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.703225+0000", 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.703223+0000", 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.703315+0000", 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1818999999999996e-05 2024-09-14T04:22:34.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.703327+0000", 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1975e-05 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.703333+0000", 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5899999999999998e-06 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.703373+0000", 2024-09-14T04:22:34.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0188999999999998e-05 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.703499+0000", 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012584400000000001 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,2", 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.703637+0000", 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013829000000000001 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.704780+0000", 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001143261 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.705530+0000", 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000750123 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.932617+0000", 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22708630599999999 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.932666+0000", 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9137000000000001e-05 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.932701+0000", 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4783000000000002e-05 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.605 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:31681 13.9 13:9bdea33f:::benchmark_data_smithi114_288325_object31680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:24.746915+0000", 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 8.8736500760000006, 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18603208199999999, 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 31681 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.746915+0000", 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.746915+0000", 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.746915+0000", 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.746965+0000", 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0180999999999999e-05 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.746975+0000", 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0733999999999999e-05 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.746981+0000", 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1749999999999996e-06 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.747027+0000", 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6829999999999997e-05 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.747148+0000", 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012013799999999999 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,2", 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.747292+0000", 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014414199999999999 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.748388+0000", 2024-09-14T04:22:34.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00109632 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.748583+0000", 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019489600000000001 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.932867+0000", 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18428396499999999 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.932913+0000", 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6548000000000002e-05 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:24.932947+0000", 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4038000000000002e-05 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:35243 13.6 13:61e99d96:::benchmark_data_smithi114_288325_object35242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:26.669422+0000", 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 6.9511437379999998, 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.14438790900000001, 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 35243 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.613 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.669422+0000", 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.669422+0000", 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.669421+0000", 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.669495+0000", 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3705e-05 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.669503+0000", 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2500000000000006e-06 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.669508+0000", 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.763e-06 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.673427+0000", 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039191110000000003 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.673554+0000", 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012672900000000001 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,4", 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.673695+0000", 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014103499999999999 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.675465+0000", 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017705780000000001 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.681799+0000", 2024-09-14T04:22:34.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.006334063 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.813753+0000", 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13195385300000001 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.813786+0000", 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2505000000000001e-05 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.813810+0000", 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4059000000000001e-05 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:35250 13.2 13:499fd0e4:::benchmark_data_smithi114_288325_object35249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:26.671069+0000", 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 6.949496323, 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.14290143299999999, 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:34.619 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 35250 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.671069+0000", 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.671069+0000", 2024-09-14T04:22:34.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.671068+0000", 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.671095+0000", 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6968e-05 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.671107+0000", 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2150000000000001e-05 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.671112+0000", 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9989999999999999e-06 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.671172+0000", 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0341999999999999e-05 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.671282+0000", 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109911 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,4", 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.671456+0000", 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017409599999999999 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.674431+0000", 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029745800000000001 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.677042+0000", 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002610598 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.813885+0000", 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13684320999999999 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:34.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.813935+0000", 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9793000000000003e-05 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:26.813971+0000", 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6026999999999999e-05 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:34.625 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:22:34.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:34.659 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:34.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:34.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:34.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:34.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:34.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:34.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:34.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:34.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 48706 48690 132.298 79.5 0.0145678 0.00755321 2024-09-14T04:22:35.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:35.068 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:35.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:35.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:35.148 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:35.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:35.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:35.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:35.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:35.317 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:35.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:35.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:35.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:35.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:35.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:35.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:35.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:35.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:35.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:35.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:35.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-14T04:22:35.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:35.756 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:35.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:35.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:35.889 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:35.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:35.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 50434 50418 131.286 108 0.00533859 0.00761235 2024-09-14T04:22:35.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:36.035 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:36.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:36.136 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:22:36.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:36.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:36.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:36.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:36.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:36.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:36.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:36.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:36.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:36.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:36.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:36.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:36.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:36 smithi114 ceph-mon[103029]: pgmap v1268: 89 pgs: 89 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T04:22:36.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:36 smithi114 ceph-mon[106969]: pgmap v1268: 89 pgs: 89 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T04:22:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:36 smithi160 ceph-mon[88407]: pgmap v1268: 89 pgs: 89 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T04:22:36.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:36.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:36.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:36.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:36.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:36.802 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:36.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:36.831 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:22:36.832 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:22:36.848 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:22:36.848 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:22:36.864 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:22:36.864 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:22:36.880 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:22:36.880 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:22:36.898 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:22:36.899 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:22:36.904 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:36.925 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:22:36.926 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:22:36.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:36.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:36.945 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 51172 51156 127.879 46.125 0.00995903 0.00781406 2024-09-14T04:22:36.958 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:22:36.958 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:22:36.984 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:22:36.984 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:22:37.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:37.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:37.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:37.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:22:37.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:37.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:37.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:37.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:37.357 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:37.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:37.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:37.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:37 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4028008453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:37.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:37 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4028008453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:37.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:37.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:37.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:37.596 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:37.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:37.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:37.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:37.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:37 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4028008453' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:37.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:37.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:37.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:37.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:37.874 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:37.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:37.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 52545 52529 126.261 85.8125 0.00797301 0.0079155 2024-09-14T04:22:37.976 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:38.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:38.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:38.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:38.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:38.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:38.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:38.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:38.402 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:38.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:38.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:38.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:38.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:38.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:38.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:38.690 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:38 smithi114 ceph-mon[103029]: pgmap v1269: 89 pgs: 89 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-14T04:22:38.690 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:38.690 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:38.690 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:38.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:38 smithi114 ceph-mon[106969]: pgmap v1269: 89 pgs: 89 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-14T04:22:38.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:38.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:38.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:38.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:38.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:38.755 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:38.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:38.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:38.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:38 smithi160 ceph-mon[88407]: pgmap v1269: 89 pgs: 89 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-14T04:22:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:38.944 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:38.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 54164 54148 125.332 101.188 0.0035041 0.00797355 2024-09-14T04:22:38.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:38.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:39.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:39.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:39.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:39.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:39.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:39.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:39.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:39.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:39.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:39 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:22:39.059+0000 7f3162f23640 -1 mon.a@0(leader).osd e330 definitely_dead 0 2024-09-14T04:22:39.372 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:39.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:39.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:39.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:39.501 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:39.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:39.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:39.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:39.624 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:39.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:39.673 INFO:teuthology.orchestra.run.smithi114.stderr:marked out osd.2. 2024-09-14T04:22:39.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:39.748 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:39.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:39.849 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:39.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:39.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3400513975' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-14T04:22:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:39 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-14T04:22:39.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:39 smithi160 ceph-mon[88407]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-14T04:22:39.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 54912 54896 122.525 46.75 0.00291403 0.00808031 2024-09-14T04:22:39.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:39.994 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:39.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:40.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:40.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:40.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3400513975' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-14T04:22:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:39 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-14T04:22:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:39 smithi114 ceph-mon[103029]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-14T04:22:40.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3400513975' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-14T04:22:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:39 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-14T04:22:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:39 smithi114 ceph-mon[106969]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-14T04:22:40.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:22:40.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:40.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:40.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:40.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:40.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:40.364 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:40.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:40.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:40.488 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:40.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:40.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:40.616 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:40.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:22:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:22:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:22:40.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:40.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:40.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:40.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:40.862 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:40.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:40.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 54912 54896 118.3 0 - 0.00808031 2024-09-14T04:22:40.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:40.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:40.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:41.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:41.096 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:40 smithi160 ceph-mon[88407]: pgmap v1270: 89 pgs: 89 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-14T04:22:41.096 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:40 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-14T04:22:41.096 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:40 smithi160 ceph-mon[88407]: osdmap e331: 8 total, 8 up, 7 in 2024-09-14T04:22:41.110 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:41.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:41.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:40 smithi114 ceph-mon[103029]: pgmap v1270: 89 pgs: 89 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-14T04:22:41.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:40 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-14T04:22:41.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:40 smithi114 ceph-mon[103029]: osdmap e331: 8 total, 8 up, 7 in 2024-09-14T04:22:41.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:40 smithi114 ceph-mon[106969]: pgmap v1270: 89 pgs: 89 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-14T04:22:41.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:40 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-14T04:22:41.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:40 smithi114 ceph-mon[106969]: osdmap e331: 8 total, 8 up, 7 in 2024-09-14T04:22:41.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:41.234 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:41.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:41.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:41.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:41.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:41.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:41.486 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:41.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:41.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:41.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:41.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:41.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:41.712 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:41.724 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:41 smithi114 ceph-mon[106969]: osdmap e332: 8 total, 8 up, 7 in 2024-09-14T04:22:41.724 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:41 smithi114 ceph-mon[106969]: pgmap v1273: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-14T04:22:41.725 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:22:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:22:41] "GET /metrics HTTP/1.1" 200 36565 "" "Prometheus/2.51.0" 2024-09-14T04:22:41.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:41 smithi114 ceph-mon[103029]: osdmap e332: 8 total, 8 up, 7 in 2024-09-14T04:22:41.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:41 smithi114 ceph-mon[103029]: pgmap v1273: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-14T04:22:41.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:41.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:41.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:41.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:41.905 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:41.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:41.915 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:41 smithi160 ceph-mon[88407]: osdmap e332: 8 total, 8 up, 7 in 2024-09-14T04:22:41.916 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:41 smithi160 ceph-mon[88407]: pgmap v1273: 89 pgs: 89 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-14T04:22:41.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 54923 54907 114.38 0.34375 2.24607 0.00844387 2024-09-14T04:22:42.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:42.028 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:42.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:42.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:42.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:42.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:42.279 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:42.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:42.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:42.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:42.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:42.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:42.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:42.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:42.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:42.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:42.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:42.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:42.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:42.697 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:22:42.752 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:22:42.752 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:22:42.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:42.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:42.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:42.909 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:42.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:42.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:42.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 55497 55481 111.847 35.875 0.0152794 0.00893366 2024-09-14T04:22:42.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:42 smithi160 ceph-mon[88407]: osdmap e333: 8 total, 8 up, 7 in 2024-09-14T04:22:42.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/862180533' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:42.960 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:42 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:43.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:43.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:43.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:43.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:43.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:42 smithi114 ceph-mon[106969]: osdmap e333: 8 total, 8 up, 7 in 2024-09-14T04:22:43.100 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/862180533' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:43.100 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:42 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:43.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:42 smithi114 ceph-mon[103029]: osdmap e333: 8 total, 8 up, 7 in 2024-09-14T04:22:43.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/862180533' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:43.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:42 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:22:43.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:43.185 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:43.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:43.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:43.313 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:43.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:43.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:43.435 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:43.462 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:43.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:43.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-14T04:22:43.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:43.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:43.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:43.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:43.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:43.733 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:43.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:43.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:43.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:43.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:43 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:43 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:22:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:43 smithi160 ceph-mon[88407]: osdmap e334: 8 total, 8 up, 7 in 2024-09-14T04:22:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:43 smithi160 ceph-mon[88407]: pgmap v1276: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 14 MiB/s wr, 228 op/s 2024-09-14T04:22:43.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 56316 56300 109.951 51.1875 0.0222591 0.00908869 2024-09-14T04:22:43.980 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:43 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:43.980 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:43 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:22:43.980 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:43 smithi114 ceph-mon[106969]: osdmap e334: 8 total, 8 up, 7 in 2024-09-14T04:22:43.980 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:43 smithi114 ceph-mon[106969]: pgmap v1276: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 14 MiB/s wr, 228 op/s 2024-09-14T04:22:43.980 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:43 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:43.980 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:43 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:22:43.980 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:43 smithi114 ceph-mon[103029]: osdmap e334: 8 total, 8 up, 7 in 2024-09-14T04:22:43.980 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:43 smithi114 ceph-mon[103029]: pgmap v1276: 89 pgs: 89 active+clean; 3.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 14 MiB/s wr, 228 op/s 2024-09-14T04:22:43.983 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:22:43.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:44.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:44.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:44.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:44.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:44.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:44.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:44.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:44.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:44.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:44.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:44.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:44.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:44.573 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:44.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:44.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:44.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:44.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:44.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:44.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:44.867 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:44.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:44.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 57117 57101 108.136 50.0625 0.0126672 0.00924018 2024-09-14T04:22:44.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:44.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:44.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:44.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:45.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:45.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:45.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:45.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:45.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:45.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:45.319 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:45 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 27512/167574 objects degraded (16.418%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:22:45.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:45.368 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:45 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 27512/167574 objects degraded (16.418%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:22:45.368 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:45 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 27512/167574 objects degraded (16.418%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:22:45.374 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:45.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:45.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:45.384 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:22:45.385 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:22:45.406 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:22:45.407 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:22:45.425 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:22:45.425 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:22:45.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:22:45.450 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:22:45.450 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:22:45.472 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:22:45.473 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:22:45.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:45.495 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:22:45.495 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:22:45.524 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:22:45.524 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:22:45.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:45.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:45.551 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:22:45.551 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:22:45.630 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:45.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:45.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:45.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:45.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:45.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:45.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:45.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:45.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:45.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:45.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 57935 57919 106.46 51.125 0.0184808 0.00938406 2024-09-14T04:22:46.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:46.045 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:22:46.045 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:22:46.045 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:22:46.045 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:22:46.045 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:22:46.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:46.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:46.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:46.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:46.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:46.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:46.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:46.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:46.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:46.419 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:46.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:46.466 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:46 smithi160 ceph-mon[88407]: pgmap v1277: 89 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 80 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 32 MiB/s wr, 518 op/s; 27512/167574 objects degraded (16.418%); 3260/167574 objects misplaced (1.945%); 1.4 MiB/s, 1 keys/s, 22 objects/s recovering 2024-09-14T04:22:46.466 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2426693436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:46.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:46.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:46.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:46 smithi114 ceph-mon[103029]: pgmap v1277: 89 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 80 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 32 MiB/s wr, 518 op/s; 27512/167574 objects degraded (16.418%); 3260/167574 objects misplaced (1.945%); 1.4 MiB/s, 1 keys/s, 22 objects/s recovering 2024-09-14T04:22:46.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2426693436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:46.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:46 smithi114 ceph-mon[106969]: pgmap v1277: 89 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 80 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 32 MiB/s wr, 518 op/s; 27512/167574 objects degraded (16.418%); 3260/167574 objects misplaced (1.945%); 1.4 MiB/s, 1 keys/s, 22 objects/s recovering 2024-09-14T04:22:46.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2426693436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:46.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:46.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:46.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:46.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:46.736 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:46.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:46.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-14T04:22:46.837 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:46.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:46.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:46.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 58740 58724 104.855 50.3125 0.0230496 0.00953051 2024-09-14T04:22:47.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:47.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:47.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:47.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:47.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:47.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:47.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:47.311 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:22:47.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:47.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:47.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:47.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:47.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:47.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:47.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:47.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:47.632 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:47.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:47.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:47.780 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:47.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:47.818 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 4, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:22:47.818 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:22:47.818 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-14T04:22:47.818 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:22:47.818 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:22:47.819 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-14T04:22:47.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:47.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:47.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:47.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 59527 59511 103.309 49.1875 0.0237942 0.00967342 2024-09-14T04:22:48.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:48.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:48.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:48.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:48.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:48.190 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:48.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:48.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:48.328 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:48.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:48.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:48.401 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:22:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:48 smithi160 ceph-mon[88407]: pgmap v1278: 89 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 80 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 27 MiB/s wr, 437 op/s; 27512/167574 objects degraded (16.418%); 3260/167574 objects misplaced (1.945%); 1.1 MiB/s, 1 keys/s, 19 objects/s recovering 2024-09-14T04:22:48.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:48.471 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:48.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:48.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:48.580 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:48 smithi114 ceph-mon[106969]: pgmap v1278: 89 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 80 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 27 MiB/s wr, 437 op/s; 27512/167574 objects degraded (16.418%); 3260/167574 objects misplaced (1.945%); 1.1 MiB/s, 1 keys/s, 19 objects/s recovering 2024-09-14T04:22:48.580 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:48 smithi114 ceph-mon[103029]: pgmap v1278: 89 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 80 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 27 MiB/s wr, 437 op/s; 27512/167574 objects degraded (16.418%); 3260/167574 objects misplaced (1.945%); 1.1 MiB/s, 1 keys/s, 19 objects/s recovering 2024-09-14T04:22:48.616 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:48.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:48.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:48.746 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:48.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:48.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:48.920 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:48.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:48.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 60335 60319 101.881 50.5 0.0245359 0.00980949 2024-09-14T04:22:49.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:49.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:49.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:49.166 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:49.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:49.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:49.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:49.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:49.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:49.442 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:49.465 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:49.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127224 11.4 11:21e6e516:::benchmark_data_smithi114_223345_object15287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.551373+0000", 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 499.977837556, 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0083590950000001, 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.533 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127224 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551373+0000", 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551373+0000", 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551371+0000", 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551379+0000", 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4420000000000001e-06 2024-09-14T04:22:49.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551393+0000", 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4610000000000001e-05 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551400+0000", 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4579999999999998e-06 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551456+0000", 2024-09-14T04:22:49.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6212000000000003e-05 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.551467+0000", 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.075e-05 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553349+0000", 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.001882191 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553444+0000", 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4993000000000003e-05 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553534+0000", 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0431000000000002e-05 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559606+0000", 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.006072076 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559662+0000", 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5921999999999998e-05 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559669+0000", 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0160000000000003e-06 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559702+0000", 2024-09-14T04:22:49.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2906e-05 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559732+0000", 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9733000000000001e-05 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127241 11.14 11:2e01ae4a:::benchmark_data_smithi114_223345_object15304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.572850+0000", 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 499.95636056400002, 2024-09-14T04:22:49.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98064877299999997, 2024-09-14T04:22:49.541 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.541 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:49.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:49.541 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127241 2024-09-14T04:22:49.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.541 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572850+0000", 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572850+0000", 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572849+0000", 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:49.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572853+0000", 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8460000000000001e-06 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572864+0000", 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.098e-05 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572868+0000", 2024-09-14T04:22:49.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4410000000000003e-06 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572922+0000", 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3471999999999999e-05 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.572935+0000", 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2975e-05 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.548843+0000", 2024-09-14T04:22:49.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.975908687 2024-09-14T04:22:49.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.548971+0000", 2024-09-14T04:22:49.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724200000000001 2024-09-14T04:22:49.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.549084+0000", 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113316 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551528+0000", 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00244378 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553317+0000", 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017892120000000001 2024-09-14T04:22:49.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553369+0000", 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1722999999999999e-05 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553463+0000", 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4036000000000006e-05 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.553499+0000", 2024-09-14T04:22:49.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5998999999999999e-05 2024-09-14T04:22:49.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.548 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127248 11.4 11:26baefff:::benchmark_data_smithi114_223345_object15311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.587658+0000", 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 499.94155260500003, 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.97257697799999998, 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:49.549 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127248 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587658+0000", 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587658+0000", 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587656+0000", 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587665+0000", 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3259999999999996e-06 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587683+0000", 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8300000000000001e-05 2024-09-14T04:22:49.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587689+0000", 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7420000000000003e-06 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587749+0000", 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0062000000000002e-05 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.587761+0000", 2024-09-14T04:22:49.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2285e-05 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555352+0000", 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96759091900000005 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555578+0000", 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000226176 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555671+0000", 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2947000000000003e-05 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559633+0000", 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039615220000000003 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559737+0000", 2024-09-14T04:22:49.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104633 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560169+0000", 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043182499999999999 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560203+0000", 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3831000000000002e-05 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560235+0000", 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1977000000000002e-05 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127262 11.14 11:28c1fb4b:::benchmark_data_smithi114_223345_object15325:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.611254+0000", 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 499.91795643400002, 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94832032799999999, 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.556 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127262 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611254+0000", 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611254+0000", 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611252+0000", 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611260+0000", 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9840000000000001e-06 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611276+0000", 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5164000000000001e-05 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611282+0000", 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4060000000000002e-06 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611339+0000", 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7014e-05 2024-09-14T04:22:49.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.611351+0000", 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1579999999999999e-05 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551549+0000", 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.940198321 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551673+0000", 2024-09-14T04:22:49.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012361499999999999 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.551780+0000", 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000107011 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555307+0000", 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003527002 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.555312+0000", 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7950000000000004e-06 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559499+0000", 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041868319999999997 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559541+0000", 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1721999999999998e-05 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.559574+0000", 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3407e-05 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.36273.0:127264 11.4 11:260f74ee:::benchmark_data_smithi114_223345_object15327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:14:29.613838+0000", 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 499.91537284899999, 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94978247400000004, 2024-09-14T04:22:49.563 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.36273", 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 127264 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613838+0000", 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613838+0000", 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613836+0000", 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613843+0000", 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1550000000000004e-06 2024-09-14T04:22:49.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613855+0000", 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2069000000000001e-05 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613859+0000", 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3599999999999998e-06 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613896+0000", 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7217e-05 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:29.613904+0000", 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2099999999999996e-06 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557498+0000", 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.94359476099999995 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557706+0000", 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020764700000000001 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.557796+0000", 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0552999999999999e-05 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.560072+0000", 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022751780000000001 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.562269+0000", 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021972390000000001 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563542+0000", 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001273167 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563586+0000", 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4351000000000003e-05 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:14:30.563620+0000", 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3630999999999997e-05 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.570 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:46642 12.0 12:046bca62:::benchmark_data_smithi114_253692_object46641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:17:47.286839+0000", 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 302.24237123099999, 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82783152400000004, 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 46642 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286839+0000", 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286839+0000", 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286839+0000", 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:49.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286894+0000", 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5449999999999999e-05 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286902+0000", 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2449999999999998e-06 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286906+0000", 2024-09-14T04:22:49.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3850000000000002e-06 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286955+0000", 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9156000000000002e-05 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for peered", 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:47.286964+0000", 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6309999999999998e-06 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.102517+0000", 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.81555286000000005 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.102653+0000", 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000135635 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.102776+0000", 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012279700000000001 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.104401+0000", 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0016249019999999999 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.105350+0000", 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00094917700000000005 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.114580+0000", 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0092306699999999998 2024-09-14T04:22:49.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.114633+0000", 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2324000000000003e-05 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:17:48.114671+0000", 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7902999999999999e-05 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:119405 12.f 12:fb31b8ab:::benchmark_data_smithi114_253692_object1426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.068770+0000", 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 232.46044064500001, 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41861637699999998, 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119405 2024-09-14T04:22:49.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068770+0000", 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068770+0000", 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068769+0000", 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068775+0000", 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1519999999999997e-06 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068784+0000", 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8950000000000004e-06 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068789+0000", 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0499999999999999e-06 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.068945+0000", 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015627999999999999 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.069042+0000", 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6781999999999996e-05 2024-09-14T04:22:49.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,5", 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.069107+0000", 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5267999999999999e-05 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073284+0000", 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041767339999999997 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073309+0000", 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5202999999999999e-05 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487312+0000", 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41400287899999999 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487350+0000", 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7935000000000002e-05 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487386+0000", 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6297999999999997e-05 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:119407 12.0 12:03c363b5:::benchmark_data_smithi114_253692_object1428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.072825+0000", 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 232.45638535699999, 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42696216799999998, 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.584 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119407 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072825+0000", 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072825+0000", 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072824+0000", 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072831+0000", 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8689999999999998e-06 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072841+0000", 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.01e-05 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072847+0000", 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7359999999999998e-06 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073024+0000", 2024-09-14T04:22:49.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017740799999999999 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073145+0000", 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000121032 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073253+0000", 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000108347 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077106+0000", 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003852725 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487199+0000", 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41009261000000002 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499662+0000", 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012463741 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499745+0000", 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2350999999999998e-05 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499787+0000", 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2494999999999999e-05 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.590 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46463.0:119401 12.b e296/284 12:d0be7710:::benchmark_data_smithi114_253692_object1422:head v 296'7335, mlcod=296'7332)", 2024-09-14T04:22:49.591 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.072948+0000", 2024-09-14T04:22:49.591 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 232.456262385, 2024-09-14T04:22:49.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40976830199999997, 2024-09-14T04:22:49.591 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:49.592 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072948+0000", 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072948+0000", 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072948+0000", 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072952+0000", 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5759999999999999e-06 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072959+0000", 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1909999999999997e-06 2024-09-14T04:22:49.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.072963+0000", 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0180000000000003e-06 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073111+0000", 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000148228 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073130+0000", 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8751000000000001e-05 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.482688+0000", 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.409558329 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.482716+0000", 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8469000000000001e-05 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.596 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:119412 12.0 12:09c6de14:::benchmark_data_smithi114_253692_object1433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.073189+0000", 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 232.45602125400001, 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42671745500000002, 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:22:49.597 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119412 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073189+0000", 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073189+0000", 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073188+0000", 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073193+0000", 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7269999999999998e-06 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073202+0000", 2024-09-14T04:22:49.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1390000000000001e-06 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073207+0000", 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5750000000000002e-06 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077157+0000", 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039506919999999996 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077241+0000", 2024-09-14T04:22:49.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3615000000000005e-05 2024-09-14T04:22:49.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:22:49.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478681+0000", 2024-09-14T04:22:49.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40144028700000001 2024-09-14T04:22:49.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487230+0000", 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0085489810000000006 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487373+0000", 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014276600000000001 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499796+0000", 2024-09-14T04:22:49.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012422812 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499875+0000", 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8677999999999994e-05 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.499907+0000", 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2048000000000002e-05 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:119413 12.5 12:a8b15bec:::benchmark_data_smithi114_253692_object1434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.073417+0000", 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 232.455793785, 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.410459833, 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119413 2024-09-14T04:22:49.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073417+0000", 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073417+0000", 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073416+0000", 2024-09-14T04:22:49.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:49.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073420+0000", 2024-09-14T04:22:49.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9700000000000001e-06 2024-09-14T04:22:49.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073430+0000", 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0068000000000001e-05 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073435+0000", 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4530000000000001e-06 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073490+0000", 2024-09-14T04:22:49.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4994999999999998e-05 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073602+0000", 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111681 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.073691+0000", 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8565000000000003e-05 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478791+0000", 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.405100505 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483354+0000", 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045633610000000002 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483757+0000", 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000402326 2024-09-14T04:22:49.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483838+0000", 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1552999999999997e-05 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.483876+0000", 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8114000000000001e-05 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46463.0:119403 12.b e296/284 12:d65c6516:::benchmark_data_smithi114_253692_object1424:head v 296'7336, mlcod=296'7332)", 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.076066+0000", 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 232.45314425500001, 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41100507200000003, 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076066+0000", 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076066+0000", 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076062+0000", 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999962 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076071+0000", 2024-09-14T04:22:49.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8529999999999998e-06 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076079+0000", 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0059999999999992e-06 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.076083+0000", 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3699999999999997e-06 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478585+0000", 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40250170400000002 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478612+0000", 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7090999999999999e-05 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487000+0000", 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0083881979999999995 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.487071+0000", 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0820000000000003e-05 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46463.0:119416 12.1 12:806ab969:::benchmark_data_smithi114_253692_object1437:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:18:57.077231+0000", 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 232.45197929400001, 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41513983199999999, 2024-09-14T04:22:49.615 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.616 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46463", 2024-09-14T04:22:49.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:22:49.617 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 119416 2024-09-14T04:22:49.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.617 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077231+0000", 2024-09-14T04:22:49.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077231+0000", 2024-09-14T04:22:49.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077230+0000", 2024-09-14T04:22:49.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:49.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077236+0000", 2024-09-14T04:22:49.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0020000000000003e-06 2024-09-14T04:22:49.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.077246+0000", 2024-09-14T04:22:49.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0258000000000001e-05 2024-09-14T04:22:49.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.478517+0000", 2024-09-14T04:22:49.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40127036599999999 2024-09-14T04:22:49.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.479436+0000", 2024-09-14T04:22:49.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00091930500000000004 2024-09-14T04:22:49.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.479538+0000", 2024-09-14T04:22:49.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010226 2024-09-14T04:22:49.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:22:49.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.479631+0000", 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3214999999999994e-05 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.491421+0000", 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011790007999999999 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.491487+0000", 2024-09-14T04:22:49.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5579000000000004e-05 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.492268+0000", 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000780655 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.492326+0000", 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8851000000000003e-05 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:18:57.492371+0000", 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4453000000000003e-05 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:51050 13.1 13:87b73c84:::benchmark_data_smithi114_288325_object51049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:36.448389+0000", 2024-09-14T04:22:49.624 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 13.08082127, 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.44763401899999999, 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 51050 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.448389+0000", 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.448389+0000", 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.448388+0000", 2024-09-14T04:22:49.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:49.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.448478+0000", 2024-09-14T04:22:49.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0148999999999994e-05 2024-09-14T04:22:49.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.448491+0000", 2024-09-14T04:22:49.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2499e-05 2024-09-14T04:22:49.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.448497+0000", 2024-09-14T04:22:49.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9410000000000004e-06 2024-09-14T04:22:49.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.448543+0000", 2024-09-14T04:22:49.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6731999999999999e-05 2024-09-14T04:22:49.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.448672+0000", 2024-09-14T04:22:49.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000128949 2024-09-14T04:22:49.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.448808+0000", 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013606100000000001 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.451076+0000", 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022679100000000002 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.456218+0000", 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0051420950000000002 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895911+0000", 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43969259700000002 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895961+0000", 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9839999999999997e-05 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.896023+0000", 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2398999999999997e-05 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:51057 13.1 13:8f2fa3ff:::benchmark_data_smithi114_288325_object51056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:36.451046+0000", 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 13.078163934999999, 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.44509817499999998, 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 51057 2024-09-14T04:22:49.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.451046+0000", 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.451046+0000", 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.451038+0000", 2024-09-14T04:22:49.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999914 2024-09-14T04:22:49.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.451102+0000", 2024-09-14T04:22:49.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3855000000000004e-05 2024-09-14T04:22:49.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.451113+0000", 2024-09-14T04:22:49.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.153e-05 2024-09-14T04:22:49.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.451118+0000", 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1599999999999997e-06 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.451158+0000", 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9026999999999998e-05 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.451264+0000", 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106947 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.451383+0000", 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118922 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.453901+0000", 2024-09-14T04:22:49.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0025176259999999998 2024-09-14T04:22:49.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.456243+0000", 2024-09-14T04:22:49.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023421850000000001 2024-09-14T04:22:49.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.896071+0000", 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43982809899999997 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.896111+0000", 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0117999999999998e-05 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.896145+0000", 2024-09-14T04:22:49.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3220999999999999e-05 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:51058 13.1 13:81d64199:::benchmark_data_smithi114_288325_object51057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:36.454164+0000", 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 13.075046559, 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.44208587100000002, 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.641 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.642 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.642 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:49.642 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:49.642 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 51058 2024-09-14T04:22:49.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.642 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.454164+0000", 2024-09-14T04:22:49.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.454164+0000", 2024-09-14T04:22:49.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.454162+0000", 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.454280+0000", 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011761599999999999 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.454293+0000", 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2986999999999999e-05 2024-09-14T04:22:49.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.454299+0000", 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0120000000000002e-06 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.456264+0000", 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0019653729999999999 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.456377+0000", 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011265 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.456503+0000", 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126141 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895810+0000", 2024-09-14T04:22:49.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43930692100000002 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895862+0000", 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2170999999999999e-05 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.896188+0000", 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00032554300000000002 2024-09-14T04:22:49.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.896222+0000", 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4683e-05 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.896250+0000", 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7460000000000001e-05 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.650 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:51064 13.4 13:2544c0ee:::benchmark_data_smithi114_288325_object51063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:36.457233+0000", 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 13.071977716999999, 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.44581005400000001, 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 51064 2024-09-14T04:22:49.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.457233+0000", 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.457233+0000", 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.457231+0000", 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.457283+0000", 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2370000000000002e-05 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.457297+0000", 2024-09-14T04:22:49.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3089e-05 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.457302+0000", 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8370000000000002e-06 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.457342+0000", 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0114000000000003e-05 2024-09-14T04:22:49.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.457463+0000", 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012038499999999999 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.457584+0000", 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012089 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.902851+0000", 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.44526773600000003 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.902857+0000", 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8300000000000001e-06 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.902888+0000", 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0752e-05 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.902984+0000", 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6148000000000002e-05 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.903043+0000", 2024-09-14T04:22:49.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8544000000000001e-05 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.46760.0:51058 13.1 e330/321 ondisk, result = 0)", 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:36.459064+0000", 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 13.070145952000001, 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43680132999999999, 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.658 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.459064+0000", 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.459064+0000", 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.459063+0000", 2024-09-14T04:22:49.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:49.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.459069+0000", 2024-09-14T04:22:49.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8880000000000002e-06 2024-09-14T04:22:49.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895789+0000", 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.436720053 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895796+0000", 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4350000000000002e-06 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895846+0000", 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0096999999999998e-05 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895855+0000", 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6369999999999994e-06 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895866+0000", 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0387000000000001e-05 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.46760.0:51050 13.1 e330/321 ondisk, result = 0)", 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:36.460649+0000", 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 13.068561473999999, 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.435383939, 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.460649+0000", 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.460649+0000", 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.460648+0000", 2024-09-14T04:22:49.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.460654+0000", 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1340000000000001e-06 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895788+0000", 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43513365999999998 2024-09-14T04:22:49.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895796+0000", 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5939999999999994e-06 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895898+0000", 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010176299999999999 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895905+0000", 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5950000000000004e-06 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.896033+0000", 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012807399999999999 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.667 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:51072 13.4 13:255683f5:::benchmark_data_smithi114_288325_object51071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:36.461132+0000", 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 13.068078233, 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.44214294100000001, 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 51072 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.461132+0000", 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.461132+0000", 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.461130+0000", 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:22:49.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.461173+0000", 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3044000000000003e-05 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895759+0000", 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43458562000000001 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.895774+0000", 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5345e-05 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.898144+0000", 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0023702770000000001 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:22:49.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.898324+0000", 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018013999999999999 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.898479+0000", 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000154327 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.902862+0000", 2024-09-14T04:22:49.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0043828829999999997 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.903106+0000", 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024478500000000002 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.903147+0000", 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0250000000000003e-05 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:22:49.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.903232+0000", 2024-09-14T04:22:49.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5677e-05 2024-09-14T04:22:49.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:22:49.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:22:49.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:22:49.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.903275+0000", 2024-09-14T04:22:49.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2721e-05 2024-09-14T04:22:49.674 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.674 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.674 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.675 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:22:49.675 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:22:49.675 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:22:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:49.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:49.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:49.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:49.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:49.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:49.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:49.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:49.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:49.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:49.946 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 61104 61088 100.465 48.0625 0.0235745 0.00994698 2024-09-14T04:22:50.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:50.084 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:50.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:50.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:50.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:50.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:50.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:50.335 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:50 smithi114 ceph-mon[103029]: pgmap v1279: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s; 27512/178794 objects degraded (15.388%); 3009/178794 objects misplaced (1.683%); 2.9 MiB/s, 1 keys/s, 46 objects/s recovering 2024-09-14T04:22:50.335 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:50 smithi114 ceph-mon[106969]: pgmap v1279: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s; 27512/178794 objects degraded (15.388%); 3009/178794 objects misplaced (1.683%); 2.9 MiB/s, 1 keys/s, 46 objects/s recovering 2024-09-14T04:22:50.358 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:50.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:50.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:50.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:50.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:50.600 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:50.632 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:50.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:50.636 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:50 smithi160 ceph-mon[88407]: pgmap v1279: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 51 MiB/s wr, 819 op/s; 27512/178794 objects degraded (15.388%); 3009/178794 objects misplaced (1.683%); 2.9 MiB/s, 1 keys/s, 46 objects/s recovering 2024-09-14T04:22:50.637 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:22:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:22:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:22:50.678 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-14T04:22:50.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:50.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:50.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:50.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:50.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:50.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:50.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:50.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:50.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 61896 61880 99.1582 49.5 0.0238903 0.0100777 2024-09-14T04:22:51.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:51.055 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:51.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:51.084 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:51.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:51.174 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:22:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:51.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:51.235 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:22:51.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:51.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:51.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:51.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:51.468 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:51.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:51.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:51 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3929176938' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:22:51.569 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:51.620 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:51.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3929176938' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:22:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:51 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3929176938' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:22:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:51.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:22:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:22:51] "GET /metrics HTTP/1.1" 200 36565 "" "Prometheus/2.51.0" 2024-09-14T04:22:51.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:51.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:51.747 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:51.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:51.849 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:51.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:51.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:51.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:22:51.946818+0000 min lat: 0.00116686 max lat: 2.64328 avg lat: 0.0102138 2024-09-14T04:22:51.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:22:51.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 62643 62627 97.8464 46.6875 0.0146167 0.0102138 2024-09-14T04:22:51.993 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:52.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:52.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:52.160 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:52.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:52.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:52.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:52.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:52.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:52.451 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:52.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:52.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:52 smithi160 ceph-mon[88407]: pgmap v1280: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 45 MiB/s wr, 721 op/s; 27512/179535 objects degraded (15.324%); 3009/179535 objects misplaced (1.676%); 2.4 MiB/s, 1 keys/s, 39 objects/s recovering 2024-09-14T04:22:52.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:52 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 27512/179535 objects degraded (15.324%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:22:52.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:52 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:52.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:52 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3929176938' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:22:52.508 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:52 smithi160 ceph-mon[88407]: osdmap e335: 8 total, 8 up, 7 in 2024-09-14T04:22:52.536 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:52 smithi114 ceph-mon[106969]: pgmap v1280: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 45 MiB/s wr, 721 op/s; 27512/179535 objects degraded (15.324%); 3009/179535 objects misplaced (1.676%); 2.4 MiB/s, 1 keys/s, 39 objects/s recovering 2024-09-14T04:22:52.536 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:52 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 27512/179535 objects degraded (15.324%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:22:52.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:52 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:52.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:52 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3929176938' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:22:52.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:52 smithi114 ceph-mon[106969]: osdmap e335: 8 total, 8 up, 7 in 2024-09-14T04:22:52.537 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:52 smithi114 ceph-mon[103029]: pgmap v1280: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 45 MiB/s wr, 721 op/s; 27512/179535 objects degraded (15.324%); 3009/179535 objects misplaced (1.676%); 2.4 MiB/s, 1 keys/s, 39 objects/s recovering 2024-09-14T04:22:52.537 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:52 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 27512/179535 objects degraded (15.324%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:22:52.537 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:52 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:22:52.537 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:52 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3929176938' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:22:52.537 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:52 smithi114 ceph-mon[103029]: osdmap e335: 8 total, 8 up, 7 in 2024-09-14T04:22:52.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:52.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:52.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:52.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:52.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:52.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:52.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:52.808 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:52.850 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:52.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:52.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 63431 63415 96.661 49.25 0.0211799 0.0103381 2024-09-14T04:22:52.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:53.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:53.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:53.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:53.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:53.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:53.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:53.269 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:53.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:53.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:53.336 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:22:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:22:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-14T04:22:53.336 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:22:53.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:53.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:53.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:53.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:53.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:53.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:53.639 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:53.685 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:53.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:53.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:53.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:53.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:53.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:53.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:53.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:53.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:53.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:53.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 16 64190 64174 95.4889 47.4375 0.0214785 0.0104657 2024-09-14T04:22:53.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:53.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:54.024 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:22:54.024 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:22:54.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:54.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-14T04:22:54.084 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:54.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:54.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:54.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:54.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:54.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:54.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:54.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:54.355 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:54.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:54.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:54.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:54.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:54.544 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:22:54.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:54.630 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:22:54.630 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:22:54.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:54.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:54.640 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:22:54.640 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:22:54.659 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:22:54.659 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:22:54.680 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:22:54.681 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:22:54.711 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:22:54.712 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:22:54.735 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:22:54.736 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:22:54.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:54.766 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:22:54.766 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:22:54.766 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:54 smithi114 ceph-mon[106969]: pgmap v1282: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 43 MiB/s wr, 683 op/s; 27512/180999 objects degraded (15.200%); 3009/180999 objects misplaced (1.662%); 2.3 MiB/s, 0 keys/s, 37 objects/s recovering 2024-09-14T04:22:54.766 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:54 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/963185557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:54.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:54 smithi114 ceph-mon[103029]: pgmap v1282: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 43 MiB/s wr, 683 op/s; 27512/180999 objects degraded (15.200%); 3009/180999 objects misplaced (1.662%); 2.3 MiB/s, 0 keys/s, 37 objects/s recovering 2024-09-14T04:22:54.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:54 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/963185557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:54.777 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:54.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:54.789 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:22:54.790 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:22:54.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:54.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:54.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:54.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 64920 64904 94.3292 45.625 0.0108871 0.0105959 2024-09-14T04:22:55.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:55.061 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:54 smithi160 ceph-mon[88407]: pgmap v1282: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 43 MiB/s wr, 683 op/s; 27512/180999 objects degraded (15.200%); 3009/180999 objects misplaced (1.662%); 2.3 MiB/s, 0 keys/s, 37 objects/s recovering 2024-09-14T04:22:55.062 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:54 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/963185557' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:22:55.082 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:55.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:55.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:22:55.208 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:55.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:55.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:22:55.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:55.351 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:55.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:55.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:55.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:55.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:55.579 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:55.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:55.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:55.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:55.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:55.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:55.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:55.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:55 smithi160 ceph-mon[88407]: pgmap v1283: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 49 MiB/s wr, 785 op/s; 27512/191148 objects degraded (14.393%); 2758/191148 objects misplaced (1.443%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:22:55.922 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:55.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:55.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 65706 65690 93.3017 49.125 0.0722373 0.0107099 2024-09-14T04:22:56.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:56.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:56.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:56.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:55 smithi114 ceph-mon[106969]: pgmap v1283: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 49 MiB/s wr, 785 op/s; 27512/191148 objects degraded (14.393%); 2758/191148 objects misplaced (1.443%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:22:56.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:55 smithi114 ceph-mon[103029]: pgmap v1283: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 49 MiB/s wr, 785 op/s; 27512/191148 objects degraded (14.393%); 2758/191148 objects misplaced (1.443%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:22:56.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:56.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:56.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:56.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:56.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:56.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:56.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:56.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:56.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:56.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:56.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:56.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:56.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:56.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:56.653 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:22:56.653 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:22:56.692 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:56.733 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:56.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:56.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:56.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:56.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:56.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 66472 66456 92.2922 47.875 0.0295525 0.0108288 2024-09-14T04:22:56.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:57.000 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:22:57.001 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "" 2024-09-14T04:22:57.001 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:22:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:57.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:57.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:57.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:57.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:57.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:57.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:57.307 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:57.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:57.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:57.409 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:57.451 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:57.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-14T04:22:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:57.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:57.553 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:57.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:57.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:57.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:22:57.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:57.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:57.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:57.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:57.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:57.907 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:57.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:57.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 16 67184 67168 91.2531 44.5 0.0287747 0.0109509 2024-09-14T04:22:57.969 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:22:57.996 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:22:58.008 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:58.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:58.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:58.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:58.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:58.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:58.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:58.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:58.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:58.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:58 smithi114 ceph-mon[106969]: pgmap v1284: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 49 MiB/s wr, 785 op/s; 27512/191148 objects degraded (14.393%); 2758/191148 objects misplaced (1.443%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:22:58.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:58 smithi114 ceph-mon[103029]: pgmap v1284: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 49 MiB/s wr, 785 op/s; 27512/191148 objects degraded (14.393%); 2758/191148 objects misplaced (1.443%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:22:58.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:22:58.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:58 smithi160 ceph-mon[88407]: pgmap v1284: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 49 MiB/s wr, 785 op/s; 27512/191148 objects degraded (14.393%); 2758/191148 objects misplaced (1.443%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:22:58.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:22:58.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:58.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:58.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:22:58.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:22:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:58.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:22:58.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:22:58.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:58.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:22:58.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:58.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:58.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:22:58.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:22:58.947 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 67920 67904 90.2902 46 0.0218088 0.0110689 2024-09-14T04:22:58.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:58.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:22:59.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:59.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:59.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:59.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:59.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:59.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:59.349 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:59.392 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:59.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:59.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:59.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:59.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:59.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:22:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 27512/201870 objects degraded (13.629%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:22:59.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:22:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 27512/201870 objects degraded (13.629%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:22:59.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:22:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:22:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 27512/201870 objects degraded (13.629%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:22:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:59.681 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:59.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:22:59.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:22:59.821 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:22:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:59.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:22:59.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:22:59.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:22:59.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 68673 68657 89.3896 47.0625 0.0205307 0.011181 2024-09-14T04:22:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:22:59.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:00.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:00.075 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:00.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:00.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:00.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:00.203 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:00.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:00.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:00.329 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:00.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:00.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:23:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:23:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:23:00.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:00 smithi160 ceph-mon[88407]: pgmap v1285: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 48 MiB/s wr, 768 op/s; 27512/201870 objects degraded (13.629%); 2502/201870 objects misplaced (1.239%); 3.2 MiB/s, 50 objects/s recovering 2024-09-14T04:23:00.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:00.474 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:00.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:00.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:00 smithi114 ceph-mon[103029]: pgmap v1285: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 48 MiB/s wr, 768 op/s; 27512/201870 objects degraded (13.629%); 2502/201870 objects misplaced (1.239%); 3.2 MiB/s, 50 objects/s recovering 2024-09-14T04:23:00.522 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:00 smithi114 ceph-mon[106969]: pgmap v1285: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 48 MiB/s wr, 768 op/s; 27512/201870 objects degraded (13.629%); 2502/201870 objects misplaced (1.239%); 3.2 MiB/s, 50 objects/s recovering 2024-09-14T04:23:00.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:00.600 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:00.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:00.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:00.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:00.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:00.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:00.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:00.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:00.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 69464 69448 88.5741 49.4375 0.00687195 0.0112836 2024-09-14T04:23:00.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:00.993 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:00.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:01.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:01.140 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:01.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:01.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:01.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:01.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:01.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:01.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:01.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:01.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:01.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:01.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:01.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:01.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:01.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:01.703 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:23:01.703 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:23:01.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:01.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:01.736 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:23:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:23:01] "GET /metrics HTTP/1.1" 200 36595 "" "Prometheus/2.51.0" 2024-09-14T04:23:01.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:01.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:01.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:01.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:01.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 70183 70167 87.7013 44.9375 0.0219054 0.0113958 2024-09-14T04:23:01.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:01.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:02.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:02.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:02.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:02.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:02.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:02.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:02.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:02.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:02.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:02.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:02 smithi114 ceph-mon[103029]: pgmap v1286: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 48 MiB/s wr, 768 op/s; 27512/202584 objects degraded (13.581%); 2502/202584 objects misplaced (1.235%); 3.2 MiB/s, 50 objects/s recovering 2024-09-14T04:23:02.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:02 smithi114 ceph-mon[106969]: pgmap v1286: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 48 MiB/s wr, 768 op/s; 27512/202584 objects degraded (13.581%); 2502/202584 objects misplaced (1.235%); 3.2 MiB/s, 50 objects/s recovering 2024-09-14T04:23:02.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:02.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:02.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:02.492 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:02 smithi160 ceph-mon[88407]: pgmap v1286: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 48 MiB/s wr, 768 op/s; 27512/202584 objects degraded (13.581%); 2502/202584 objects misplaced (1.235%); 3.2 MiB/s, 50 objects/s recovering 2024-09-14T04:23:02.522 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:02.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:02.624 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:02.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:02.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:02.727 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 4, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:23:02.727 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:23:02.728 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.8392235526993006 2024-09-14T04:23:02.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 7 0.8392235526993006 2024-09-14T04:23:02.775 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:02.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:02.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:02.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:02.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:02.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 70898 70882 86.8579 44.6875 0.0318987 0.0115064 2024-09-14T04:23:02.961 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:02.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:03.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:03.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:03.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:03.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:03.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:03.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:03.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:23:03.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:03.381 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:03.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:03.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:03.505 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:03.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:03.606 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:03.631 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:03.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:03.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:03.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:03.790 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:03.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:03.806 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:23:03.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:03.915 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:03.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:03.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 71524 71508 85.9398 39.125 0.0469725 0.0116283 2024-09-14T04:23:04.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:04.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:04.051 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:04.059 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:23:04.059 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:23:04.081 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:23:04.081 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:23:04.099 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:23:04.099 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:23:04.117 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:23:04.117 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:23:04.143 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:04.157 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:23:04.158 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:23:04.183 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:23:04.183 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:23:04.192 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:04.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:04.207 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:23:04.207 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:23:04.239 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:23:04.243 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:23:04.243 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:23:04.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:04.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:04.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:04.355 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:04 smithi160 ceph-mon[88407]: pgmap v1287: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 43 MiB/s wr, 686 op/s; 27512/203979 objects degraded (13.488%); 2502/203979 objects misplaced (1.227%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:23:04.356 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:04 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3624112403' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:23:04.356 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:04 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:23:04.356 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:04 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2881631854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:04.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:04.462 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:04.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:04.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:04.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:04.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:04.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:04 smithi114 ceph-mon[103029]: pgmap v1287: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 43 MiB/s wr, 686 op/s; 27512/203979 objects degraded (13.488%); 2502/203979 objects misplaced (1.227%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:23:04.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:04 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3624112403' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:23:04.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:04 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:23:04.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:04 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2881631854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:04 smithi114 ceph-mon[106969]: pgmap v1287: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 43 MiB/s wr, 686 op/s; 27512/203979 objects degraded (13.488%); 2502/203979 objects misplaced (1.227%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:23:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:04 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3624112403' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:23:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:04 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:23:04.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:04 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2881631854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:04.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:04.738 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:04.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:04.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:04.840 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:04.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:04.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:04.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 72155 72139 85.0624 39.4375 0.0162662 0.011749 2024-09-14T04:23:04.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:05.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:05.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:05.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:05.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:05.276 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:05.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:05.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:05.321 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:05 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:23:05.322 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:05 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:23:05.322 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:05 smithi114 ceph-mon[106969]: osdmap e336: 8 total, 8 up, 7 in 2024-09-14T04:23:05.322 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:05 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 27512/203979 objects degraded (13.488%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:05.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:05 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:23:05.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:05 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:23:05.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:05 smithi114 ceph-mon[103029]: osdmap e336: 8 total, 8 up, 7 in 2024-09-14T04:23:05.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:05 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 27512/203979 objects degraded (13.488%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:05.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:05.399 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:05.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:05.494 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:05 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:23:05.494 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:05 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:23:05.494 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:05 smithi160 ceph-mon[88407]: osdmap e336: 8 total, 8 up, 7 in 2024-09-14T04:23:05.495 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:05 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 27512/203979 objects degraded (13.488%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:05.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:05.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:05.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:05.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:05.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:05.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:05.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:05.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:05.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:05.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 72858 72842 84.3007 43.9375 0.00584411 0.0118527 2024-09-14T04:23:05.987 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:05.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:06.053 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:23:06.053 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:23:06.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:72920 13.d 13:b8a2ca37:::benchmark_data_smithi114_288325_object72919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:06.044091+0000", 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0068304330000000003, 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068750189999999996, 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 72920 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:23:06.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044091+0000", 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044091+0000", 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044089+0000", 2024-09-14T04:23:06.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044136+0000", 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7413000000000002e-05 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044160+0000", 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3518000000000001e-05 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044168+0000", 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1689999999999992e-06 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044212+0000", 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4209000000000003e-05 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:23:06.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044355+0000", 2024-09-14T04:23:06.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014264599999999999 2024-09-14T04:23:06.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-14T04:23:06.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044492+0000", 2024-09-14T04:23:06.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013722100000000001 2024-09-14T04:23:06.058 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.058 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:23:06.058 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:72921 13.d 13:b739632d:::benchmark_data_smithi114_288325_object72920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:06.044181+0000", 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0067405590000000001, 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068651509999999999, 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:23:06.059 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 72921 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044181+0000", 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044181+0000", 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:23:06.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044180+0000", 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044214+0000", 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4054999999999998e-05 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044222+0000", 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8849999999999999e-06 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044226+0000", 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0539999999999996e-06 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:23:06.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.047058+0000", 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0028320989999999998 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.047183+0000", 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000124981 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.047315+0000", 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013142500000000001 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.063 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:72922 13.d 13:b3b12664:::benchmark_data_smithi114_288325_object72921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:06.044231+0000", 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0066897340000000001, 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068837519999999999, 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:23:06.064 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 72922 2024-09-14T04:23:06.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.065 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:23:06.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:23:06.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044231+0000", 2024-09-14T04:23:06.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:23:06.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044231+0000", 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044231+0000", 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044290+0000", 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9083000000000003e-05 2024-09-14T04:23:06.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044297+0000", 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8950000000000001e-06 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.044301+0000", 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3930000000000001e-06 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:23:06.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.050261+0000", 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059591679999999999 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.050381+0000", 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000120809 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 1,7", 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.050514+0000", 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013245999999999999 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.068 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:23:06.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.069 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:72923 13.5 13.96179755 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-14T04:23:06.069 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:06.045313+0000", 2024-09-14T04:23:06.069 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0056085650000000003, 2024-09-14T04:23:06.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0058679880000000002, 2024-09-14T04:23:06.069 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:23:06.069 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:23:06.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 72923 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.045313+0000", 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.045313+0000", 2024-09-14T04:23:06.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.045312+0000", 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.045431+0000", 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011961 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:23:06.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.045442+0000", 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0213e-05 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.045447+0000", 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4310000000000002e-06 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.072 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.46760.0:72920 13.d e336/321)", 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:06.049283+0000", 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.001637962, 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001943871, 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.049283+0000", 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.049283+0000", 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.049282+0000", 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.049292+0000", 2024-09-14T04:23:06.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6430000000000007e-06 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.049304+0000", 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2023e-05 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:06.049310+0000", 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1750000000000002e-06 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:23:06.076 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:23:06.076 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 5 2024-09-14T04:23:06.076 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:23:06.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:06.113 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:06.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:06.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:06.239 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:06.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:06.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:06.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:06.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:06.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:06.516 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:06.516 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:06 smithi160 ceph-mon[88407]: pgmap v1289: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 47 MiB/s wr, 748 op/s; 27512/213612 objects degraded (12.879%); 2233/213612 objects misplaced (1.045%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:23:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:06.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:06.556 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:06 smithi114 ceph-mon[103029]: pgmap v1289: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 47 MiB/s wr, 748 op/s; 27512/213612 objects degraded (12.879%); 2233/213612 objects misplaced (1.045%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:23:06.556 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:06 smithi114 ceph-mon[106969]: pgmap v1289: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 47 MiB/s wr, 748 op/s; 27512/213612 objects degraded (12.879%); 2233/213612 objects misplaced (1.045%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:23:06.618 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:06.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:06.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:06.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:06.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:06.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:06.883 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:06.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:06.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:06.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 73525 73509 83.5259 41.6875 0.0148207 0.0119656 2024-09-14T04:23:07.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:07.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:07.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:07.158 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:07.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:07.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:07.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:07.337 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:07.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:07.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:07.464 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:07.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:07.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:07.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:07.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:07.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:07.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:07.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:07.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:07.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:07.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:23:07.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:07.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:07.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:07.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:07.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 74189 74173 82.7754 41.5 0.0109161 0.0120724 2024-09-14T04:23:07.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:07.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:08.051 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:08.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:08.095 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:08.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:08.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:08.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:08.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:08.253 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 0.839224 (d6d7) 2024-09-14T04:23:08.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:08.369 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:08.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:08.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:08.493 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:08.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:08.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:08.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:08.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:08.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:08.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:08 smithi114 ceph-mon[103029]: pgmap v1290: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 47 MiB/s wr, 748 op/s; 27512/213612 objects degraded (12.879%); 2233/213612 objects misplaced (1.045%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:23:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2936952091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.8392235526993006} : dispatch 2024-09-14T04:23:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:08 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.8392235526993006} : dispatch 2024-09-14T04:23:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:08.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:08 smithi114 ceph-mon[106969]: pgmap v1290: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 47 MiB/s wr, 748 op/s; 27512/213612 objects degraded (12.879%); 2233/213612 objects misplaced (1.045%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:23:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2936952091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.8392235526993006} : dispatch 2024-09-14T04:23:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:08 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.8392235526993006} : dispatch 2024-09-14T04:23:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:08.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:08 smithi160 ceph-mon[88407]: pgmap v1290: 89 pgs: 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 47 MiB/s wr, 748 op/s; 27512/213612 objects degraded (12.879%); 2233/213612 objects misplaced (1.045%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:23:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2936952091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.8392235526993006} : dispatch 2024-09-14T04:23:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:08 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 0.8392235526993006} : dispatch 2024-09-14T04:23:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:08.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:08.739 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:08.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:08.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:23:08.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:08.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:08.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:08.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 74420 74404 81.5764 14.4375 0.00314489 0.0121304 2024-09-14T04:23:08.965 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:08.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:08.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:09.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:09.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:09.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:09.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:09.234 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:09.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:09.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:09.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:09.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:09.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:09.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:09.480 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:09.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:09.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:09.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:09.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:09.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.8392235526993006}]': finished 2024-09-14T04:23:09.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:09 smithi114 ceph-mon[103029]: osdmap e337: 8 total, 8 up, 7 in 2024-09-14T04:23:09.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.8392235526993006}]': finished 2024-09-14T04:23:09.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:09 smithi114 ceph-mon[106969]: osdmap e337: 8 total, 8 up, 7 in 2024-09-14T04:23:09.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.8392235526993006}]': finished 2024-09-14T04:23:09.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:09 smithi160 ceph-mon[88407]: osdmap e337: 8 total, 8 up, 7 in 2024-09-14T04:23:09.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:09.727 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:09.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:09.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:09.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:09.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:09.948 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 74420 74404 80.1699 0 - 0.0121304 2024-09-14T04:23:09.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:09.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:09.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:09.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:10.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:10.101 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:10.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:10.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:10.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:10.225 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:10.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:10.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-14T04:23:10.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:10.348 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:10.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:10.450 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:10.484 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:10.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:10.515 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:10 smithi114 ceph-mon[103029]: pgmap v1292: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 7 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 52 MiB/s wr, 828 op/s; 24014/222480 objects degraded (10.794%); 3494/222480 objects misplaced (1.570%); 4.1 MiB/s, 66 objects/s recovering 2024-09-14T04:23:10.515 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:10 smithi114 ceph-mon[103029]: osdmap e338: 8 total, 8 up, 7 in 2024-09-14T04:23:10.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:10 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 24014/222480 objects degraded (10.794%), 7 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:10.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:10 smithi114 ceph-mon[106969]: pgmap v1292: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 7 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 52 MiB/s wr, 828 op/s; 24014/222480 objects degraded (10.794%); 3494/222480 objects misplaced (1.570%); 4.1 MiB/s, 66 objects/s recovering 2024-09-14T04:23:10.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:10 smithi114 ceph-mon[106969]: osdmap e338: 8 total, 8 up, 7 in 2024-09-14T04:23:10.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:10 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 24014/222480 objects degraded (10.794%), 7 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:10.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:10.608 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:23:10.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:10.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:23:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:23:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:23:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:10 smithi160 ceph-mon[88407]: pgmap v1292: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 7 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 52 MiB/s wr, 828 op/s; 24014/222480 objects degraded (10.794%); 3494/222480 objects misplaced (1.570%); 4.1 MiB/s, 66 objects/s recovering 2024-09-14T04:23:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:10 smithi160 ceph-mon[88407]: osdmap e338: 8 total, 8 up, 7 in 2024-09-14T04:23:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:10 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 24014/222480 objects degraded (10.794%), 7 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:10.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:10.734 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:10.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:10.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:10.862 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:10.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:10.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 74574 74558 78.9742 4.8125 0.0271968 0.0126556 2024-09-14T04:23:10.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:10.992 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:10.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:11.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:11.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:11.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:11.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:11.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:11.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:11.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:11.410 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:11.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:11.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:11.530 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:11 smithi160 ceph-mon[88407]: osdmap e339: 8 total, 8 up, 7 in 2024-09-14T04:23:11.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:11.559 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:11.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:11.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:11 smithi114 ceph-mon[106969]: osdmap e339: 8 total, 8 up, 7 in 2024-09-14T04:23:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:23:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:23:11] "GET /metrics HTTP/1.1" 200 36590 "" "Prometheus/2.51.0" 2024-09-14T04:23:11.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:11 smithi114 ceph-mon[103029]: osdmap e339: 8 total, 8 up, 7 in 2024-09-14T04:23:11.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:11.707 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:11.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:11.773 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:11.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:11.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:11.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:11.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:23:11.948566+0000 min lat: 0.00116686 max lat: 2.64328 avg lat: 0.0127468 2024-09-14T04:23:11.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:23:11.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 75297 75281 78.411 45.1875 0.0184108 0.0127468 2024-09-14T04:23:11.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:11.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:11.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:12.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:12.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:12.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:12.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:12.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:12.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:12.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:12.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:12.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:12.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:12.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:12.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:12.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:12.613 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:12 smithi114 ceph-mon[106969]: pgmap v1295: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 7 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 32 MiB/s wr, 519 op/s; 24014/222972 objects degraded (10.770%); 3494/222972 objects misplaced (1.567%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:23:12.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:12 smithi114 ceph-mon[103029]: pgmap v1295: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 7 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 32 MiB/s wr, 519 op/s; 24014/222972 objects degraded (10.770%); 3494/222972 objects misplaced (1.567%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:23:12.636 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:12 smithi160 ceph-mon[88407]: pgmap v1295: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 7 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 32 MiB/s wr, 519 op/s; 24014/222972 objects degraded (10.770%); 3494/222972 objects misplaced (1.567%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:23:12.642 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:23:12.643 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:23:12.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:12.667 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:23:12.667 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:23:12.692 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:23:12.693 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:23:12.709 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:23:12.710 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:23:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:12.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:12.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:12.731 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:23:12.731 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:23:12.754 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:23:12.755 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:23:12.781 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:23:12.782 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:23:12.813 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:23:12.813 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:23:12.826 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:12.873 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:12.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:12.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 76003 75987 77.8488 44.125 0.0161809 0.0128403 2024-09-14T04:23:12.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:13.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:13.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:13.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:13.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:13.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:13.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:13.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:13.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:13.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:13.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:13.397 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:23:13.397 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:23:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:13.398 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:13.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:13.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:13.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:13.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:13.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:13 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1446094936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:13.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:13 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1446094936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:13.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:13.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:13.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:13 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1446094936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:13.751 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:13.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:13.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:13.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:13.921 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:13.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:13.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 76695 76679 77.2907 43.25 0.0439009 0.0129332 2024-09-14T04:23:14.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:14.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:14.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:14.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:14.190 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:14.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:14.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-14T04:23:14.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:14.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:14.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:14.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:14.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:14 smithi114 ceph-mon[103029]: pgmap v1296: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 35 MiB/s wr, 556 op/s; 28742/223635 objects degraded (12.852%); 3494/223635 objects misplaced (1.562%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:23:14.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:14 smithi114 ceph-mon[106969]: pgmap v1296: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 35 MiB/s wr, 556 op/s; 28742/223635 objects degraded (12.852%); 3494/223635 objects misplaced (1.562%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:23:14.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:14.477 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:14.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:14.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:14.635 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:14.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:14.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:14.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:14 smithi160 ceph-mon[88407]: pgmap v1296: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 2 peering, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 35 MiB/s wr, 556 op/s; 28742/223635 objects degraded (12.852%); 3494/223635 objects misplaced (1.562%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:23:14.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:14.775 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:23:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:14.792 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:14.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:14.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:14.946 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:14.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:14.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:14.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 15 77357 77342 76.7216 41.4375 0.0196115 0.0130287 2024-09-14T04:23:15.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:15.072 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:15.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:15.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:15.174 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:15.213 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:15.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:15.286 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:23:15.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:15.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:15.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:15.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:15.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:15.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:15.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:15.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:15.618 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:15.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:15 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 28742/223635 objects degraded (12.852%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:15.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2822020434' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:23:15.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:15 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 28742/223635 objects degraded (12.852%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:15.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2822020434' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:23:15.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:15.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:15 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 28742/223635 objects degraded (12.852%), 8 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2822020434' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:23:15.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:23:15.721 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:15.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:15.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:15.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:15.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:15.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:15.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 78027 78011 76.1761 41.8125 0.00974953 0.0131203 2024-09-14T04:23:16.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:16.030 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:16.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:16.120 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:16.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:16.171 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:16.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:16.248 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 4, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:23:16.248 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:23:16.248 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-14T04:23:16.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-14T04:23:16.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:16.315 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:16.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:16.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:16.456 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:16.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:16.555 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:16 smithi114 ceph-mon[106969]: pgmap v1297: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 43 MiB/s wr, 685 op/s; 35269/228933 objects degraded (15.406%); 3193/228933 objects misplaced (1.395%); 5.1 MiB/s, 82 objects/s recovering 2024-09-14T04:23:16.555 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:16 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:23:16.555 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:16 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2822020434' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:23:16.555 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:16 smithi114 ceph-mon[106969]: osdmap e340: 8 total, 8 up, 7 in 2024-09-14T04:23:16.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:16 smithi114 ceph-mon[103029]: pgmap v1297: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 43 MiB/s wr, 685 op/s; 35269/228933 objects degraded (15.406%); 3193/228933 objects misplaced (1.395%); 5.1 MiB/s, 82 objects/s recovering 2024-09-14T04:23:16.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:16 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:23:16.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:16 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2822020434' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:23:16.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:16 smithi114 ceph-mon[103029]: osdmap e340: 8 total, 8 up, 7 in 2024-09-14T04:23:16.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:16.580 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:16.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:16 smithi160 ceph-mon[88407]: pgmap v1297: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 43 MiB/s wr, 685 op/s; 35269/228933 objects degraded (15.406%); 3193/228933 objects misplaced (1.395%); 5.1 MiB/s, 82 objects/s recovering 2024-09-14T04:23:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:16 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:23:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:16 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2822020434' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:23:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:16 smithi160 ceph-mon[88407]: osdmap e340: 8 total, 8 up, 7 in 2024-09-14T04:23:16.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:16.711 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:16.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:16.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:16.837 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:16.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:16.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:16.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 78641 78625 75.5945 38.375 0.0213075 0.0132021 2024-09-14T04:23:16.960 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:16.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:17.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:17.101 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:17.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:17.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:17.211 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:17.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:17.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:17.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:17.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:17.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:17.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:17.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:17.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:17.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:17.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:17.642 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:17.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:17.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:17.772 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:17.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:17.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:17.895 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:17.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:17.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:17.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 79259 79243 75.0343 38.625 0.0167337 0.0133202 2024-09-14T04:23:17.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:18.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:18.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:18.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:18.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:23:18.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:18.190 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:18.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:18.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:18.338 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:18.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:18.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:18.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:18.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:18.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:18.537 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:18 smithi114 ceph-mon[103029]: pgmap v1299: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 17 MiB/s wr, 273 op/s; 35269/228933 objects degraded (15.406%); 3193/228933 objects misplaced (1.395%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:23:18.537 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:18 smithi114 ceph-mon[106969]: pgmap v1299: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 17 MiB/s wr, 273 op/s; 35269/228933 objects degraded (15.406%); 3193/228933 objects misplaced (1.395%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:23:18.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:18.606 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:18 smithi160 ceph-mon[88407]: pgmap v1299: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 17 MiB/s wr, 273 op/s; 35269/228933 objects degraded (15.406%); 3193/228933 objects misplaced (1.395%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:23:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:18.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:18.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:18.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:18.772 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:23:18.772 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:23:18.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:18.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:18.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:18.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:18.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:18.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 79936 79920 74.5459 42.3125 0.0315188 0.0134075 2024-09-14T04:23:19.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:19.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:19.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:19.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:19.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:19.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:19.281 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:19.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:19.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:19.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:19.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:19.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:19.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:19.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:19.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:19.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-14T04:23:19.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:19.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:19.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:19.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:19.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:19.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:19.920 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:23:19.921 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:23:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:23:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:23:19.922 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:23:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:23:19.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:19.942 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:23:19.943 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:23:19.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 80637 80621 74.0938 43.8125 0.0142299 0.0134902 2024-09-14T04:23:19.972 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:19.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:19.974 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:23:19.974 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:23:20.009 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:23:20.009 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:23:20.036 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:23:20.036 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:23:20.060 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:23:20.061 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:23:20.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:20.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:20.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:20.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:20.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:20.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:20.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:20.396 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:20.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:20.498 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:20.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:20.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:20.551 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:20 smithi114 ceph-mon[106969]: pgmap v1300: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 36 MiB/s wr, 570 op/s; 35269/238119 objects degraded (14.812%); 3091/238119 objects misplaced (1.298%); 2.8 MiB/s, 45 objects/s recovering 2024-09-14T04:23:20.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:20 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/238119 objects degraded (14.812%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:20.552 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:20 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2055181933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:20.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:20 smithi114 ceph-mon[103029]: pgmap v1300: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 36 MiB/s wr, 570 op/s; 35269/238119 objects degraded (14.812%); 3091/238119 objects misplaced (1.298%); 2.8 MiB/s, 45 objects/s recovering 2024-09-14T04:23:20.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:20 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/238119 objects degraded (14.812%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:20.552 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:20 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2055181933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:20.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:20 smithi160 ceph-mon[88407]: pgmap v1300: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 36 MiB/s wr, 570 op/s; 35269/238119 objects degraded (14.812%); 3091/238119 objects misplaced (1.298%); 2.8 MiB/s, 45 objects/s recovering 2024-09-14T04:23:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:20 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/238119 objects degraded (14.812%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:20 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2055181933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:23:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:23:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:23:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:20.669 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:20.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:20.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:20.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:20.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:20.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:20.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 81357 81341 73.6721 45 0.0060332 0.013568 2024-09-14T04:23:20.958 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:20.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:21.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:21.084 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:21.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:21.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:21.210 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:21.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:21.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:21.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:21.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:21.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:21.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:21.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:21.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:21.515 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:21.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:21.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:23:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:23:21] "GET /metrics HTTP/1.1" 200 36590 "" "Prometheus/2.51.0" 2024-09-14T04:23:21.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:21.638 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:21.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:21.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:21.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:21.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:21.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:21.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:21.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:21.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:21.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 82053 82037 73.2411 43.5 0.0243256 0.0136474 2024-09-14T04:23:22.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:22.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:22.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:22.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:22.185 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:22.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:22.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:22.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:22.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-14T04:23:22.295 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:22.332 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:22.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:22.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:22.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:22.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:22.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:22 smithi114 ceph-mon[106969]: pgmap v1301: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 33 MiB/s wr, 527 op/s; 35269/238800 objects degraded (14.769%); 3091/238800 objects misplaced (1.294%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:23:22.533 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:22 smithi114 ceph-mon[103029]: pgmap v1301: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 33 MiB/s wr, 527 op/s; 35269/238800 objects degraded (14.769%); 3091/238800 objects misplaced (1.294%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:23:22.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:22.607 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:22.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:22.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:22 smithi160 ceph-mon[88407]: pgmap v1301: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 33 MiB/s wr, 527 op/s; 35269/238800 objects degraded (14.769%); 3091/238800 objects misplaced (1.294%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:23:22.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:22.737 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:22.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:22.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:22.880 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:22.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:22.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 82781 82765 72.8503 45.5 0.0311658 0.0137213 2024-09-14T04:23:22.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:23.007 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:23.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:23.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:23.135 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:23.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:23.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:23.236 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:23.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:23:23.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:23.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:23.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:23.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:23.433 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:23.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:23.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:23.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:23.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:23.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:23.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:23.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:23.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:23.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:23.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:23.919 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:23.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:23.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:23.949 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 83482 83466 72.4469 43.8125 0.0437629 0.0137971 2024-09-14T04:23:24.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:24.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:24.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:24.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:24.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:24.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:24.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:24.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:24.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:24.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:24.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:24.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:24.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:24.444 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:24.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:24.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:24.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:24.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:24.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:24.686 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:24.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:24.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:24.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:24.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:24.857 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:24.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:24.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 84202 84186 72.0709 45 0.0113342 0.0138689 2024-09-14T04:23:24.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:24.984 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:24.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:25.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:25.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:24 smithi114 ceph-mon[106969]: pgmap v1302: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 34 MiB/s wr, 546 op/s; 35269/240030 objects degraded (14.694%); 3091/240030 objects misplaced (1.288%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:23:25.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/240030 objects degraded (14.694%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:25.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:24 smithi114 ceph-mon[103029]: pgmap v1302: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 34 MiB/s wr, 546 op/s; 35269/240030 objects degraded (14.694%); 3091/240030 objects misplaced (1.288%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:23:25.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/240030 objects degraded (14.694%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:25.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:25.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:25.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:24 smithi160 ceph-mon[88407]: pgmap v1302: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 34 MiB/s wr, 546 op/s; 35269/240030 objects degraded (14.694%); 3091/240030 objects misplaced (1.288%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:23:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/240030 objects degraded (14.694%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:25.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:25.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:25.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:25.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:25.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:25.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:25.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:25.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:25.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:25.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:25.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:25.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:25.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:25.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:25.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:25.851 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:25.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:25.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 84901 84885 71.6873 43.6875 0.0137927 0.0139438 2024-09-14T04:23:25.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:25.993 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:25.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:26.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:25 smithi114 ceph-mon[103029]: pgmap v1303: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 42 MiB/s wr, 679 op/s; 35269/249321 objects degraded (14.146%); 2971/249321 objects misplaced (1.192%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:26.004 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:25 smithi114 ceph-mon[106969]: pgmap v1303: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 42 MiB/s wr, 679 op/s; 35269/249321 objects degraded (14.146%); 2971/249321 objects misplaced (1.192%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:26.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:26.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:26.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:25 smithi160 ceph-mon[88407]: pgmap v1303: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 42 MiB/s wr, 679 op/s; 35269/249321 objects degraded (14.146%); 2971/249321 objects misplaced (1.192%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:26.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:26.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:26.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:26.283 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:23:26.283 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:23:26.283 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.284 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:84930 13.0 e340/338 13:000cbf3c:::benchmark_data_smithi114_288325_object84929:head v 340'5195, mlcod=0'0)", 2024-09-14T04:23:26.284 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:25.997933+0000", 2024-09-14T04:23:26.284 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0047046040000000003, 2024-09-14T04:23:26.284 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0047337450000000001, 2024-09-14T04:23:26.284 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:26.284 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:26.285 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:23:26.285 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:26.285 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.285 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:26.285 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:25.997933+0000", 2024-09-14T04:23:26.285 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:26.286 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.286 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.286 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:26.286 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:25.997933+0000", 2024-09-14T04:23:26.286 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:26.286 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.286 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.286 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:26.286 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:25.997934+0000", 2024-09-14T04:23:26.287 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.15e-06 2024-09-14T04:23:26.287 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.287 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.287 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:26.287 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:25.997964+0000", 2024-09-14T04:23:26.287 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9617000000000001e-05 2024-09-14T04:23:26.287 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.288 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.288 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:26.288 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:25.997977+0000", 2024-09-14T04:23:26.288 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3725e-05 2024-09-14T04:23:26.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.288 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.288 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:26.288 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:25.997986+0000", 2024-09-14T04:23:26.288 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4130000000000006e-06 2024-09-14T04:23:26.289 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.289 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.289 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:26.289 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:25.998060+0000", 2024-09-14T04:23:26.289 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3869999999999996e-05 2024-09-14T04:23:26.289 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.289 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.289 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:25.998091+0000", 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0997999999999997e-05 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:84931 13.0 e340/338)", 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:26.000712+0000", 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001925274, 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002004086, 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:26.290 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:26.291 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:23:26.291 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:26.291 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.291 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:26.291 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:26.000712+0000", 2024-09-14T04:23:26.291 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:26.291 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.291 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.291 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:26.292 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:26.000712+0000", 2024-09-14T04:23:26.292 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:26.292 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.292 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.292 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:26.292 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:26.000714+0000", 2024-09-14T04:23:26.292 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4729999999999999e-06 2024-09-14T04:23:26.292 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.292 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.292 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:26.000741+0000", 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6720000000000002e-05 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:26.000754+0000", 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3885e-05 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:26.000763+0000", 2024-09-14T04:23:26.293 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6459999999999999e-06 2024-09-14T04:23:26.294 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:26.294 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:26.294 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:26.294 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:26.294 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:23:26.294 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 2 2024-09-14T04:23:26.294 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:23:26.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:26.422 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:26.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:26.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:26.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:26.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:26.551 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:26.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:26.695 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:26.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:26.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:26.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:26.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:26.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:26 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2376826784' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:23:26.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:26 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2376826784' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:23:26.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:26.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 85566 85550 71.2856 41.5625 0.0425844 0.0140229 2024-09-14T04:23:26.957 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:26.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:27.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:27.083 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:27.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:26 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2376826784' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:23:27.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:27.209 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:27.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:27.225 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:23:27.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:27.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:27.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:27.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:27.465 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:27.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:27.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:27.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:27.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:27.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:27.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:27.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:27.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:27.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:27.863 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:27.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:27.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:27 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:23:27.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:27 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2376826784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:23:27.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:27 smithi114 ceph-mon[103029]: osdmap e341: 8 total, 8 up, 7 in 2024-09-14T04:23:27.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:27 smithi114 ceph-mon[103029]: pgmap v1305: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 42 MiB/s wr, 679 op/s; 35269/249321 objects degraded (14.146%); 2971/249321 objects misplaced (1.192%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:27.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:27 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:23:27.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:27 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2376826784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:23:27.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:27 smithi114 ceph-mon[106969]: osdmap e341: 8 total, 8 up, 7 in 2024-09-14T04:23:27.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:27 smithi114 ceph-mon[106969]: pgmap v1305: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 42 MiB/s wr, 679 op/s; 35269/249321 objects degraded (14.146%); 2971/249321 objects misplaced (1.192%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:27.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 86253 86237 70.9126 42.9375 0.0120062 0.0140964 2024-09-14T04:23:27.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:27.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:27.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:28.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:28.139 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:28.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:28.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:27 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:23:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:27 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2376826784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:23:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:27 smithi160 ceph-mon[88407]: osdmap e341: 8 total, 8 up, 7 in 2024-09-14T04:23:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:27 smithi160 ceph-mon[88407]: pgmap v1305: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 42 MiB/s wr, 679 op/s; 35269/249321 objects degraded (14.146%); 2971/249321 objects misplaced (1.192%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:28.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:28.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:28.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:28.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:28.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:28.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:28.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:28.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:28.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:28.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:28.670 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:28.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:28.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:28.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:28.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:28.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:28.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:28.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:28.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:28.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 86943 86927 70.5516 43.125 0.0202787 0.0141677 2024-09-14T04:23:29.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:29.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:29.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:29.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:29.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:29.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:29.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:29.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:29.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:29.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:29.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:29.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:29.635 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:29.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:29.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:29.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:29.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:29.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:29.926 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:29.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:29.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 87627 87611 70.1952 42.75 0.0256232 0.0142396 2024-09-14T04:23:29.986 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:23:30.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:30.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:30.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:30.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:30.204 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:30.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:30.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:30.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:30.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:30.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:30.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:30 smithi114 ceph-mon[103029]: pgmap v1306: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 44 MiB/s wr, 701 op/s; 35269/259182 objects degraded (13.608%); 2852/259182 objects misplaced (1.100%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:23:30.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:30 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/259182 objects degraded (13.608%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:30 smithi114 ceph-mon[106969]: pgmap v1306: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 44 MiB/s wr, 701 op/s; 35269/259182 objects degraded (13.608%); 2852/259182 objects misplaced (1.100%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:23:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:30 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/259182 objects degraded (13.608%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:30.432 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:30.473 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:30.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:30.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:30.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:30.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:30.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:30.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:30.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-14T04:23:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:30 smithi160 ceph-mon[88407]: pgmap v1306: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 44 MiB/s wr, 701 op/s; 35269/259182 objects degraded (13.608%); 2852/259182 objects misplaced (1.100%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:23:30.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:30 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/259182 objects degraded (13.608%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:23:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:23:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:23:30.703 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:30.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:30.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:30.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:30.882 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:30.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:30.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 88293 88277 69.8335 41.625 0.0276163 0.014313 2024-09-14T04:23:30.985 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:31.019 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:31.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:31.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:31.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:31.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:31.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:31.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:31.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:31.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:31.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:31.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:31.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:31.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:31.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:31.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:31.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:31.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:31.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:31.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:31.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:31.727 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:23:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:23:31] "GET /metrics HTTP/1.1" 200 36610 "" "Prometheus/2.51.0" 2024-09-14T04:23:31.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:31.819 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:31.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:31.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:31.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:31.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:23:31.950219+0000 min lat: 0.00116686 max lat: 2.64328 avg lat: 0.0144019 2024-09-14T04:23:31.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:23:31.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 88856 88840 69.4004 35.1875 0.0178434 0.0144019 2024-09-14T04:23:32.047 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:32.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:32.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:32.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:32.256 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:32.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:32.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:32.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:32 smithi114 ceph-mon[106969]: pgmap v1307: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 44 MiB/s wr, 698 op/s; 35269/259752 objects degraded (13.578%); 2852/259752 objects misplaced (1.098%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:23:32.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:23:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:32 smithi114 ceph-mon[103029]: pgmap v1307: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 44 MiB/s wr, 698 op/s; 35269/259752 objects degraded (13.578%); 2852/259752 objects misplaced (1.098%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:23:32.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:23:32.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:32.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:32.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:32.512 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:32.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:32.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:32.644 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:32.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:32 smithi160 ceph-mon[88407]: pgmap v1307: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 44 MiB/s wr, 698 op/s; 35269/259752 objects degraded (13.578%); 2852/259752 objects misplaced (1.098%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:23:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:23:32.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:32.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:32.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:32.874 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:23:32.875 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:23:32.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:32.905 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:23:32.905 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:23:32.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:32.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:32.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:32.928 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:23:32.929 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:23:32.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 89411 89395 68.9718 34.6875 0.0523234 0.0144927 2024-09-14T04:23:32.951 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:23:32.951 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:23:32.976 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:23:32.976 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:23:33.001 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:23:33.001 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:23:33.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:33.028 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:23:33.028 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:23:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:33.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:33.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:33.053 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:23:33.054 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:23:33.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:33.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:33.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:33.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:33.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:33.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:33.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:33.440 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:33.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:33.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:33.515 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:33 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2218762282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:33.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:33 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2218762282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:33.542 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:33.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-14T04:23:33.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:33.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:33 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2218762282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:33.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:33.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:33.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:33.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:33.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:33.829 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:33.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:33.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:33.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 90051 90035 68.6184 40 0.0369358 0.0145676 2024-09-14T04:23:33.956 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:33.980 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:33.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:34.082 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:34.116 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:34.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:34.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:34.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:34 smithi114 ceph-mon[103029]: pgmap v1308: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 44 MiB/s wr, 696 op/s; 35269/260943 objects degraded (13.516%); 2852/260943 objects misplaced (1.093%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:23:34.264 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:34 smithi114 ceph-mon[106969]: pgmap v1308: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 44 MiB/s wr, 696 op/s; 35269/260943 objects degraded (13.516%); 2852/260943 objects misplaced (1.093%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:23:34.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:34.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:34.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:34.371 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:34.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:34.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:34.421 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:34.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:34.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:34.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:34.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:34.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:34.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:34 smithi160 ceph-mon[88407]: pgmap v1308: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 44 MiB/s wr, 696 op/s; 35269/260943 objects degraded (13.516%); 2852/260943 objects misplaced (1.093%); 1.5 MiB/s, 23 objects/s recovering 2024-09-14T04:23:34.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:34.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:34.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:34.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:34.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:34.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:34.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:34.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:34.950 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 90610 90594 68.2126 34.9375 0.0306999 0.0146537 2024-09-14T04:23:35.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:35.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:35.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:35.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:35.230 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:35.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:35.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:35.358 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:35.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:35.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:35 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/260943 objects degraded (13.516%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:35.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:35 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/260943 objects degraded (13.516%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:35.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:35 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/260943 objects degraded (13.516%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:35.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:35.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:35.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:35.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:35.612 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:35.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:35.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:35.738 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:35.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:35.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:35.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:35.868 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:35.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:35.951 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 91297 91281 67.9117 42.9375 0.017949 0.0147193 2024-09-14T04:23:35.969 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:36.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:36.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:36.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:36.165 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:36.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:36.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:36 smithi114 ceph-mon[106969]: pgmap v1309: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 41 MiB/s wr, 663 op/s; 35269/269226 objects degraded (13.100%); 2741/269226 objects misplaced (1.018%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:36.249 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:36 smithi114 ceph-mon[103029]: pgmap v1309: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 41 MiB/s wr, 663 op/s; 35269/269226 objects degraded (13.100%); 2741/269226 objects misplaced (1.018%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:36.267 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:36.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:36.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:36.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:23:36.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:36.458 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:36.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:36.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:36.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:36.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:36 smithi160 ceph-mon[88407]: pgmap v1309: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 41 MiB/s wr, 663 op/s; 35269/269226 objects degraded (13.100%); 2741/269226 objects misplaced (1.018%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:36.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:36.727 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:36.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:36.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:36.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:36.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:36.951 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 91919 91903 67.57 38.875 0.0237329 0.0147929 2024-09-14T04:23:36.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:37.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:37.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:37.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:37.126 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:37.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:37.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:37.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:37.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:37.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:37.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:37.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:37.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:37.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:37.528 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:37.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:37.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:37.698 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:37.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:37.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:37.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:37.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:37.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:37.951 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 92559 92543 67.2494 40 0.0267646 0.0148634 2024-09-14T04:23:38.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:38.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:38.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:38.135 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:38.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:38.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:38.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:38.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:38.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:38.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:38.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:38.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:38 smithi160 ceph-mon[88407]: pgmap v1310: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 40 MiB/s wr, 636 op/s; 35269/269226 objects degraded (13.100%); 2741/269226 objects misplaced (1.018%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:38.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:38.475 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:38 smithi114 ceph-mon[106969]: pgmap v1310: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 40 MiB/s wr, 636 op/s; 35269/269226 objects degraded (13.100%); 2741/269226 objects misplaced (1.018%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:38.475 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:38.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:38 smithi114 ceph-mon[103029]: pgmap v1310: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 40 MiB/s wr, 636 op/s; 35269/269226 objects degraded (13.100%); 2741/269226 objects misplaced (1.018%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:38.476 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:38.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:38.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:38.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:38.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:38.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:38.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:38.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:38.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:38.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:38.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:38.951 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 93195 93179 66.9333 39.75 0.04688 0.0149339 2024-09-14T04:23:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:38.962 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:38.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:39.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:39.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:39.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:39.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:39.236 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:39.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:39.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:39.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:39.362 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:39.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:39.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:39.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:39.514 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:39.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:39.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:39.639 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:39.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:39.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:39.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:39.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:39.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:39.945 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:39.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:39.951 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 93805 93789 66.6059 38.125 0.0387304 0.0150072 2024-09-14T04:23:40.047 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:40.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:40.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:40.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:40.194 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:40.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:40.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:40.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:40.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:40.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:40.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:40 smithi114 ceph-mon[103029]: pgmap v1311: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 50 MiB/s wr, 796 op/s; 35269/278004 objects degraded (12.687%); 2602/278004 objects misplaced (0.936%); 1.9 MiB/s, 30 objects/s recovering 2024-09-14T04:23:40.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:40 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/278004 objects degraded (12.687%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:40.460 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:40 smithi114 ceph-mon[106969]: pgmap v1311: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 50 MiB/s wr, 796 op/s; 35269/278004 objects degraded (12.687%); 2602/278004 objects misplaced (0.936%); 1.9 MiB/s, 30 objects/s recovering 2024-09-14T04:23:40.460 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:40 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/278004 objects degraded (12.687%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:40.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:40.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:40.491 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:40 smithi160 ceph-mon[88407]: pgmap v1311: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 50 MiB/s wr, 796 op/s; 35269/278004 objects degraded (12.687%); 2602/278004 objects misplaced (0.936%); 1.9 MiB/s, 30 objects/s recovering 2024-09-14T04:23:40.491 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:40 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/278004 objects degraded (12.687%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:23:40.491 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:23:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:23:40.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:23:40.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:40.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:40.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:40.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:40.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:40.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:40.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:40.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:40.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:40.951 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 94441 94425 66.3039 39.75 0.0251766 0.0150753 2024-09-14T04:23:41.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:41.038 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:41.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:41.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:41.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:41.166 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:41.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:41.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:41.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:41.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:41.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:41.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:41.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:41.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:41.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:41.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:41.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:23:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:23:41] "GET /metrics HTTP/1.1" 200 36608 "" "Prometheus/2.51.0" 2024-09-14T04:23:41.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:41.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:41.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:41.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:41.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:41.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:41.869 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:23:41.869 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:23:41.940 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:41.951 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 90 13 95062 95049 66.0005 39 0.0358263 0.0151452 2024-09-14T04:23:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:41.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:41.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:42.075 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:42.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:42.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:42.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:42.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:42.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:42.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:42.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:42.344 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:42.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:42.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:41 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:42.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:41 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:42.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:41 smithi114 ceph-mon[103029]: pgmap v1312: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 34 MiB/s wr, 540 op/s; 35269/278649 objects degraded (12.657%); 2602/278649 objects misplaced (0.934%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:23:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:41 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:41 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:41 smithi114 ceph-mon[106969]: pgmap v1312: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 34 MiB/s wr, 540 op/s; 35269/278649 objects degraded (12.657%); 2602/278649 objects misplaced (0.934%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:23:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:41 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:41 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:42.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:41 smithi160 ceph-mon[88407]: pgmap v1312: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 34 MiB/s wr, 540 op/s; 35269/278649 objects degraded (12.657%); 2602/278649 objects misplaced (0.934%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:23:42.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:42.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:42.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:42.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:42.591 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:42.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:42.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-14T04:23:42.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:42.730 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:42.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:42.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:42.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:42.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0217 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 95062 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 65.9994 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 48.4315 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 233.875 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 0 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1055 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 774.912 2024-09-14T04:23:42.952 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3742 2024-09-14T04:23:42.953 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 0 2024-09-14T04:23:42.953 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0151466 2024-09-14T04:23:42.953 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0481204 2024-09-14T04:23:42.953 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 2.64328 2024-09-14T04:23:42.953 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00116686 2024-09-14T04:23:42.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:42.962 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-14T04:23:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:42.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:43.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-14T04:23:43.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:43.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:43.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:43.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:43.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:43.295 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:23:43.295 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:23:43.325 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:23:43.325 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:23:43.347 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:23:43.348 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:23:43.364 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:23:43.364 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:23:43.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:43.392 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:23:43.392 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:23:43.399 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:43.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:43.419 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:23:43.419 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:23:43.463 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:23:43.463 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:23:43.492 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:23:43.492 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:23:43.502 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:43.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:43.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:43.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:43.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:43.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:43.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:43.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:43.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:43.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:43.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:43.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:43.949 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:43.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:44.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:44.076 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:44.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:44.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:44.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:44.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:44.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:44.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:44.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:44.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:44.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:44.464 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:44.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:44.557 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:44 smithi114 ceph-mon[106969]: pgmap v1313: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 35 MiB/s wr, 556 op/s; 35269/279789 objects degraded (12.606%); 2602/279789 objects misplaced (0.930%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:23:44.558 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:44 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4218634281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:44.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:44 smithi114 ceph-mon[103029]: pgmap v1313: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 35 MiB/s wr, 556 op/s; 35269/279789 objects degraded (12.606%); 2602/279789 objects misplaced (0.930%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:23:44.558 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:44 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4218634281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:44.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:44.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:44.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:44 smithi160 ceph-mon[88407]: pgmap v1313: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 35 MiB/s wr, 556 op/s; 35269/279789 objects degraded (12.606%); 2602/279789 objects misplaced (0.930%); 1.3 MiB/s, 20 objects/s recovering 2024-09-14T04:23:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:44 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4218634281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:44.659 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:44.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:44.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:44.734 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:44.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:44.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:44.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:44.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:44.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:44.996 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:44.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:45.098 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:45.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:45.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:45.191 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:23:45.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:45.262 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:45.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:45.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:45.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:45 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1936280706' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:23:45.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:45 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:23:45.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:45 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/279789 objects degraded (12.606%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:23:45.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:45 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1936280706' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:23:45.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:45 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:23:45.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:45 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/279789 objects degraded (12.606%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:23:45.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:45.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:45.510 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:45.553 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:45.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:45.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:45 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1936280706' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:23:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:45 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:23:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:45 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/279789 objects degraded (12.606%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:23:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:45.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:45.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:45.811 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:45.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:45.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:45.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:45.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:46.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:46.093 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:46.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:46.140 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:46.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:46.218 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:46.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:46.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:46.361 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:46.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:46.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:46.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:46.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:46.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:23:46.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:46.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:46 smithi114 ceph-mon[106969]: pgmap v1314: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 42 MiB/s wr, 706 op/s; 35269/284142 objects degraded (12.412%); 2364/284142 objects misplaced (0.832%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:23:46.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:46 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:23:46.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:46 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:23:46.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:46 smithi114 ceph-mon[106969]: osdmap e342: 8 total, 8 up, 7 in 2024-09-14T04:23:46.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:46 smithi114 ceph-mon[103029]: pgmap v1314: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 42 MiB/s wr, 706 op/s; 35269/284142 objects degraded (12.412%); 2364/284142 objects misplaced (0.832%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:23:46.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:46 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:23:46.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:46 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:23:46.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:46 smithi114 ceph-mon[103029]: osdmap e342: 8 total, 8 up, 7 in 2024-09-14T04:23:46.645 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:46.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:46.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:46 smithi160 ceph-mon[88407]: pgmap v1314: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 42 MiB/s wr, 706 op/s; 35269/284142 objects degraded (12.412%); 2364/284142 objects misplaced (0.832%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:23:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:46 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:23:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:46 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:23:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:46 smithi160 ceph-mon[88407]: osdmap e342: 8 total, 8 up, 7 in 2024-09-14T04:23:46.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:46.776 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:46.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:46.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:46.919 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:46.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:47.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:47.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:47.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:47.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:47.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:47.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:47.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:47.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:47.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:47.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:47.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:47.430 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:47.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:47.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:47.556 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:47.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:47.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:47.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:47.685 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:47.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:47.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:47.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:47.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:47.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:47.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:47.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:48.060 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:48.096 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:48.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:48.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:48.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:48.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:48.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:48.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:48.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:48.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:48.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:48 smithi114 ceph-mon[106969]: pgmap v1316: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 33 MiB/s wr, 571 op/s; 35269/284142 objects degraded (12.412%); 2364/284142 objects misplaced (0.832%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:23:48.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:48 smithi114 ceph-mon[103029]: pgmap v1316: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 33 MiB/s wr, 571 op/s; 35269/284142 objects degraded (12.412%); 2364/284142 objects misplaced (0.832%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:23:48.471 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:48.494 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:48.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:48.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:48.617 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:23:48.617 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:23:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:23:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:23:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206308 11.6 11:63734b78:::benchmark_data_smithi114_223345_object94371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:23:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.960890+0000", 2024-09-14T04:23:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 470.65344818699998, 2024-09-14T04:23:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.33796224600000002, 2024-09-14T04:23:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206308 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960890+0000", 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960890+0000", 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960890+0000", 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8599999999999999e-07 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960892+0000", 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.074e-06 2024-09-14T04:23:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960898+0000", 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8900000000000004e-06 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960903+0000", 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6430000000000003e-06 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.964467+0000", 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035639030000000002 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.964568+0000", 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000101437 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,2", 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.972215+0000", 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0076467030000000004 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.984009+0000", 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011793483 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.984065+0000", 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6131999999999997e-05 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.298573+0000", 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31450850299999999 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.298734+0000", 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016041199999999999 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.298852+0000", 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011868 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:54910 12.c 12:380aa551:::benchmark_data_smithi114_253692_object54909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:17:53.600122+0000", 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 355.01421569500002, 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.14566506500000001, 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:23:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 54910 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600122+0000", 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600122+0000", 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600124+0000", 2024-09-14T04:23:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6279999999999999e-06 2024-09-14T04:23:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600147+0000", 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2909999999999999e-05 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600160+0000", 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3202e-05 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600168+0000", 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7079999999999997e-06 2024-09-14T04:23:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600224+0000", 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6178999999999999e-05 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600371+0000", 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014686899999999999 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-14T04:23:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600516+0000", 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000144943 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.603830+0000", 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033139240000000002 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.603921+0000", 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1420000000000002e-05 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.745695+0000", 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.14177393699999999 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.745747+0000", 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2148999999999998e-05 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.745788+0000", 2024-09-14T04:23:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0195999999999999e-05 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:59150 12.3 12:cd0072a9:::benchmark_data_smithi114_253692_object59149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:17:56.926213+0000", 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 351.68812472899998, 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.19487368399999999, 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:23:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:23:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:23:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 59150 2024-09-14T04:23:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926213+0000", 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926213+0000", 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926215+0000", 2024-09-14T04:23:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3799999999999999e-06 2024-09-14T04:23:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926236+0000", 2024-09-14T04:23:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0849999999999999e-05 2024-09-14T04:23:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926248+0000", 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1831000000000001e-05 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926255+0000", 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2729999999999997e-06 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926315+0000", 2024-09-14T04:23:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0115000000000003e-05 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926446+0000", 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013073300000000001 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926563+0000", 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011694 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.928752+0000", 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002189346 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.929480+0000", 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072826499999999997 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.120965+0000", 2024-09-14T04:23:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.191485022 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.121030+0000", 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4620000000000001e-05 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.121087+0000", 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7309000000000003e-05 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:59166 12.3 12:c2097a96:::benchmark_data_smithi114_253692_object59165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:17:56.962037+0000", 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 351.65230135899998, 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.159174488, 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:23:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 59166 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962037+0000", 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962037+0000", 2024-09-14T04:23:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962039+0000", 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8479999999999999e-06 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962062+0000", 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3563000000000001e-05 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962077+0000", 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4324999999999999e-05 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962084+0000", 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2320000000000004e-06 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962141+0000", 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6993000000000001e-05 2024-09-14T04:23:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962288+0000", 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014693099999999999 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.968967+0000", 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0066796920000000001 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.644 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.971082+0000", 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021143569999999999 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.971135+0000", 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3498000000000002e-05 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.121132+0000", 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.149997043 2024-09-14T04:23:48.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.121179+0000", 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7141000000000001e-05 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.121211+0000", 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1865000000000001e-05 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.646 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.647 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.647 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:103395 12.3 12:c488e3d1:::benchmark_data_smithi114_253692_object103394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:23:48.647 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.404797+0000", 2024-09-14T04:23:48.647 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 310.20954091300001, 2024-09-14T04:23:48.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.343085796, 2024-09-14T04:23:48.647 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.647 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.647 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.647 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:23:48.648 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:23:48.648 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:23:48.648 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103395 2024-09-14T04:23:48.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.648 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404797+0000", 2024-09-14T04:23:48.648 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404797+0000", 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404799+0000", 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3859999999999999e-06 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404826+0000", 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7393e-05 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404889+0000", 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2936999999999997e-05 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404900+0000", 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0825999999999999e-05 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.650 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405289+0000", 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00038889300000000002 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405410+0000", 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012131899999999999 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405544+0000", 2024-09-14T04:23:48.651 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013358200000000001 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.409765+0000", 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042217369999999997 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.412481+0000", 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027151839999999998 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747776+0000", 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.33529530499999999 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747838+0000", 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1815000000000004e-05 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747883+0000", 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5419e-05 2024-09-14T04:23:48.653 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:103415 12.3 12:c72de31a:::benchmark_data_smithi114_253692_object103414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.421345+0000", 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 310.19299336900002, 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32664568199999999, 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:23:48.654 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103415 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421345+0000", 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421345+0000", 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.656 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.656 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421346+0000", 2024-09-14T04:23:48.656 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2699999999999998e-07 2024-09-14T04:23:48.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.656 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.656 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.656 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421367+0000", 2024-09-14T04:23:48.656 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1509999999999999e-05 2024-09-14T04:23:48.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.657 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.657 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421376+0000", 2024-09-14T04:23:48.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0890000000000006e-06 2024-09-14T04:23:48.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.657 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.657 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421383+0000", 2024-09-14T04:23:48.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3829999999999997e-06 2024-09-14T04:23:48.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421447+0000", 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4517999999999995e-05 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421584+0000", 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000136441 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.658 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421728+0000", 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000144258 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.424854+0000", 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003126438 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.425776+0000", 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00092118199999999999 2024-09-14T04:23:48.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747921+0000", 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32214522000000001 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747961+0000", 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0049000000000002e-05 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747990+0000", 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9666999999999999e-05 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:103428 12.3 12:c8a48380:::benchmark_data_smithi114_253692_object103427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.440286+0000", 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 310.17405178799999, 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30781922699999997, 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.661 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103428 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440286+0000", 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.662 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440286+0000", 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440289+0000", 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5229999999999998e-06 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440334+0000", 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5388e-05 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.663 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.664 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.664 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440348+0000", 2024-09-14T04:23:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:48.665 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:48.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:48.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3825999999999999e-05 2024-09-14T04:23:48.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440354+0000", 2024-09-14T04:23:48.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8479999999999997e-06 2024-09-14T04:23:48.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440403+0000", 2024-09-14T04:23:48.666 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9422999999999998e-05 2024-09-14T04:23:48.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440526+0000", 2024-09-14T04:23:48.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122595 2024-09-14T04:23:48.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:23:48.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440629+0000", 2024-09-14T04:23:48.667 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010343799999999999 2024-09-14T04:23:48.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.443293+0000", 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026637369999999998 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.443766+0000", 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00047331499999999998 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.668 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.748020+0000", 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30425396100000002 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.748073+0000", 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2438000000000002e-05 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.748106+0000", 2024-09-14T04:23:48.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2734999999999997e-05 2024-09-14T04:23:48.672 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.672 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.672 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.672 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:114041 12.c 12:398ecbb4:::benchmark_data_smithi114_253692_object114040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:23:48.672 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:50.692744+0000", 2024-09-14T04:23:48.672 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 297.92159415600003, 2024-09-14T04:23:48.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.17576045100000001, 2024-09-14T04:23:48.672 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 114041 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692744+0000", 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692744+0000", 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692745+0000", 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5899999999999995e-07 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692765+0000", 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9955000000000001e-05 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692773+0000", 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5779999999999999e-06 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692780+0000", 2024-09-14T04:23:48.675 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3169999999999998e-06 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692838+0000", 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7986999999999999e-05 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692955+0000", 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000116982 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.693074+0000", 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000119327 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.697950+0000", 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0048754949999999997 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.700581+0000", 2024-09-14T04:23:48.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002631605 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.868407+0000", 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.167826214 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.868463+0000", 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6079999999999998e-05 2024-09-14T04:23:48.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.868504+0000", 2024-09-14T04:23:48.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1051999999999999e-05 2024-09-14T04:23:48.679 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.679 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.679 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.680 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:31605 13.3 e329/328 13:cc044c86:::benchmark_data_smithi114_288325_object31604:head v 329'1972, mlcod=329'1970)", 2024-09-14T04:23:48.680 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:24.685581+0000", 2024-09-14T04:23:48.680 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 83.928757504999993, 2024-09-14T04:23:48.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.239790843, 2024-09-14T04:23:48.680 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.680 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.680 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.680 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685581+0000", 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685581+0000", 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685582+0000", 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3939999999999999e-06 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685609+0000", 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7186e-05 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685626+0000", 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6368999999999999e-05 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685634+0000", 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6559999999999998e-06 2024-09-14T04:23:48.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685731+0000", 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6457000000000002e-05 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685751+0000", 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0236000000000001e-05 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687942+0000", 2024-09-14T04:23:48.684 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00219118 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925372+0000", 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.237429365 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:31611 13.9 e329/324 13:927299fd:::benchmark_data_smithi114_288325_object31610:head v 329'1946, mlcod=329'1944)", 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:24.687573+0000", 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 83.926765298999996, 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.24159929999999999, 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687573+0000", 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687573+0000", 2024-09-14T04:23:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687574+0000", 2024-09-14T04:23:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.35e-06 2024-09-14T04:23:48.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687668+0000", 2024-09-14T04:23:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3508e-05 2024-09-14T04:23:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687681+0000", 2024-09-14T04:23:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2840000000000001e-05 2024-09-14T04:23:48.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687689+0000", 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0620000000000001e-06 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925334+0000", 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23764569399999999 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925360+0000", 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5554e-05 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.929134+0000", 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003774374 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.929172+0000", 2024-09-14T04:23:48.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7917999999999998e-05 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:31614 13.5 e329/324 13:ad7120f7:::benchmark_data_smithi114_288325_object31613:head v 329'1959, mlcod=329'1957)", 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:24.688736+0000", 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 83.925602401000006, 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.24212169, 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.691 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.688736+0000", 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.688736+0000", 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.688737+0000", 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.85e-07 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.688796+0000", 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9463999999999999e-05 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925276+0000", 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23647989899999999 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925287+0000", 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1161e-05 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925367+0000", 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9553000000000002e-05 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925387+0000", 2024-09-14T04:23:48.694 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0109e-05 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.930803+0000", 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054157509999999999 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.930857+0000", 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4968e-05 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.695 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.696 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:31644 13.3 e329/328 13:ce3478ec:::benchmark_data_smithi114_288325_object31643:head v 329'1973, mlcod=329'1970)", 2024-09-14T04:23:48.696 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:24.702025+0000", 2024-09-14T04:23:48.696 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 83.912312843999999, 2024-09-14T04:23:48.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.22626843099999999, 2024-09-14T04:23:48.696 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.696 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.696 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.702025+0000", 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.702025+0000", 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.702027+0000", 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.511e-06 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.702065+0000", 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8262000000000001e-05 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925266+0000", 2024-09-14T04:23:48.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.223200974 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925284+0000", 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8142999999999999e-05 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925452+0000", 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000167718 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925476+0000", 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3584e-05 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.928209+0000", 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027334019999999998 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.928294+0000", 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4837000000000001e-05 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.700 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:96209 13.0 e341/338 13:08b5302f:::benchmark_data_smithi114_288325_object1144:head v 341'5944, mlcod=0'0)", 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:44.635281+0000", 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 3.9790566800000002, 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017404581999999998, 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.635281+0000", 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.635281+0000", 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.635282+0000", 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9199999999999999e-07 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.635289+0000", 2024-09-14T04:23:48.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5490000000000003e-06 2024-09-14T04:23:48.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.635307+0000", 2024-09-14T04:23:48.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8172000000000002e-05 2024-09-14T04:23:48.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.635315+0000", 2024-09-14T04:23:48.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1659999999999994e-06 2024-09-14T04:23:48.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.638093+0000", 2024-09-14T04:23:48.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027773860000000002 2024-09-14T04:23:48.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.638124+0000", 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0898999999999997e-05 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.652639+0000", 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.01451506 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.652686+0000", 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7358000000000003e-05 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:96442 13.0 e341/338 13:0ff02df8:::benchmark_data_smithi114_288325_object1377:head v 341'5954, mlcod=0'0)", 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:44.966234+0000", 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 3.648103689, 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016641467, 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.706 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.966234+0000", 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.966234+0000", 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.966236+0000", 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.13e-06 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.966245+0000", 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3360000000000001e-06 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.708 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.966261+0000", 2024-09-14T04:23:48.709 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6494000000000002e-05 2024-09-14T04:23:48.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.966269+0000", 2024-09-14T04:23:48.709 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7200000000000006e-06 2024-09-14T04:23:48.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.967533+0000", 2024-09-14T04:23:48.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001263854 2024-09-14T04:23:48.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.967567+0000", 2024-09-14T04:23:48.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3726999999999997e-05 2024-09-14T04:23:48.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.982824+0000", 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015257506000000001 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:44.982876+0000", 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1700000000000003e-05 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.711 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:96599 13.0 e341/338 13:0e2f4470:::benchmark_data_smithi114_288325_object1534:head v 341'5963, mlcod=0'0)", 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:45.195464+0000", 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 3.418874255, 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.020559600000000001, 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.195464+0000", 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.712 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.195464+0000", 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.195465+0000", 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5000000000000002e-07 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.195471+0000", 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0329999999999997e-06 2024-09-14T04:23:48.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.195479+0000", 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6670000000000002e-06 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.195486+0000", 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0419999999999997e-06 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.195541+0000", 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4591e-05 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.195575+0000", 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3862000000000002e-05 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.215926+0000", 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.020351273999999999 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.216024+0000", 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7380999999999995e-05 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:96600 13.0 e341/338 13:068fa558:::benchmark_data_smithi114_288325_object1535:head v 341'5964, mlcod=0'0)", 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:45.198913+0000", 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 3.4154251630000001, 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017272216999999999, 2024-09-14T04:23:48.716 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.717 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.717 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.717 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.198913+0000", 2024-09-14T04:23:48.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.198913+0000", 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.198915+0000", 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0490000000000002e-06 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.198924+0000", 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9339999999999995e-06 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.198939+0000", 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5130000000000001e-05 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.198949+0000", 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5650000000000007e-06 2024-09-14T04:23:48.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.200942+0000", 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001993632 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.200977+0000", 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4897999999999997e-05 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.720 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.216152+0000", 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015174998 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.216185+0000", 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3011e-05 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:96601 13.0 e341/338 13:0a25a092:::benchmark_data_smithi114_288325_object1536:head v 341'5965, mlcod=0'0)", 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:45.202279+0000", 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 3.4120591510000002, 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013936884, 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.202279+0000", 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.722 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.202279+0000", 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.202280+0000", 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7299999999999995e-07 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.202287+0000", 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.34e-06 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.202301+0000", 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4198e-05 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.202309+0000", 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2390000000000002e-06 2024-09-14T04:23:48.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.208740+0000", 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0064301710000000002 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.208773+0000", 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3648e-05 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.216188+0000", 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.007414683 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:45.216216+0000", 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7932e-05 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:97149 13.0 e342/338 13:00044452:::benchmark_data_smithi114_288325_object2084:head v 342'5991, mlcod=0'0)", 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:46.015299+0000", 2024-09-14T04:23:48.726 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 2.599039613, 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.01496899, 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.015299+0000", 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.727 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.015299+0000", 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.015300+0000", 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1739999999999999e-06 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.728 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.015306+0000", 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5699999999999998e-06 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.015319+0000", 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3064999999999999e-05 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.015326+0000", 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4110000000000001e-06 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.019910+0000", 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0045843999999999998 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.019941+0000", 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1081999999999998e-05 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.030228+0000", 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010286399 2024-09-14T04:23:48.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.030268+0000", 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9888999999999997e-05 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:97236 13.0 e342/338 13:076e1087:::benchmark_data_smithi114_288325_object2171:head v 342'6002, mlcod=0'0)", 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:46.138689+0000", 2024-09-14T04:23:48.731 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 2.4756493399999999, 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013895131, 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.138689+0000", 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.732 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.138689+0000", 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.138690+0000", 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.232e-06 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.138699+0000", 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9279999999999999e-06 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.733 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.138724+0000", 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4783e-05 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.138733+0000", 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7760000000000003e-06 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.138801+0000", 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8582000000000002e-05 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.734 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.138842+0000", 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0794000000000003e-05 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.152532+0000", 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013690522 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.152584+0000", 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1514000000000002e-05 2024-09-14T04:23:48.735 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:97767 13.0 e342/338 13:063bbe1b:::benchmark_data_smithi114_288325_object2702:head v 342'6036, mlcod=0'0)", 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:46.912856+0000", 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 1.701481864, 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016139908000000001, 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.736 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.912856+0000", 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.912856+0000", 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.912857+0000", 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3099999999999997e-07 2024-09-14T04:23:48.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.912862+0000", 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9860000000000002e-06 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.912872+0000", 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4900000000000006e-06 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.912877+0000", 2024-09-14T04:23:48.738 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9810000000000003e-06 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.912926+0000", 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9638999999999997e-05 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.912954+0000", 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7769000000000001e-05 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.739 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.928957+0000", 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016002951000000001 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:46.928996+0000", 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9360999999999998e-05 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:23:48.740 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:23:48.741 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:23:48.741 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:48 smithi160 ceph-mon[88407]: pgmap v1316: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 33 MiB/s wr, 571 op/s; 35269/284142 objects degraded (12.412%); 2364/284142 objects misplaced (0.832%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:23:48.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:48.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:23:48.794 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:48.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:48.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:48.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:48.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:48.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:49.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:49.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:49.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:49.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:49.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:49.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:49.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:49.369 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:49.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:49.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:49.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:49.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:49.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:49.636 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:49.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:49.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:49.778 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:49.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:49.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:49.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:49.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:50.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:50.040 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:50.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:50.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:50.165 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:50.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:50.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:50.301 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:50.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:50.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:50 smithi114 ceph-mon[106969]: pgmap v1317: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 591 op/s; 35269/274776 objects degraded (12.836%); 2229/274776 objects misplaced (0.811%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:23:50.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:50 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/274776 objects degraded (12.836%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:23:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:50 smithi114 ceph-mon[103029]: pgmap v1317: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 591 op/s; 35269/274776 objects degraded (12.836%); 2229/274776 objects misplaced (0.811%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:23:50.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:50 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/274776 objects degraded (12.836%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:23:50.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:50.426 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:50.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:50.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:50.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:50.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:50.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:50.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-14T04:23:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:50 smithi160 ceph-mon[88407]: pgmap v1317: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 591 op/s; 35269/274776 objects degraded (12.836%); 2229/274776 objects misplaced (0.811%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:23:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:50 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/274776 objects degraded (12.836%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:23:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:23:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:23:50.216Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:23:50.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:50.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:50.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:50.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:50.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:50.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:50.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:50.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:50.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:50.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:51.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:51.104 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:51.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:51.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:51.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:51.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:51.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:51.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:51.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:51.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:51.499 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:51.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:51.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:51.631 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:51.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:51.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:51.756 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:51.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:51.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:51.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:23:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:23:51] "GET /metrics HTTP/1.1" 200 36608 "" "Prometheus/2.51.0" 2024-09-14T04:23:51.910 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:51.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:51.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:52.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:52.037 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:52.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:52.139 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:52.177 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:52.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:52.241 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:52 smithi114 ceph-mon[106969]: pgmap v1318: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 14 MiB/s wr, 593 op/s; 35269/274065 objects degraded (12.869%); 2229/274065 objects misplaced (0.813%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:23:52.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:52 smithi114 ceph-mon[103029]: pgmap v1318: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 14 MiB/s wr, 593 op/s; 35269/274065 objects degraded (12.869%); 2229/274065 objects misplaced (0.813%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:23:52.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:52.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:52.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:52.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:52.447 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:52.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:52.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:52.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:52.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:52.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:52 smithi160 ceph-mon[88407]: pgmap v1318: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 14 MiB/s wr, 593 op/s; 35269/274065 objects degraded (12.869%); 2229/274065 objects misplaced (0.813%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:23:52.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:52.727 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:52.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:52.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:52.854 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:52.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:52.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:53.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:53.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:53.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-14T04:23:53.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:53.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:53.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:53.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:53.272 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:53.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:53.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:53.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:53.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:53.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:53.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:53.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:53.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:53.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:53.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:23:53.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:53.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:53.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:53.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:53.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:53.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:53.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:53.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:54.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:54.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:54.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:54.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:54.176 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:54.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:54.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:54.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:54.250 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:54.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:54.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:54.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:54.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:54.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:54.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:54.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:54 smithi114 ceph-mon[106969]: pgmap v1319: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 11 MiB/s wr, 595 op/s; 35269/272889 objects degraded (12.924%); 2229/272889 objects misplaced (0.817%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:23:54.524 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:54.525 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:54 smithi114 ceph-mon[103029]: pgmap v1319: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 11 MiB/s wr, 595 op/s; 35269/272889 objects degraded (12.924%); 2229/272889 objects misplaced (0.817%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:23:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:54.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:54.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:54.615 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:54.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:54 smithi160 ceph-mon[88407]: pgmap v1319: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 11 MiB/s wr, 595 op/s; 35269/272889 objects degraded (12.924%); 2229/272889 objects misplaced (0.817%); 2.3 MiB/s, 37 objects/s recovering 2024-09-14T04:23:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:54.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:54.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:54.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:54.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:54.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:54.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:54.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:55.008 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:55.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:55.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:55.133 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:55.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:55.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:55.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:55.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:55.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:55.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:55.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:55.466 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:23:55.466 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:23:55.466 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:55.467 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:55 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/272889 objects degraded (12.924%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:23:55.467 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:23:55.467 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:23:55.467 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:55.467 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:55 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/272889 objects degraded (12.924%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:23:55.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:55.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:55.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:55.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:55.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:23:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:23:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:23:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:55 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/272889 objects degraded (12.924%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:23:55.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:55.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:55.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:55.813 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:55.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:55.915 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:55.961 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:55.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:56.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:56.095 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:56.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:56.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:56.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:56.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:56.321 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:23:56.321 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:23:56.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:56.344 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:23:56.344 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:23:56.365 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:23:56.366 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:23:56.379 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:56.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:56.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:56.385 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:23:56.385 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:23:56.419 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:23:56.419 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:23:56.450 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:23:56.451 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:23:56.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:56 smithi114 ceph-mon[103029]: pgmap v1320: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 102 B/s wr, 674 op/s; 35269/263910 objects degraded (13.364%); 2094/263910 objects misplaced (0.793%); 1.7 MiB/s, 26 objects/s recovering 2024-09-14T04:23:56.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:56 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/83253033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:56.452 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:56 smithi114 ceph-mon[106969]: pgmap v1320: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 102 B/s wr, 674 op/s; 35269/263910 objects degraded (13.364%); 2094/263910 objects misplaced (0.793%); 1.7 MiB/s, 26 objects/s recovering 2024-09-14T04:23:56.452 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:56 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/83253033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:56.478 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:23:56.478 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:23:56.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:56.508 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:23:56.508 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:23:56.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:56.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:56.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:56.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:56.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:56.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:56.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:56.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:56.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:56.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:56.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:56 smithi160 ceph-mon[88407]: pgmap v1320: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 102 B/s wr, 674 op/s; 35269/263910 objects degraded (13.364%); 2094/263910 objects misplaced (0.793%); 1.7 MiB/s, 26 objects/s recovering 2024-09-14T04:23:56.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:56 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/83253033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:23:56.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:56.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:57.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:57.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:57.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:57.163 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:57.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:57.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:57.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:57.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:57.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:57.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:57.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:57.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:57.553 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:57.576 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:57.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:57.678 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:57.718 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:57.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:57.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:23:57.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:57.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:23:57.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:57.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:57.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:57.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:58.076 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:23:58.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:58.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:58.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:58.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:58.282 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:58.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:58.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:23:58.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:58.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:23:58.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:23:58 smithi114 ceph-mon[106969]: pgmap v1321: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 85 B/s rd, 85 B/s wr, 565 op/s; 35269/263910 objects degraded (13.364%); 2094/263910 objects misplaced (0.793%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:58.497 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:23:58 smithi114 ceph-mon[103029]: pgmap v1321: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 85 B/s rd, 85 B/s wr, 565 op/s; 35269/263910 objects degraded (13.364%); 2094/263910 objects misplaced (0.793%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:58.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:58.564 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:58.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:58.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:23:58.690 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:58.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:23:58.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:58.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:58.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:58.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:23:58 smithi160 ceph-mon[88407]: pgmap v1321: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 85 B/s rd, 85 B/s wr, 565 op/s; 35269/263910 objects degraded (13.364%); 2094/263910 objects misplaced (0.793%); 1.4 MiB/s, 22 objects/s recovering 2024-09-14T04:23:58.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:58.975 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:58.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:58.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:23:59.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:23:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:59.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:59.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:23:59.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:59.237 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:59.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:59.339 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:23:59.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:59.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:23:59.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:23:59.529 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:59.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:23:59.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:59.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:59.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:59.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:59.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:23:59.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:23:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:59.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:23:59.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:23:59.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:23:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:23:59.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:00.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:00.059 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:00.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:00.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:00.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:00.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:00.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:00.291 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:00.320 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:00.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:00.392 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:24:00.392 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:24:00.422 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:24:00.449 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:00.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:00.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:24:00.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:00.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:00 smithi114 ceph-mon[103029]: pgmap v1322: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 85 B/s rd, 85 B/s wr, 821 op/s; 35269/254580 objects degraded (13.854%); 1950/254580 objects misplaced (0.766%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:24:00.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:00 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/254580 objects degraded (13.854%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:00.560 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3470684389' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:00.561 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:00 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:00.561 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:00 smithi114 ceph-mon[106969]: pgmap v1322: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 85 B/s rd, 85 B/s wr, 821 op/s; 35269/254580 objects degraded (13.854%); 1950/254580 objects misplaced (0.766%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:24:00.561 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:00 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/254580 objects degraded (13.854%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:00.561 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3470684389' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:00.561 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:00 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:00.604 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:00.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:24:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:24:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:24:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:00 smithi160 ceph-mon[88407]: pgmap v1322: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 85 B/s rd, 85 B/s wr, 821 op/s; 35269/254580 objects degraded (13.854%); 1950/254580 objects misplaced (0.766%); 2.2 MiB/s, 34 objects/s recovering 2024-09-14T04:24:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:00 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/254580 objects degraded (13.854%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3470684389' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:00 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:00.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:00.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:00.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:00.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:00.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:00.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:01.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:01.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:01.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:01.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:01.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:01.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:01.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:01.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:01.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:01.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:01.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:01.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:01.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:01.578 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:01.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:01.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:01.720 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:01.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:01.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:01.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:01.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:01.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:01.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:24:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:24:01] "GET /metrics HTTP/1.1" 200 36613 "" "Prometheus/2.51.0" 2024-09-14T04:24:01.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:01 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:24:01.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:01 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:24:01.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:01 smithi114 ceph-mon[103029]: osdmap e343: 8 total, 8 up, 7 in 2024-09-14T04:24:01.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:01 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:24:01.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:01 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:24:01.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:01 smithi114 ceph-mon[106969]: osdmap e343: 8 total, 8 up, 7 in 2024-09-14T04:24:01.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:01 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:24:01.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:01 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:24:01.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:01 smithi160 ceph-mon[88407]: osdmap e343: 8 total, 8 up, 7 in 2024-09-14T04:24:01.949 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:01.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:01.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:02.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:02.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:02.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:02.198 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-14T04:24:02.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:02.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:02.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:02.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:02.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:02.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:02.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:02.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:02.523 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:02.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:02.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:02.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:02.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:02.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:02.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:02.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:02.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:02.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:02 smithi114 ceph-mon[103029]: pgmap v1324: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 670 op/s; 35269/253932 objects degraded (13.889%); 1950/253932 objects misplaced (0.768%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T04:24:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:02 smithi114 ceph-mon[106969]: pgmap v1324: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 670 op/s; 35269/253932 objects degraded (13.889%); 1950/253932 objects misplaced (0.768%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T04:24:02.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:02.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:02.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:02 smithi160 ceph-mon[88407]: pgmap v1324: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 670 op/s; 35269/253932 objects degraded (13.889%); 1950/253932 objects misplaced (0.768%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T04:24:03.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:03.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:03.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:03.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:03.175 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:03.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:03.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:03.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:03.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:03.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:03.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:03.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:03.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:03.561 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:03.604 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:03.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:03.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:03.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:03.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:03.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:03.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:03.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:03.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:04.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:04.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:04.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:04.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:04.156 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:04.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:04.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:04.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:04.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:04.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:04.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:04.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:04.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:04.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:04.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:04.693 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:04.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:04.731 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:04.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:04.757 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:04 smithi114 ceph-mon[103029]: pgmap v1325: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 671 op/s; 35269/252750 objects degraded (13.954%); 1950/252750 objects misplaced (0.772%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T04:24:04.757 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:04 smithi114 ceph-mon[106969]: pgmap v1325: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 671 op/s; 35269/252750 objects degraded (13.954%); 1950/252750 objects misplaced (0.772%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T04:24:04.765 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:04.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:04.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:04.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:04 smithi160 ceph-mon[88407]: pgmap v1325: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 671 op/s; 35269/252750 objects degraded (13.954%); 1950/252750 objects misplaced (0.772%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T04:24:04.958 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:05.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:05.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:05.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:05.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:05.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:05.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:05.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:05.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:05.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:05.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:05.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:05.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:05.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:05.594 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:05 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/252750 objects degraded (13.954%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:05.594 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:05 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/252750 objects degraded (13.954%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:05.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:05.650 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:05.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:05.752 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:05.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:05.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:05.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:05.901 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:05 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/252750 objects degraded (13.954%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:05.952 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:05.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:06.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:06.080 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:06.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:06.128 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:06.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:06.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:06.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:06.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:06.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:06.350 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:06.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:06.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:06.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:06.476 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:06.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:06.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:06.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:06.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:06.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:06 smithi114 ceph-mon[103029]: pgmap v1326: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 35269/244530 objects degraded (14.423%); 1812/244530 objects misplaced (0.741%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:24:06.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:06 smithi114 ceph-mon[106969]: pgmap v1326: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 35269/244530 objects degraded (14.423%); 1812/244530 objects misplaced (0.741%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:24:06.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:06.759 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:06.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:06.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:06.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:06.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:06 smithi160 ceph-mon[88407]: pgmap v1326: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 35269/244530 objects degraded (14.423%); 1812/244530 objects misplaced (0.741%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:24:06.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:07.029 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:07.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:07.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:07.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:07.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:07.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:07.287 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:07.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:07.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:24:07.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:07.430 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:07.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:07.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:07.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:07.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:07.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:07.719 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:07.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:07.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:07.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:07.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:07.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:07.992 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:07.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:08.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:08.117 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:08.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:08.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:08.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:08.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:08.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:08.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:08.394 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:08.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:08.462 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:24:08.462 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:24:08.482 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:24:08.482 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:24:08.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:08.502 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:24:08.502 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:24:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:08.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:08.525 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:24:08.525 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:24:08.552 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:24:08.552 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:24:08.581 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:24:08.582 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:24:08.614 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:24:08.614 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:24:08.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:08.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:08.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:08.651 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:24:08.652 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:24:08.751 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:08.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:08.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:08 smithi114 ceph-mon[106969]: pgmap v1327: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 35269/244530 objects degraded (14.423%); 1812/244530 objects misplaced (0.741%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:24:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:24:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/790244292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:08 smithi114 ceph-mon[103029]: pgmap v1327: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 35269/244530 objects degraded (14.423%); 1812/244530 objects misplaced (0.741%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:24:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:24:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/790244292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:08.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:08.929 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:08.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:09.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:09.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:09.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:09.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:09.158 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:09.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:08 smithi160 ceph-mon[88407]: pgmap v1327: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 35269/244530 objects degraded (14.423%); 1812/244530 objects misplaced (0.741%); 1.8 MiB/s, 28 objects/s recovering 2024-09-14T04:24:09.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:24:09.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:09.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/790244292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:09.198 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:09.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:09.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:09.323 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:09.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:09.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:09.449 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:09.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:09.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:09.577 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:09.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:09.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:09.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-14T04:24:09.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:09.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:09.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:09.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:09.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:09.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:10.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:10.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:10.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:10.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/244530 objects degraded (14.423%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:10.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/244530 objects degraded (14.423%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:10.144 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:10.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/244530 objects degraded (14.423%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:10.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:10.286 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:10.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:10.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:10.413 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:10.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:10.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:10.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:10.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:10.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:24:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:24:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:24:10.700 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:10.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:10.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:10.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:10.825 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:10.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:10.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:10 smithi114 ceph-mon[103029]: pgmap v1328: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 35269/235791 objects degraded (14.958%); 1653/235791 objects misplaced (0.701%); 1.9 MiB/s, 29 objects/s recovering 2024-09-14T04:24:10.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:10 smithi114 ceph-mon[106969]: pgmap v1328: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 35269/235791 objects degraded (14.958%); 1653/235791 objects misplaced (0.701%); 1.9 MiB/s, 29 objects/s recovering 2024-09-14T04:24:10.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:10.966 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:10.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:11.035 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:11.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:11.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:11.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:10 smithi160 ceph-mon[88407]: pgmap v1328: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 35269/235791 objects degraded (14.958%); 1653/235791 objects misplaced (0.701%); 1.9 MiB/s, 29 objects/s recovering 2024-09-14T04:24:11.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:11.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:11.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:11.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:11.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:11.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:11.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:11.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:11.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:11.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:11.571 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:24:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:24:11] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-09-14T04:24:11.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:11.624 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:11.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:11.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:11.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:11.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:11.828 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:24:11.829 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:24:11.829 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:24:11.829 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:24:11.829 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:24:11.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:11.879 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:11.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:11.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:12.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:12.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:12.036 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:12.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:12.146 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:12.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:12.248 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:12.293 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:12.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:12.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:12.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:12.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:12.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:12.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:12.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:12.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:12.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:12.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-14T04:24:12.706 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:12.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:12.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:12.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:12.835 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:12.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:12.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:12.955 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:12 smithi114 ceph-mon[103029]: pgmap v1329: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 584 op/s; 35269/235215 objects degraded (14.994%); 1653/235215 objects misplaced (0.703%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T04:24:12.955 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:12 smithi114 ceph-mon[106969]: pgmap v1329: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 584 op/s; 35269/235215 objects degraded (14.994%); 1653/235215 objects misplaced (0.703%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T04:24:12.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:12.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:12.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:24:13.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:12 smithi160 ceph-mon[88407]: pgmap v1329: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 584 op/s; 35269/235215 objects degraded (14.994%); 1653/235215 objects misplaced (0.703%); 1.7 MiB/s, 27 objects/s recovering 2024-09-14T04:24:13.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:13.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:13.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:13.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:13.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:13.265 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:13.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:13.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:13.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:13.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:13.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:13.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:13.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:13.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:13.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:13.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:13.808 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:13.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:13.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:13.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:13.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:14.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:14.057 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:14.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:14.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:14.203 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:14.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:14.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:14.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:14.351 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:14.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:14.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:14.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:14.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:14.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:14.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:14.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:14.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:14.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:14.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:14.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:14.905 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:14.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:15.006 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:15.031 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:15.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:15.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:14 smithi114 ceph-mon[106969]: pgmap v1330: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.7 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 602 op/s; 35269/232251 objects degraded (15.186%); 1653/232251 objects misplaced (0.712%); 1.5 MiB/s, 24 objects/s recovering 2024-09-14T04:24:15.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/232251 objects degraded (15.186%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:15.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:14 smithi114 ceph-mon[103029]: pgmap v1330: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.7 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 602 op/s; 35269/232251 objects degraded (15.186%); 1653/232251 objects misplaced (0.712%); 1.5 MiB/s, 24 objects/s recovering 2024-09-14T04:24:15.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/232251 objects degraded (15.186%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:15.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:15.156 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:15.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:14 smithi160 ceph-mon[88407]: pgmap v1330: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.7 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 602 op/s; 35269/232251 objects degraded (15.186%); 1653/232251 objects misplaced (0.712%); 1.5 MiB/s, 24 objects/s recovering 2024-09-14T04:24:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/232251 objects degraded (15.186%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:15.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:15.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:15.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:15.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:15.444 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:15.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:15.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:15.568 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:15.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:15.670 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:15.706 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:15 smithi114 ceph-mon[103029]: pgmap v1331: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 35269/225669 objects degraded (15.629%); 1489/225669 objects misplaced (0.660%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:24:15.707 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:15 smithi114 ceph-mon[106969]: pgmap v1331: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 35269/225669 objects degraded (15.629%); 1489/225669 objects misplaced (0.660%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:24:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:15.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:15.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:15.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:15.875 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:15.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:15.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:15.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:16.024 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:16.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:16.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:15 smithi160 ceph-mon[88407]: pgmap v1331: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 35269/225669 objects degraded (15.629%); 1489/225669 objects misplaced (0.660%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:24:16.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:16.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:16.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:16.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:16.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:16.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:16.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:16.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:16.585 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:16.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:16.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:16.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:16.701 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:24:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:16.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:16.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:16.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:16.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:16.944 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:16 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/467883783' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:24:16.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:16 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/467883783' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:24:16.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:16.973 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:16.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:17.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:17.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:17.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:17.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:16 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/467883783' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:24:17.226 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:17.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:17.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:17.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:17.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:17.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:17.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:17.524 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:17.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:17.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:17.672 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:17.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:17.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:17.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:17.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:17.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:17.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:17.956 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:17.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:17.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:17 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:24:17.978 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:17 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/467883783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:24:17.979 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:17 smithi114 ceph-mon[103029]: osdmap e344: 8 total, 8 up, 7 in 2024-09-14T04:24:17.979 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:17 smithi114 ceph-mon[103029]: pgmap v1333: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 628 op/s; 35269/225669 objects degraded (15.629%); 1489/225669 objects misplaced (0.660%); 2.0 MiB/s, 32 objects/s recovering 2024-09-14T04:24:17.979 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:17 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:24:17.979 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:17 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/467883783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:24:17.979 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:17 smithi114 ceph-mon[106969]: osdmap e344: 8 total, 8 up, 7 in 2024-09-14T04:24:17.979 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:17 smithi114 ceph-mon[106969]: pgmap v1333: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 628 op/s; 35269/225669 objects degraded (15.629%); 1489/225669 objects misplaced (0.660%); 2.0 MiB/s, 32 objects/s recovering 2024-09-14T04:24:18.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:18.097 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:18.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:17 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:24:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:17 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/467883783' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:24:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:17 smithi160 ceph-mon[88407]: osdmap e344: 8 total, 8 up, 7 in 2024-09-14T04:24:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:17 smithi160 ceph-mon[88407]: pgmap v1333: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 628 op/s; 35269/225669 objects degraded (15.629%); 1489/225669 objects misplaced (0.660%); 2.0 MiB/s, 32 objects/s recovering 2024-09-14T04:24:18.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:18.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:18.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:18.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:18.363 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:18.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:18.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:18.487 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:18.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:18.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:18.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:18.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:18.715 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:18.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:18.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:18.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:18.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:18.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:18.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:19.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:19.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:19.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:19.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:19.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:19.249 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:19.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:19.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:19.374 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:19.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:19.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:19.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:19.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:19.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:19.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:19.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:19.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:19.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:19.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:19.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:19.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:24:19.819 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:19.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:19.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-14T04:24:19.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:19.963 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:19.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:20.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:20.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:20.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:20.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:20.230 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:20.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:20.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:20.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:20.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:20.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:20.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:20.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:20.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:20 smithi114 ceph-mon[103029]: pgmap v1334: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 35269/216855 objects degraded (16.264%); 1334/216855 objects misplaced (0.615%); 2.0 MiB/s, 31 objects/s recovering 2024-09-14T04:24:20.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:20 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 35269/216855 objects degraded (16.264%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:20.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:20 smithi114 ceph-mon[106969]: pgmap v1334: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 35269/216855 objects degraded (16.264%); 1334/216855 objects misplaced (0.615%); 2.0 MiB/s, 31 objects/s recovering 2024-09-14T04:24:20.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:20 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 35269/216855 objects degraded (16.264%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:20.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:24:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:24:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:24:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:20 smithi160 ceph-mon[88407]: pgmap v1334: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 35269/216855 objects degraded (16.264%); 1334/216855 objects misplaced (0.615%); 2.0 MiB/s, 31 objects/s recovering 2024-09-14T04:24:20.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:20 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 35269/216855 objects degraded (16.264%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:20.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:20.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:20.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:20.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:20.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:20.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:20.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:20.931 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:24:20.931 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:24:20.931 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:24:20.931 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:24:20.931 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:127243 11.f 11:fbb4aa71:::benchmark_data_smithi114_223345_object15306:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:29.573066+0000", 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 591.35463153199998, 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0061637379999999, 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 127243 2024-09-14T04:24:20.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573066+0000", 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573066+0000", 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573068+0000", 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0770000000000002e-06 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573074+0000", 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9479999999999996e-06 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573088+0000", 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4579000000000001e-05 2024-09-14T04:24:20.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573096+0000", 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5950000000000001e-06 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573160+0000", 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3986999999999996e-05 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:20.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.573169+0000", 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9150000000000002e-06 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.553115+0000", 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97994636099999999 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.555491+0000", 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023760389999999999 2024-09-14T04:24:20.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.555765+0000", 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00027377800000000002 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.567674+0000", 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011908372 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.937 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.568212+0000", 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00053813199999999998 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.579134+0000", 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010922189000000001 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.579179+0000", 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5142000000000002e-05 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.579230+0000", 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0624000000000002e-05 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:127260 11.f 11:f98b95da:::benchmark_data_smithi114_223345_object15323:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:14:29.606687+0000", 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 591.32101003399998, 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97418977699999998, 2024-09-14T04:24:20.939 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 127260 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606687+0000", 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606687+0000", 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606689+0000", 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8199999999999999e-06 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:20.941 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606694+0000", 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3950000000000003e-06 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606707+0000", 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3210999999999999e-05 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606714+0000", 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2649999999999999e-06 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606770+0000", 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5887999999999998e-05 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:29.606787+0000", 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7133999999999999e-05 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.568243+0000", 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96145623899999999 2024-09-14T04:24:20.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.568340+0000", 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6434000000000005e-05 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.568452+0000", 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000112323 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:20.944 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.574771+0000", 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0063187019999999998 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.577844+0000", 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00307355 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.579275+0000", 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0014305190000000001 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.945 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.946 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:20.946 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.580790+0000", 2024-09-14T04:24:20.946 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015148749999999999 2024-09-14T04:24:20.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.946 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.946 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:20.946 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:14:30.580877+0000", 2024-09-14T04:24:20.946 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7422000000000004e-05 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206303 11.c 11:3418ab87:::benchmark_data_smithi114_223345_object94366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.960541+0000", 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 502.967156123, 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.33908080299999999, 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:20.947 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206303 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960541+0000", 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:20.948 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960541+0000", 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960543+0000", 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8509999999999999e-06 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960548+0000", 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6739999999999996e-06 2024-09-14T04:24:20.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960563+0000", 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5281000000000002e-05 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.960571+0000", 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1009999999999992e-06 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.965441+0000", 2024-09-14T04:24:20.950 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0048702119999999996 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.965585+0000", 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000143156 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.965712+0000", 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012754899999999999 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.972223+0000", 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.006511317 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.972368+0000", 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014410399999999999 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.297537+0000", 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32516935299999999 2024-09-14T04:24:20.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.297660+0000", 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122853 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.299622+0000", 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019623520000000001 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206312 11.f 11:f5f3464b:::benchmark_data_smithi114_223345_object94375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.973030+0000", 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 502.95466781599998, 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32513937399999998, 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206312 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.954 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973030+0000", 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973030+0000", 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:20.955 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973031+0000", 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1069999999999999e-06 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973035+0000", 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8299999999999998e-06 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973044+0000", 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.217e-06 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.956 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973049+0000", 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1309999999999997e-06 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973106+0000", 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7473999999999999e-05 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973236+0000", 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013012000000000001 2024-09-14T04:24:20.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973368+0000", 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000131524 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977449+0000", 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0040805299999999997 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:20.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977463+0000", 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4245e-05 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.297832+0000", 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32036935300000002 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.298033+0000", 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020055 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.298169+0000", 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000136293 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206314 11.c 11:34914179:::benchmark_data_smithi114_223345_object94377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.973883+0000", 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 502.953814647, 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32750190400000001, 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:20.960 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:20.961 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:20.961 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:20.961 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:24:20.961 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:24:20.962 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206314 2024-09-14T04:24:20.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.962 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:20.962 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.962 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:20.962 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973883+0000", 2024-09-14T04:24:20.962 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.962 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973883+0000", 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973884+0000", 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8400000000000003e-07 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973888+0000", 2024-09-14T04:24:20.963 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1250000000000003e-06 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973902+0000", 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4521e-05 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973909+0000", 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1110000000000002e-06 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.973979+0000", 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9716999999999997e-05 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.974121+0000", 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014170099999999999 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.974257+0000", 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013602300000000001 2024-09-14T04:24:20.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.980526+0000", 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062688589999999999 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.981607+0000", 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010813590000000001 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.299736+0000", 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31812907899999998 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.299790+0000", 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4342999999999998e-05 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.301385+0000", 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001594182 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.967 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206317 11.c 11:3f6a083c:::benchmark_data_smithi114_223345_object94380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.977365+0000", 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 502.95033196100002, 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32865713099999999, 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:24:20.968 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206317 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977365+0000", 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977365+0000", 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977367+0000", 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.33e-06 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977371+0000", 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5730000000000001e-06 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977380+0000", 2024-09-14T04:24:20.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1799999999999996e-06 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977384+0000", 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6349999999999997e-06 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977443+0000", 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8427999999999997e-05 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977552+0000", 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000109881 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,4", 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.977628+0000", 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5611000000000005e-05 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.980534+0000", 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029058560000000001 2024-09-14T04:24:20.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.987282+0000", 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0067482999999999996 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.304204+0000", 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.31692202899999999 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.304393+0000", 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000188238 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.306023+0000", 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00163007 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.36273.0:206323 11.f 11:f3df4957:::benchmark_data_smithi114_223345_object94386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:15:57.991379+0000", 2024-09-14T04:24:20.974 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 502.936318454, 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30869569899999999, 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.36273", 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1849518322", 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 206323 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:20.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991379+0000", 2024-09-14T04:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991379+0000", 2024-09-14T04:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:20.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:20.978 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991380+0000", 2024-09-14T04:24:20.978 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.438e-06 2024-09-14T04:24:20.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.978 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.978 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991385+0000", 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7720000000000002e-06 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991397+0000", 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1904e-05 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991403+0000", 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6389999999999997e-06 2024-09-14T04:24:20.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991458+0000", 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5751000000000002e-05 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991588+0000", 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000129682 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,7", 2024-09-14T04:24:20.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.991722+0000", 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000133776 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.994513+0000", 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027913439999999999 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:57.994555+0000", 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1752000000000003e-05 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.299991+0000", 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.305436229 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.300038+0000", 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6699000000000001e-05 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:15:58.300075+0000", 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6712999999999999e-05 2024-09-14T04:24:20.982 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:74318 13.0 13:0cc60ffe:::benchmark_data_smithi114_288325_object74317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:08.291017+0000", 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 72.636680143999996, 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4396549950000002, 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:20.983 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74318 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.291017+0000", 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.291017+0000", 2024-09-14T04:24:20.984 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.291019+0000", 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.968e-06 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.291047+0000", 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7750999999999999e-05 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.985 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.291060+0000", 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2559e-05 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.291065+0000", 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8189999999999997e-06 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.291127+0000", 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1872999999999995e-05 2024-09-14T04:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.291140+0000", 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2320000000000001e-05 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.256696+0000", 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96555636499999997 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:20.987 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.256709+0000", 2024-09-14T04:24:20.988 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3336000000000001e-05 2024-09-14T04:24:20.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.718349+0000", 2024-09-14T04:24:20.988 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4616392519999999 2024-09-14T04:24:20.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:20.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.718488+0000", 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013939700000000001 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.718601+0000", 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113447 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.727031+0000", 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0084294829999999998 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.989 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.727042+0000", 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0903e-05 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.727084+0000", 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1924e-05 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.727156+0000", 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2385000000000006e-05 2024-09-14T04:24:20.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.730672+0000", 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035162129999999998 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:74335 13.0 13:08841a74:::benchmark_data_smithi114_288325_object74334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:08.311666+0000", 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 72.616031848000006, 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4283968109999998, 2024-09-14T04:24:20.991 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74335 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.311666+0000", 2024-09-14T04:24:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.311666+0000", 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.311668+0000", 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0789999999999999e-06 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.993 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.311694+0000", 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6044e-05 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.311708+0000", 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4198e-05 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.311714+0000", 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.562e-06 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.311780+0000", 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5574000000000006e-05 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.311793+0000", 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3216e-05 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257001+0000", 2024-09-14T04:24:20.995 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94520790399999999 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257010+0000", 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2900000000000008e-06 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.724428+0000", 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4674173509999999 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.996 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.724612+0000", 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018438300000000001 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.724828+0000", 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021556799999999999 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.733290+0000", 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0084627680000000007 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.997 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.733354+0000", 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3614000000000005e-05 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.735665+0000", 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023107539999999999 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.735728+0000", 2024-09-14T04:24:20.998 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3088000000000003e-05 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.740062+0000", 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043344179999999996 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:74338 13.0 13:0a82b096:::benchmark_data_smithi114_288325_object74337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:08.315278+0000", 2024-09-14T04:24:20.999 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 72.612419514999999, 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4313198539999998, 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74338 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.315278+0000", 2024-09-14T04:24:21.000 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.315278+0000", 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.315279+0000", 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.096e-06 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.315300+0000", 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0934999999999999e-05 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.315310+0000", 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0175e-05 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.315316+0000", 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6400000000000002e-06 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.315385+0000", 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8997000000000004e-05 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.315397+0000", 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2515e-05 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257042+0000", 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94164518699999999 2024-09-14T04:24:21.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257050+0000", 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9349999999999994e-06 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.730734+0000", 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4736839179999999 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.004 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.730918+0000", 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018338499999999999 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.731131+0000", 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021309599999999999 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.742920+0000", 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011789035 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.005 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.742937+0000", 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7575e-05 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.742983+0000", 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6065e-05 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.743024+0000", 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0408999999999999e-05 2024-09-14T04:24:21.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.746598+0000", 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003573891 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:74372 13.0 13:0c66ae69:::benchmark_data_smithi114_288325_object74371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:08.363018+0000", 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 72.564679476999999, 2024-09-14T04:24:21.007 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.386752263, 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74372 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.008 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363018+0000", 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363018+0000", 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363020+0000", 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.72e-06 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363080+0000", 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0516000000000001e-05 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363096+0000", 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5591999999999999e-05 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363105+0000", 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5990000000000008e-06 2024-09-14T04:24:21.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363162+0000", 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6239999999999997e-05 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363170+0000", 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2500000000000006e-06 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257078+0000", 2024-09-14T04:24:21.011 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.89390779600000003 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257085+0000", 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3699999999999997e-06 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.733393+0000", 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4763082759999999 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.012 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.733516+0000", 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012232599999999999 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.733634+0000", 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000118878 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.742946+0000", 2024-09-14T04:24:21.013 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0093110129999999999 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.746660+0000", 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003714387 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.746764+0000", 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010421600000000001 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.014 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.746803+0000", 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8420999999999997e-05 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.749770+0000", 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029676630000000002 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.015 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:74373 13.0 13:073eb4ff:::benchmark_data_smithi114_288325_object74372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:08.363126+0000", 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 72.564571301000001, 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3893445639999999, 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74373 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363126+0000", 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363126+0000", 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363127+0000", 2024-09-14T04:24:21.017 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.44e-07 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363176+0000", 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9633999999999999e-05 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363186+0000", 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6339999999999996e-06 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.018 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363193+0000", 2024-09-14T04:24:21.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6760000000000001e-06 2024-09-14T04:24:21.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.019 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363223+0000", 2024-09-14T04:24:21.019 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0069e-05 2024-09-14T04:24:21.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.019 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.019 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.020 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.363227+0000", 2024-09-14T04:24:21.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9709999999999998e-06 2024-09-14T04:24:21.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.020 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.020 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.020 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257111+0000", 2024-09-14T04:24:21.020 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.893884656 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257118+0000", 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8630000000000002e-06 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.740115+0000", 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4829970770000001 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.021 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.740245+0000", 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012974800000000001 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.740374+0000", 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012950799999999999 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.746671+0000", 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062969189999999998 2024-09-14T04:24:21.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.746724+0000", 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3016999999999997e-05 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.749843+0000", 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031184070000000001 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.749906+0000", 2024-09-14T04:24:21.023 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3256999999999994e-05 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.752471+0000", 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002564584 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:74376 13.0 13:09a035e9:::benchmark_data_smithi114_288325_object74375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-14T04:24:21.024 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:08.368095+0000", 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 72.559602259000002, 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4096478330000002, 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74376 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.368095+0000", 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.368095+0000", 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.368096+0000", 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0380000000000001e-06 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.026 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.368137+0000", 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0392999999999998e-05 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.368148+0000", 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1328999999999999e-05 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.368153+0000", 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4269999999999999e-06 2024-09-14T04:24:21.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.368199+0000", 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5191999999999997e-05 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.368206+0000", 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3780000000000003e-06 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.028 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257144+0000", 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.88893850799999996 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257150+0000", 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.037e-06 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.752512+0000", 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4953620400000001 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.029 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.752616+0000", 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103986 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.752768+0000", 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015109499999999999 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.758988+0000", 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062205469999999999 2024-09-14T04:24:21.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.768566+0000", 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0095778340000000003 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.774129+0000", 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0055629019999999998 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.774200+0000", 2024-09-14T04:24:21.031 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1353000000000006e-05 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.777743+0000", 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035427739999999998 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:74387 13.0 13:0a794a59:::benchmark_data_smithi114_288325_object74386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:08.395705+0000", 2024-09-14T04:24:21.032 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 72.531992599000006, 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3855051290000002, 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74387 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.395705+0000", 2024-09-14T04:24:21.033 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.395705+0000", 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.395706+0000", 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3519999999999999e-06 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.034 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.395758+0000", 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1857999999999997e-05 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.395772+0000", 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3750999999999999e-05 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.395779+0000", 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2529999999999998e-06 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.035 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.395843+0000", 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4449000000000006e-05 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.395851+0000", 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8280000000000003e-06 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257172+0000", 2024-09-14T04:24:21.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.86132114100000001 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257178+0000", 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7830000000000004e-06 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.754854+0000", 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4976760069999999 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.754972+0000", 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011777699999999999 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.755086+0000", 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011356100000000001 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.758998+0000", 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039121759999999998 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.768645+0000", 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0096476600000000006 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.777824+0000", 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0091781320000000003 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.777877+0000", 2024-09-14T04:24:21.039 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3220000000000002e-05 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.781210+0000", 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033331810000000002 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:74408 13.0 13:05a5ac74:::benchmark_data_smithi114_288325_object74407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:08.447709+0000", 2024-09-14T04:24:21.040 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 72.479988859000002, 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.33688326, 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74408 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.447709+0000", 2024-09-14T04:24:21.041 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.447709+0000", 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.447709+0000", 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8400000000000004e-07 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.447733+0000", 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3479999999999999e-05 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.447741+0000", 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7290000000000006e-06 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.447746+0000", 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5199999999999999e-06 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.447799+0000", 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2803000000000002e-05 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.447809+0000", 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8810000000000008e-06 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257202+0000", 2024-09-14T04:24:21.044 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.80939298400000004 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257208+0000", 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5259999999999999e-06 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.756948+0000", 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.499740278 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.757049+0000", 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000100499 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.757150+0000", 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000101594 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.761574+0000", 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0044240340000000003 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.768699+0000", 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0071247050000000003 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.781268+0000", 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012569169999999999 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.781312+0000", 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3226999999999998e-05 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.784592+0000", 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0032802460000000001 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:74420 13.0 13:0d590947:::benchmark_data_smithi114_288325_object74419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:08.497363+0000", 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 72.430334082000002, 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.291021175, 2024-09-14T04:24:21.048 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74420 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.497363+0000", 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.050 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.050 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.497363+0000", 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.497365+0000", 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.691e-06 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.051 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.497386+0000", 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1392000000000001e-05 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.497399+0000", 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2437e-05 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.497405+0000", 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7769999999999999e-06 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.497455+0000", 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0691e-05 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:08.497463+0000", 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1200000000000002e-06 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257232+0000", 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.75976872600000001 2024-09-14T04:24:21.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:09.257239+0000", 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8129999999999998e-06 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.759033+0000", 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.501793905 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.759196+0000", 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016353299999999999 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.759372+0000", 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017578299999999999 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.761690+0000", 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023175420000000001 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.055 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.772548+0000", 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010857843000000001 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.784666+0000", 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012118370999999999 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.784736+0000", 2024-09-14T04:24:21.056 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0077e-05 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:10.788384+0000", 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036484740000000001 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:118334 13.0 13:09a71e48:::benchmark_data_smithi114_288325_object23269:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:24:18.810397+0000", 2024-09-14T04:24:21.057 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 2.1173000900000001, 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.12693594699999999, 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 118334 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.810397+0000", 2024-09-14T04:24:21.058 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.810397+0000", 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.810398+0000", 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3500000000000005e-07 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.059 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.810401+0000", 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3239999999999999e-06 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.810410+0000", 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4600000000000003e-06 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.810416+0000", 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3799999999999999e-06 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.875664+0000", 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.065248076000000002 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.876391+0000", 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072692600000000003 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.902185+0000", 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.025793951999999998 2024-09-14T04:24:21.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.902268+0000", 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2397000000000004e-05 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.902420+0000", 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000152096 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.916496+0000", 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014075762 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.916536+0000", 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0182999999999998e-05 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.934328+0000", 2024-09-14T04:24:21.063 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017792569000000001 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.934374+0000", 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5355000000000002e-05 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.937333+0000", 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0029596319999999998 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.064 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:118421 13.0 13:077b1d2a:::benchmark_data_smithi114_288325_object23356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:24:18.963088+0000", 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 1.9646092959999999, 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.109623085, 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.065 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 118421 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.963088+0000", 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.963088+0000", 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.066 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.963089+0000", 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2589999999999999e-06 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.963095+0000", 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.801e-06 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.963109+0000", 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3813e-05 2024-09-14T04:24:21.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.963117+0000", 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6629999999999993e-06 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.023736+0000", 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.060619656000000001 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:24:21.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.024504+0000", 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00076771599999999999 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.041668+0000", 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017163734 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.041788+0000", 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000119805 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.041970+0000", 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018233299999999999 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.057293+0000", 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015323279 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.069196+0000", 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011903244 2024-09-14T04:24:21.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.069274+0000", 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7618000000000001e-05 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.069325+0000", 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0485000000000002e-05 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.072711+0000", 2024-09-14T04:24:21.071 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033866790000000001 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:118424 13.0 13:0f075621:::benchmark_data_smithi114_288325_object23359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:24:18.966994+0000", 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 1.960703225, 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.110773918, 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.072 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 118424 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.966994+0000", 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.073 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.966994+0000", 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.966996+0000", 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7e-06 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.967001+0000", 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1680000000000003e-06 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.074 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.967014+0000", 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2933e-05 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.967021+0000", 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1500000000000002e-06 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.032285+0000", 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.065263654000000004 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.033143+0000", 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000858032 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.057320+0000", 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024177321000000002 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.057395+0000", 2024-09-14T04:24:21.076 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4498000000000004e-05 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.057509+0000", 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000113964 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.069215+0000", 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011706161 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.072801+0000", 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003586152 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.074992+0000", 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021914069999999998 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.075032+0000", 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9959999999999997e-05 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.077768+0000", 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027358180000000001 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:118432 13.0 13:01968393:::benchmark_data_smithi114_288325_object23367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:24:18.977340+0000", 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 1.950357637, 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.10358943599999999, 2024-09-14T04:24:21.079 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 118432 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.977340+0000", 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.977340+0000", 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.977341+0000", 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.677e-06 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.977347+0000", 2024-09-14T04:24:21.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0349999999999998e-06 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.977364+0000", 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6337999999999998e-05 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:18.977372+0000", 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2770000000000005e-06 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.082 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.033179+0000", 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.055806715999999999 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.033974+0000", 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00079540199999999996 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.066341+0000", 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.032367088000000002 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.066412+0000", 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1160000000000003e-05 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,6", 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.066551+0000", 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013892599999999999 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.073613+0000", 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0070618299999999998 2024-09-14T04:24:21.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.077808+0000", 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041946709999999996 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.078026+0000", 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000218168 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.078066+0000", 2024-09-14T04:24:21.085 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9994999999999999e-05 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:19.080929+0000", 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0028631529999999998 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:21.086 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:24:21.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:21.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:21.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:21.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:21.279 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:21.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:21.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:21.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:21.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:21.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:21.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:21.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:21.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:21.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:21.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:21.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:21.824 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:21.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:24:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:24:21] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-09-14T04:24:21.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:21.954 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:21.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:22.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:22.092 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:22.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:22.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-14T04:24:22.193 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:22.218 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:22.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:22.306 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:22 smithi114 ceph-mon[106969]: pgmap v1335: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 629 op/s; 35269/216330 objects degraded (16.303%); 1334/216330 objects misplaced (0.617%); 2.0 MiB/s, 31 objects/s recovering 2024-09-14T04:24:22.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:22 smithi114 ceph-mon[103029]: pgmap v1335: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 629 op/s; 35269/216330 objects degraded (16.303%); 1334/216330 objects misplaced (0.617%); 2.0 MiB/s, 31 objects/s recovering 2024-09-14T04:24:22.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:22.363 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:22.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:22.435 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:22.436 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:22.438 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:24:22.439 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:24:22.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:22.470 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:24:22.470 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:24:22.503 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:24:22.503 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:24:22.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:22.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:22.533 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:24:22.533 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:24:22.561 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:24:22.561 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:24:22.589 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:24:22.589 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:24:22.618 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:24:22.618 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:24:22.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:22.636 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:24:22.637 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:24:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:22.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:22.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:22 smithi160 ceph-mon[88407]: pgmap v1335: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 629 op/s; 35269/216330 objects degraded (16.303%); 1334/216330 objects misplaced (0.617%); 2.0 MiB/s, 31 objects/s recovering 2024-09-14T04:24:22.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:22.776 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:22.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:22.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:22.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:22.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:23.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:23.028 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:23.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:23.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:23.157 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:23.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:23.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:23.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:23.294 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:23.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:23.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:23.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:23 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1254219096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:24:23.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1254219096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:24:23.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:23.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:23.420 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:23.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:23.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:23.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:23.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:23.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:23.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:23 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1254219096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:24:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:23.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:23.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:23.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:23.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:23.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:23.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:23.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:23.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:23.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:23.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:24.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:24.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:24.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:24.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:24.236 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:24.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:24.338 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:24.362 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:24.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:24.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:24.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:24.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:24.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:24.628 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:24.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:24.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:24.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:24.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:24.872 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:24.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:24.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:24.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:24 smithi114 ceph-mon[106969]: pgmap v1336: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 34905/213444 objects degraded (16.353%); 1334/213444 objects misplaced (0.625%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:24:24.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 34905/213444 objects degraded (16.353%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:24.929 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:24 smithi114 ceph-mon[103029]: pgmap v1336: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 34905/213444 objects degraded (16.353%); 1334/213444 objects misplaced (0.625%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:24:24.930 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 34905/213444 objects degraded (16.353%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:24.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:25.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:25.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:25.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:24 smithi160 ceph-mon[88407]: pgmap v1336: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 34905/213444 objects degraded (16.353%); 1334/213444 objects misplaced (0.625%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:24:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 34905/213444 objects degraded (16.353%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:25.175 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:25.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:25.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:25.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:25.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:25.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:25.475 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:25.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:25.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:25.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:25.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:25.703 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:25.729 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:25.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:25.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:25.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:25.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:25.879 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:25.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:25.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:26.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:26.128 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:26.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:26.230 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:26.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:26.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:26.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:26.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:26.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:26.506 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:26.532 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:26.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:26.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:26.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:26.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:26.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:26.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:26.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:26.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:26.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:26.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:26.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:27.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:27.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:26 smithi114 ceph-mon[106969]: pgmap v1337: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 34905/207039 objects degraded (16.859%); 1138/207039 objects misplaced (0.550%); 3.3 MiB/s, 53 objects/s recovering 2024-09-14T04:24:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:26 smithi114 ceph-mon[103029]: pgmap v1337: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 34905/207039 objects degraded (16.859%); 1138/207039 objects misplaced (0.550%); 3.3 MiB/s, 53 objects/s recovering 2024-09-14T04:24:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:26 smithi160 ceph-mon[88407]: pgmap v1337: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 34905/207039 objects degraded (16.859%); 1138/207039 objects misplaced (0.550%); 3.3 MiB/s, 53 objects/s recovering 2024-09-14T04:24:27.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:27.193 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:27.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:27.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:27.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:27.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:27.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:27.465 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:27.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:27.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:27.591 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:27.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:27.688 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:24:27.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:27.719 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:27.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:27.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:27.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:27.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:27.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:27 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2260357332' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:27.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:27 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:27.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:27 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2260357332' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:27.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:27 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:27.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:24:27.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:28.000 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:28.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:28.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:28.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:28.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:28.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:27 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2260357332' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:28.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:27 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:28.245 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:28.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:28.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:28.373 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:24:28.373 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:24:28.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:28.414 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:28.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:28.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:28.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:28.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:28.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:28.666 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:28.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:28.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:28.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:28.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:28.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:28.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:28.963 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:28.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:28.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:29.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:28 smithi114 ceph-mon[106969]: pgmap v1338: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 595 op/s; 34905/207039 objects degraded (16.859%); 1138/207039 objects misplaced (0.550%); 3.2 MiB/s, 51 objects/s recovering 2024-09-14T04:24:29.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:28 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:24:29.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:28 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:24:29.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:28 smithi114 ceph-mon[106969]: osdmap e345: 8 total, 8 up, 7 in 2024-09-14T04:24:29.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:28 smithi114 ceph-mon[103029]: pgmap v1338: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 595 op/s; 34905/207039 objects degraded (16.859%); 1138/207039 objects misplaced (0.550%); 3.2 MiB/s, 51 objects/s recovering 2024-09-14T04:24:29.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:28 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:24:29.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:28 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:24:29.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:28 smithi114 ceph-mon[103029]: osdmap e345: 8 total, 8 up, 7 in 2024-09-14T04:24:29.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:29.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:29.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:28 smithi160 ceph-mon[88407]: pgmap v1338: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 595 op/s; 34905/207039 objects degraded (16.859%); 1138/207039 objects misplaced (0.550%); 3.2 MiB/s, 51 objects/s recovering 2024-09-14T04:24:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:28 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:24:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:28 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:24:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:28 smithi160 ceph-mon[88407]: osdmap e345: 8 total, 8 up, 7 in 2024-09-14T04:24:29.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:29.228 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:29.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:29.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:29.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:29.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:29.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:29.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:29.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:29.539 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:29.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:29.644 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:29.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:29.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:29.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:29.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:29.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:29.924 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:29.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:29.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:30.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:30.029 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:29 smithi114 ceph-mon[106969]: pgmap v1340: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 619 op/s; 34531/198255 objects degraded (17.417%); 961/198255 objects misplaced (0.485%); 4.6 MiB/s, 74 objects/s recovering 2024-09-14T04:24:30.029 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 34905/207039 objects degraded (16.859%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:30.030 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:29 smithi114 ceph-mon[103029]: pgmap v1340: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 619 op/s; 34531/198255 objects degraded (17.417%); 961/198255 objects misplaced (0.485%); 4.6 MiB/s, 74 objects/s recovering 2024-09-14T04:24:30.030 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 34905/207039 objects degraded (16.859%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:30.051 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:30.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:29 smithi160 ceph-mon[88407]: pgmap v1340: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 619 op/s; 34531/198255 objects degraded (17.417%); 961/198255 objects misplaced (0.485%); 4.6 MiB/s, 74 objects/s recovering 2024-09-14T04:24:30.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 34905/207039 objects degraded (16.859%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:30.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-14T04:24:30.153 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:30.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:30.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:30.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:30.350 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:30.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:30.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:24:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:24:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:24:30.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:30.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:30.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:30.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:30.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:30.610 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:30.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:30.712 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:30.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:30.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:30.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:30.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:30.872 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:30.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:30.974 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:31.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:31.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:31.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:24:31.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:31.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:31.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:31.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:31.294 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:31.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:31.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:31.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:31.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:31.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:31.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:31.573 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:31.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:31.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:31.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:31.711 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:24:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:24:31] "GET /metrics HTTP/1.1" 200 36613 "" "Prometheus/2.51.0" 2024-09-14T04:24:31.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:31.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:31.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:31.870 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:31.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:31.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:32.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:32.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:32.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:32.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:32.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:32.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:32.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:32.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:32.287 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:32.305 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:32.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:32.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:32.436 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:32.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:32.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:32.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:32.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:32.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:32.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:32 smithi114 ceph-mon[106969]: pgmap v1341: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 619 op/s; 34531/197730 objects degraded (17.464%); 961/197730 objects misplaced (0.486%); 4.6 MiB/s, 74 objects/s recovering 2024-09-14T04:24:32.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:32 smithi114 ceph-mon[103029]: pgmap v1341: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 619 op/s; 34531/197730 objects degraded (17.464%); 961/197730 objects misplaced (0.486%); 4.6 MiB/s, 74 objects/s recovering 2024-09-14T04:24:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:32 smithi160 ceph-mon[88407]: pgmap v1341: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 619 op/s; 34531/197730 objects degraded (17.464%); 961/197730 objects misplaced (0.486%); 4.6 MiB/s, 74 objects/s recovering 2024-09-14T04:24:32.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:32.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:32.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:32.736 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:32.811 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:32.851 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:32.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:32.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:32.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:32.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:32.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:33.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:33.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:33.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:33.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:33.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:33.268 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:33.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:33.369 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:33.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:33.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:33.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:33.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:33.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:33.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:33.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:33.706 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:33.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:33.808 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:33.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:33.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:33.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:33.975 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:33.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:33.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:34.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:34.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:34.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:34.222 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:34.224 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:24:34.224 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:24:34.224 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:24:34.224 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:24:34.224 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:24:34.224 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:24:34.225 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:24:34.225 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:24:34.225 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:24:34.225 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:24:34.242 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:24:34.243 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:24:34.262 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:24:34.262 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:24:34.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:34.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:34.297 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:24:34.297 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:24:34.368 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:34.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:34 smithi114 ceph-mon[103029]: pgmap v1342: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 614 op/s; 34179/195015 objects degraded (17.526%); 961/195015 objects misplaced (0.493%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T04:24:34.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:34 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1067895490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:34.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:34 smithi114 ceph-mon[106969]: pgmap v1342: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 614 op/s; 34179/195015 objects degraded (17.526%); 961/195015 objects misplaced (0.493%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T04:24:34.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:34 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1067895490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:34.393 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:34.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:34.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:34.494 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:34.539 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:34.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:34.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:34 smithi160 ceph-mon[88407]: pgmap v1342: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 614 op/s; 34179/195015 objects degraded (17.526%); 961/195015 objects misplaced (0.493%); 4.6 MiB/s, 73 objects/s recovering 2024-09-14T04:24:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:34 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1067895490' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:34.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:34.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:34.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:34.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:34.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:34.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:34.966 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:34.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:35.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:35.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:35.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:35.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:35.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:35.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:35.339 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:35.378 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:35.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:35.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:35.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:35.518 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:35.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:35.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:35 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 34179/195015 objects degraded (17.526%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:35.593 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:35 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 34179/195015 objects degraded (17.526%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:35.620 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:35.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:35 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 34179/195015 objects degraded (17.526%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:35.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:35.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:35.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:35.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:35.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:35.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:35.921 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:35.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:36.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:36.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:36.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:36.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:36.191 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:36.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:36.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:36.322 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:36.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:36.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:36 smithi114 ceph-mon[106969]: pgmap v1343: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 34179/189027 objects degraded (18.082%); 772/189027 objects misplaced (0.408%); 4.6 MiB/s, 72 objects/s recovering 2024-09-14T04:24:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:36 smithi114 ceph-mon[103029]: pgmap v1343: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 34179/189027 objects degraded (18.082%); 772/189027 objects misplaced (0.408%); 4.6 MiB/s, 72 objects/s recovering 2024-09-14T04:24:36.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:36.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:36.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:36.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:36.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:36.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:36.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:36 smithi160 ceph-mon[88407]: pgmap v1343: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 34179/189027 objects degraded (18.082%); 772/189027 objects misplaced (0.408%); 4.6 MiB/s, 72 objects/s recovering 2024-09-14T04:24:36.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:36.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:36.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:36.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:36.849 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:36.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:36.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:36.975 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:36.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:37.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:37.103 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:37.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:37.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:37.227 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 6, 7, 4, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:24:37.227 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:24:37.227 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 3, 6, 7, 4, 5] 2024-09-14T04:24:37.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd out 1 2024-09-14T04:24:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:37.250 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:37.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:37.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:37.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:37.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:37.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:37.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:37.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:37.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:37.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:37.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:37.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:37.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:37.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:37.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:37.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:37.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:37.959 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:37.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:37.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:38.061 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:38.099 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:38.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:38.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:38.201 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:38.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:38.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:38.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:38.370 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:38.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:38.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:38 smithi114 ceph-mon[106969]: pgmap v1344: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 34179/189027 objects degraded (18.082%); 772/189027 objects misplaced (0.408%); 4.6 MiB/s, 72 objects/s recovering 2024-09-14T04:24:38.463 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:38.463 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:38 smithi114 ceph-mon[103029]: pgmap v1344: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 34179/189027 objects degraded (18.082%); 772/189027 objects misplaced (0.408%); 4.6 MiB/s, 72 objects/s recovering 2024-09-14T04:24:38.464 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:38.469 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:24:38.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:38.504 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:38.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:38.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:38.644 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:38.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:38 smithi160 ceph-mon[88407]: pgmap v1344: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 34179/189027 objects degraded (18.082%); 772/189027 objects misplaced (0.408%); 4.6 MiB/s, 72 objects/s recovering 2024-09-14T04:24:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:38.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:38.788 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:38.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:38.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:38.919 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:38.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:39.021 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:39.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:39.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:39.102 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:24:39.102 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:24:39.102 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:24:39.102 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:24:39.102 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:24:39.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:39.196 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:39.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:39.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:39.323 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:39.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:39.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:39.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:39.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:39.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:39.583 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:39.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:39.623 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:39.623 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:39.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:39.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:39.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:39.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:39.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:39.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:39.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:39.952 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:39.975 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:39.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:40.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:40.101 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:40.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:40.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:40.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:40.245 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:40.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:40.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:40.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:40 smithi114 ceph-mon[103029]: pgmap v1345: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 689 op/s; 33799/180399 objects degraded (18.736%); 564/180399 objects misplaced (0.313%); 5.1 MiB/s, 82 objects/s recovering 2024-09-14T04:24:40.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:40 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 33799/180399 objects degraded (18.736%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:40.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2149793631' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:24:40.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:40 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:24:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:40 smithi114 ceph-mon[106969]: pgmap v1345: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 689 op/s; 33799/180399 objects degraded (18.736%); 564/180399 objects misplaced (0.313%); 5.1 MiB/s, 82 objects/s recovering 2024-09-14T04:24:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:40 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 33799/180399 objects degraded (18.736%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2149793631' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:24:40.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:40 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:24:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:40.393 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:40.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:40.494 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:40.547 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:40.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:40.570 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:24:40.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:40 smithi160 ceph-mon[88407]: pgmap v1345: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 689 op/s; 33799/180399 objects degraded (18.736%); 564/180399 objects misplaced (0.313%); 5.1 MiB/s, 82 objects/s recovering 2024-09-14T04:24:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:40 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 33799/180399 objects degraded (18.736%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2149793631' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:24:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:40 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:24:40.659 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:24:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:24:40.216Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:24:40.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:40.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:40.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:40.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:40.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:40.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:40.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:40.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:40.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:41.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:41.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:41.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:41.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-14T04:24:41.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:41.210 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:41.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:41.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:41.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:41.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:41.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:41.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:41.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:41 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:24:41.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:41 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:24:41.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:41 smithi114 ceph-mon[106969]: osdmap e346: 8 total, 8 up, 7 in 2024-09-14T04:24:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:41 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:24:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:41 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:24:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:41 smithi114 ceph-mon[103029]: osdmap e346: 8 total, 8 up, 7 in 2024-09-14T04:24:41.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:41.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:41.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:41.480 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:41 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:24:41.480 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:41 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:24:41.480 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:41 smithi160 ceph-mon[88407]: osdmap e346: 8 total, 8 up, 7 in 2024-09-14T04:24:41.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:41.605 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:41.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:41.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:41.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:41.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:41.806 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:24:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:24:41] "GET /metrics HTTP/1.1" 200 36604 "" "Prometheus/2.51.0" 2024-09-14T04:24:41.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:41.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:41.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:41.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:42.016 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:42.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:42.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:42.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:42.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:42.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:42.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:42.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:42.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:42.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:42.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:42.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:42.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:42.551 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:42.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:42.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:42.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:42.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:42.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:42.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:42.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:42.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:42 smithi114 ceph-mon[103029]: pgmap v1347: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 595 op/s; 33799/179871 objects degraded (18.791%); 564/179871 objects misplaced (0.314%); 4.8 MiB/s, 76 objects/s recovering 2024-09-14T04:24:42.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:42 smithi114 ceph-mon[106969]: pgmap v1347: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 595 op/s; 33799/179871 objects degraded (18.791%); 564/179871 objects misplaced (0.314%); 4.8 MiB/s, 76 objects/s recovering 2024-09-14T04:24:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:42 smithi160 ceph-mon[88407]: pgmap v1347: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 595 op/s; 33799/179871 objects degraded (18.791%); 564/179871 objects misplaced (0.314%); 4.8 MiB/s, 76 objects/s recovering 2024-09-14T04:24:42.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:42.980 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:42.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:43.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:43.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:43.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:43.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:43.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:43.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:43.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:43.393 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:43.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:43.494 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:43.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:43.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:43.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:43.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:43.653 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:43.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:43.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:43.782 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:43.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:43.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:43.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:43.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:44.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:44.077 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:44.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:44.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:44.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:44.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:44.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:44.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:44.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:44.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:24:44.382 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:44.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:44.490 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:44.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:44.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:44 smithi114 ceph-mon[103029]: pgmap v1348: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 33415/177006 objects degraded (18.878%); 564/177006 objects misplaced (0.319%); 4.9 MiB/s, 77 objects/s recovering 2024-09-14T04:24:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:44 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 33415/177006 objects degraded (18.878%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:44 smithi114 ceph-mon[106969]: pgmap v1348: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 33415/177006 objects degraded (18.878%); 564/177006 objects misplaced (0.319%); 4.9 MiB/s, 77 objects/s recovering 2024-09-14T04:24:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:44 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 33415/177006 objects degraded (18.878%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:44.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:44.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:44.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:44.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:44.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:44.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:44 smithi160 ceph-mon[88407]: pgmap v1348: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 33415/177006 objects degraded (18.878%); 564/177006 objects misplaced (0.319%); 4.9 MiB/s, 77 objects/s recovering 2024-09-14T04:24:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:44 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 33415/177006 objects degraded (18.878%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:44.917 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:44.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:45.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:45.046 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:24:45.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:45.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:45.064 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:24:45.065 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:24:45.086 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:24:45.087 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:24:45.104 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:24:45.104 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:24:45.145 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:24:45.145 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:24:45.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:45.190 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:24:45.190 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:24:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:45.194 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:45.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:45.215 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:24:45.216 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:24:45.244 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:24:45.244 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:24:45.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:45.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:45.427 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:45.451 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:45.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:45.553 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:45.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:45.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:45.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:45.733 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:45.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:45.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:45.862 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:45 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2334793260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:45.862 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:45 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2334793260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:45.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:45.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:45 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2334793260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:45.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:46.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:46.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:46.115 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:46 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:24:46.019+0000 7f3162f23640 -1 mon.a@0(leader).osd e346 definitely_dead 0 2024-09-14T04:24:46.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:46.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:46.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:46.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:46.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:46.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:46.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:46.398 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:46.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:46.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:46.539 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:46.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:46.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:46.588 INFO:teuthology.orchestra.run.smithi114.stderr:marked out osd.1. 2024-09-14T04:24:46.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:46.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:46.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:46.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:46.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:46.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:46.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:46 smithi114 ceph-mon[103029]: pgmap v1349: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 606 op/s; 33415/170841 objects degraded (19.559%); 323/170841 objects misplaced (0.189%); 5.2 MiB/s, 83 objects/s recovering 2024-09-14T04:24:46.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3875822580' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-14T04:24:46.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:46 smithi114 ceph-mon[103029]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-14T04:24:46.841 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:46 smithi114 ceph-mon[106969]: pgmap v1349: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 606 op/s; 33415/170841 objects degraded (19.559%); 323/170841 objects misplaced (0.189%); 5.2 MiB/s, 83 objects/s recovering 2024-09-14T04:24:46.841 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3875822580' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-14T04:24:46.841 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:46 smithi114 ceph-mon[106969]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-14T04:24:46.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:46 smithi160 ceph-mon[88407]: pgmap v1349: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 606 op/s; 33415/170841 objects degraded (19.559%); 323/170841 objects misplaced (0.189%); 5.2 MiB/s, 83 objects/s recovering 2024-09-14T04:24:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3875822580' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-14T04:24:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:46 smithi160 ceph-mon[88407]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-14T04:24:46.911 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:46.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:47.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:47.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:47.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:47.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:47.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:47.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:47.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:47.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:47.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:47.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:47.407 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:47.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:47.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:47.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:47.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:47.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:47.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:47.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:47.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:47.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:47.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:47.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:47 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3875822580' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-14T04:24:47.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:47 smithi114 ceph-mon[106969]: osdmap e347: 8 total, 8 up, 6 in 2024-09-14T04:24:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3875822580' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-14T04:24:47.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:47 smithi114 ceph-mon[103029]: osdmap e347: 8 total, 8 up, 6 in 2024-09-14T04:24:47.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:47.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:47 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3875822580' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-14T04:24:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:47 smithi160 ceph-mon[88407]: osdmap e347: 8 total, 8 up, 6 in 2024-09-14T04:24:47.921 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:47.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:48.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:48.046 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:48.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:48.079 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:48.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:48.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:48.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:48.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:48.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:48.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:48.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:48.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:48.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:48.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:48.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:48.541 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:48.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:48.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:48.665 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:48.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:48.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:48.790 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:48.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:48.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:48 smithi114 ceph-mon[103029]: pgmap v1351: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 398 op/s; 33415/170841 objects degraded (19.559%); 323/170841 objects misplaced (0.189%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T04:24:48.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:48 smithi114 ceph-mon[103029]: 13.0 starting backfill to osd.4 from (0'0,0'0] MAX to 346'8336 2024-09-14T04:24:48.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:48 smithi114 ceph-mon[103029]: osdmap e348: 8 total, 8 up, 6 in 2024-09-14T04:24:48.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:48 smithi114 ceph-mon[106969]: pgmap v1351: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 398 op/s; 33415/170841 objects degraded (19.559%); 323/170841 objects misplaced (0.189%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T04:24:48.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:48 smithi114 ceph-mon[106969]: 13.0 starting backfill to osd.4 from (0'0,0'0] MAX to 346'8336 2024-09-14T04:24:48.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:48 smithi114 ceph-mon[106969]: osdmap e348: 8 total, 8 up, 6 in 2024-09-14T04:24:48.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:48 smithi160 ceph-mon[88407]: pgmap v1351: 89 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 398 op/s; 33415/170841 objects degraded (19.559%); 323/170841 objects misplaced (0.189%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T04:24:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:48 smithi160 ceph-mon[88407]: 13.0 starting backfill to osd.4 from (0'0,0'0] MAX to 346'8336 2024-09-14T04:24:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:48 smithi160 ceph-mon[88407]: osdmap e348: 8 total, 8 up, 6 in 2024-09-14T04:24:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:48.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:48.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:24:49.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:49.060 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:49.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:49.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:49.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:49.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:49.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:49.336 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:49.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:49.437 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:49.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:49.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:49.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:49.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:49.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:49.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:49.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:49.745 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:49.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:49.788 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:24:49.788 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:24:49.847 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:49.868 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:49.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[103029]: osdmap e349: 8 total, 8 up, 6 in 2024-09-14T04:24:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[103029]: 13.a starting backfill to osd.3 from (0'0,0'0] MAX to 346'8334 2024-09-14T04:24:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[103029]: 13.c starting backfill to osd.4 from (0'0,0'0] MAX to 346'8469 2024-09-14T04:24:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[103029]: 13.4 starting backfill to osd.0 from (0'0,0'0] MAX to 346'8503 2024-09-14T04:24:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[103029]: 13.d starting backfill to osd.5 from (0'0,0'0] MAX to 346'8538 2024-09-14T04:24:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 33415/170841 objects degraded (19.559%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[106969]: osdmap e349: 8 total, 8 up, 6 in 2024-09-14T04:24:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[106969]: 13.a starting backfill to osd.3 from (0'0,0'0] MAX to 346'8334 2024-09-14T04:24:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[106969]: 13.c starting backfill to osd.4 from (0'0,0'0] MAX to 346'8469 2024-09-14T04:24:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[106969]: 13.4 starting backfill to osd.0 from (0'0,0'0] MAX to 346'8503 2024-09-14T04:24:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[106969]: 13.d starting backfill to osd.5 from (0'0,0'0] MAX to 346'8538 2024-09-14T04:24:49.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:49 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 33415/170841 objects degraded (19.559%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:49 smithi160 ceph-mon[88407]: osdmap e349: 8 total, 8 up, 6 in 2024-09-14T04:24:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:49 smithi160 ceph-mon[88407]: 13.a starting backfill to osd.3 from (0'0,0'0] MAX to 346'8334 2024-09-14T04:24:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:49 smithi160 ceph-mon[88407]: 13.c starting backfill to osd.4 from (0'0,0'0] MAX to 346'8469 2024-09-14T04:24:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:49 smithi160 ceph-mon[88407]: 13.4 starting backfill to osd.0 from (0'0,0'0] MAX to 346'8503 2024-09-14T04:24:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:49 smithi160 ceph-mon[88407]: 13.d starting backfill to osd.5 from (0'0,0'0] MAX to 346'8538 2024-09-14T04:24:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:49 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 33415/170841 objects degraded (19.559%), 9 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:49.970 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:50.016 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:50.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:50.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:50.141 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:50.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:50.242 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:50.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:50.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:50.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:50.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:50.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:50.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:50.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:50.616 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:24:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:24:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:24:50.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:50.655 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:50.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:50.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-14T04:24:50.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:50.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:50.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:50.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:50 smithi114 ceph-mon[106969]: pgmap v1354: 89 pgs: 2 activating+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 remapped+peering, 5 peering, 2 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 652 op/s; 21132/165252 objects degraded (12.788%); 8550/165252 objects misplaced (5.174%); 10 MiB/s, 164 objects/s recovering 2024-09-14T04:24:50.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:50 smithi114 ceph-mon[106969]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-14T04:24:50.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:50 smithi114 ceph-mon[103029]: pgmap v1354: 89 pgs: 2 activating+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 remapped+peering, 5 peering, 2 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 652 op/s; 21132/165252 objects degraded (12.788%); 8550/165252 objects misplaced (5.174%); 10 MiB/s, 164 objects/s recovering 2024-09-14T04:24:50.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:50 smithi114 ceph-mon[103029]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-14T04:24:50.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:50 smithi160 ceph-mon[88407]: pgmap v1354: 89 pgs: 2 activating+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 remapped+peering, 5 peering, 2 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 652 op/s; 21132/165252 objects degraded (12.788%); 8550/165252 objects misplaced (5.174%); 10 MiB/s, 164 objects/s recovering 2024-09-14T04:24:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:50 smithi160 ceph-mon[88407]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-14T04:24:50.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:50.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:51.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:51.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:51.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:51.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:51.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:51.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:51.315 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:51.327 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:51.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:51.367 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:51.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:51.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:51.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:51.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:51.591 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:24:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:24:51] "GET /metrics HTTP/1.1" 200 36604 "" "Prometheus/2.51.0" 2024-09-14T04:24:51.598 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:51.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:51.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:51.745 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:51.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:51.847 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:51.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:51.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:51.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:52.021 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:52.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:52.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:52.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:52.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:52.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:52.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:52.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:52.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:52.433 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:52.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:52.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:52.545 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 7, 4, 5] out_osds: [2, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:24:52.545 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:24:52.545 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 3, 6, 7, 4, 5] 2024-09-14T04:24:52.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd out 7 2024-09-14T04:24:52.557 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:52.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:52.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:52.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:52.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:52.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:52.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:52.841 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:52.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:52.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:52 smithi114 ceph-mon[103029]: pgmap v1355: 89 pgs: 2 activating+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 remapped+peering, 5 peering, 2 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 329 op/s; 21132/164907 objects degraded (12.814%); 8550/164907 objects misplaced (5.185%); 7.7 MiB/s, 124 objects/s recovering 2024-09-14T04:24:52.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:52 smithi114 ceph-mon[106969]: pgmap v1355: 89 pgs: 2 activating+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 remapped+peering, 5 peering, 2 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 329 op/s; 21132/164907 objects degraded (12.814%); 8550/164907 objects misplaced (5.185%); 7.7 MiB/s, 124 objects/s recovering 2024-09-14T04:24:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:52 smithi160 ceph-mon[88407]: pgmap v1355: 89 pgs: 2 activating+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 remapped+peering, 5 peering, 2 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 329 op/s; 21132/164907 objects degraded (12.814%); 8550/164907 objects misplaced (5.185%); 7.7 MiB/s, 124 objects/s recovering 2024-09-14T04:24:52.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:52.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:52.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:53.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:53.125 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:53.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:53.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:53.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:53.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:53.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:53.401 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:53.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:53.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:53.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:53.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:53.658 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:53.669 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:24:53.708 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:53.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:53.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:53.857 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:53.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:53.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:24:53.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:53.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/608114697' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:53.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:24:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:53.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/608114697' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:53.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:53.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:53.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:53.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:54.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:54.110 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:54.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:54.145 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:24:54.145 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:24:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:24:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:24:54.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/608114697' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:24:54.165 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:24:54.165 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:24:54.185 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:24:54.186 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:24:54.204 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:24:54.204 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:24:54.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:24:54.230 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:24:54.230 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:24:54.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:54.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:24:54.263 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:24:54.264 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:24:54.287 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:24:54.288 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:24:54.320 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:24:54.321 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:24:54.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:54.401 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:54.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:54.503 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:54.525 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:54.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:54.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:54.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:54.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:54.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:54.775 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:54.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:54.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:54.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:54.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:55.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:55.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:55.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:55.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[106969]: pgmap v1356: 89 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 2 activating+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 368 op/s; 28550/163584 objects degraded (17.453%); 19404/163584 objects misplaced (11.862%); 10 MiB/s, 165 objects/s recovering 2024-09-14T04:24:55.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[106969]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-14T04:24:55.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:24:55.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/608114697' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:24:55.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[106969]: osdmap e350: 8 total, 8 up, 6 in 2024-09-14T04:24:55.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1962249863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:55.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:24:55.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 28550/163584 objects degraded (17.453%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:55.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[103029]: pgmap v1356: 89 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 2 activating+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 368 op/s; 28550/163584 objects degraded (17.453%); 19404/163584 objects misplaced (11.862%); 10 MiB/s, 165 objects/s recovering 2024-09-14T04:24:55.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[103029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-14T04:24:55.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:24:55.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/608114697' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:24:55.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[103029]: osdmap e350: 8 total, 8 up, 6 in 2024-09-14T04:24:55.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1962249863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:55.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:24:55.122 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:54 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 28550/163584 objects degraded (17.453%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:54 smithi160 ceph-mon[88407]: pgmap v1356: 89 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 2 activating+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 0 B/s wr, 368 op/s; 28550/163584 objects degraded (17.453%); 19404/163584 objects misplaced (11.862%); 10 MiB/s, 165 objects/s recovering 2024-09-14T04:24:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:54 smithi160 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-14T04:24:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:54 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:24:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:54 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/608114697' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:24:55.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:54 smithi160 ceph-mon[88407]: osdmap e350: 8 total, 8 up, 6 in 2024-09-14T04:24:55.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:54 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1962249863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:24:55.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:24:55.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:54 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 28550/163584 objects degraded (17.453%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:55.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:55.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:55.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:55.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:55.412 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:55.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:55.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:55.564 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:55.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:55.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:55.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:55.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:55.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:55.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:55.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:55.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:55.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:55.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:55.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:56.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:56.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:56.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:56.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:56.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:56.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:56.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:56.334 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:56.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:56.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:56.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:56.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:56.526 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:56.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:56.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:56.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:56.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:56.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:56.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:56.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:56.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:56.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:56.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:56 smithi114 ceph-mon[106969]: pgmap v1358: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 461 op/s; 29041/158163 objects degraded (18.361%); 18149/158163 objects misplaced (11.475%); 12 MiB/s, 187 objects/s recovering 2024-09-14T04:24:56.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:56 smithi114 ceph-mon[103029]: pgmap v1358: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 461 op/s; 29041/158163 objects degraded (18.361%); 18149/158163 objects misplaced (11.475%); 12 MiB/s, 187 objects/s recovering 2024-09-14T04:24:56.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:56.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:56.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:56.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:57.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:57.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:57.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:57.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:56 smithi160 ceph-mon[88407]: pgmap v1358: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 461 op/s; 29041/158163 objects degraded (18.361%); 18149/158163 objects misplaced (11.475%); 12 MiB/s, 187 objects/s recovering 2024-09-14T04:24:57.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:57.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:57.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:57.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:57.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:57.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:57.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:57.531 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:57.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:57.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:57.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:57.710 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:57.735 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:57.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:57.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:57.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:57.863 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:57.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:57.964 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:24:57.964 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:24:57.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.964 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:139468 13.3 13:cc70adbc:::benchmark_data_smithi114_288325_object44403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-14T04:24:57.964 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:24:57.859006+0000", 2024-09-14T04:24:57.964 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.014511817, 2024-09-14T04:24:57.964 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014552371999999999, 2024-09-14T04:24:57.964 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:57.964 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139468 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.859006+0000", 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.859006+0000", 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.859008+0000", 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.26e-06 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.859013+0000", 2024-09-14T04:24:57.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9949999999999996e-06 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.859023+0000", 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0740000000000001e-05 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.859030+0000", 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5649999999999998e-06 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.859072+0000", 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.2230000000000001e-05 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.859162+0000", 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0134000000000001e-05 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.859242+0000", 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9684000000000005e-05 2024-09-14T04:24:57.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.862102+0000", 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002860123 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.862350+0000", 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024797099999999999 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:57.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:139471 13.b e350/332 13:dddf5770:::benchmark_data_smithi114_288325_object44406:head v 350'8800, mlcod=329'2611)", 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:24:57.871351+0000", 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0021674260000000001, 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022845209999999999, 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871351+0000", 2024-09-14T04:24:57.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871351+0000", 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871352+0000", 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1939999999999999e-06 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:57.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871362+0000", 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5179999999999993e-06 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871376+0000", 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.467e-05 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871384+0000", 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9419999999999995e-06 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871486+0000", 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000101436 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871504+0000", 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8785000000000001e-05 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:139472 13.6 e350/324 13:6fc8e546:::benchmark_data_smithi114_288325_object44407:head v 350'8860, mlcod=350'8858)", 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:24:57.871410+0000", 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0021082660000000001, 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022726360000000002, 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871410+0000", 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871410+0000", 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871410+0000", 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1100000000000002e-07 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871415+0000", 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4000000000000002e-06 2024-09-14T04:24:57.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:24:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871423+0000", 2024-09-14T04:24:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6470000000000004e-06 2024-09-14T04:24:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:24:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871428+0000", 2024-09-14T04:24:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9840000000000001e-06 2024-09-14T04:24:57.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.977 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.977 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:24:57.977 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871467+0000", 2024-09-14T04:24:57.977 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8277e-05 2024-09-14T04:24:57.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:24:57.977 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:24:57.977 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:24:57.977 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:57.871488+0000", 2024-09-14T04:24:57.977 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1512e-05 2024-09-14T04:24:57.978 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:57.978 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:24:57.978 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:57.978 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:24:57.978 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:24:57.978 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 3 2024-09-14T04:24:57.978 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:24:57.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:58.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:58.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:58.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:58.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:58.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:58.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:58.279 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:58.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:58.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:24:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:58.429 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:58.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:24:58.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:58.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:24:58.553 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:58.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:58.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:58.679 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:58.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:58.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:58.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:24:58.805 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:58.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:58.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:24:58.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:58 smithi114 ceph-mon[106969]: pgmap v1359: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 393 op/s; 29041/158163 objects degraded (18.361%); 18149/158163 objects misplaced (11.475%); 10 MiB/s, 165 objects/s recovering 2024-09-14T04:24:58.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:58 smithi114 ceph-mon[103029]: pgmap v1359: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 393 op/s; 29041/158163 objects degraded (18.361%); 18149/158163 objects misplaced (11.475%); 10 MiB/s, 165 objects/s recovering 2024-09-14T04:24:58.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:24:58.931 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:58.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:58.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:24:58.948 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:58 smithi160 ceph-mon[88407]: pgmap v1359: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 393 op/s; 29041/158163 objects degraded (18.361%); 18149/158163 objects misplaced (11.475%); 10 MiB/s, 165 objects/s recovering 2024-09-14T04:24:59.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:59.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:59.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:59.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-14T04:24:59.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:24:59.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:59.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:59.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:24:59.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:59.339 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:59.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:59.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:24:59.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:24:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:24:59.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:24:59.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:24:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:59.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:59.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:24:59.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:59.771 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:59.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:24:59.804 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:24:59.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:24:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:24:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 29041/158163 objects degraded (18.361%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:24:59.910 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:24:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:24:59.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:00.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:00.039 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:00.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:00.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:00.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:24:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 29041/158163 objects degraded (18.361%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:00.042 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:24:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 29041/158163 objects degraded (18.361%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:00.060 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:00.141 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:00.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:00.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:00.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:00.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:00.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:00.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:00.470 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:00.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:00.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:00.608 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:00.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:00.611 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:25:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:25:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:25:00.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:00.744 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:00.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:00.846 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:00.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:00.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:00.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:00 smithi160 ceph-mon[88407]: pgmap v1360: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 461 op/s; 28261/151392 objects degraded (18.667%); 17089/151392 objects misplaced (11.288%); 11 MiB/s, 170 objects/s recovering 2024-09-14T04:25:00.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:01.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:01.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:01.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:00 smithi114 ceph-mon[103029]: pgmap v1360: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 461 op/s; 28261/151392 objects degraded (18.667%); 17089/151392 objects misplaced (11.288%); 11 MiB/s, 170 objects/s recovering 2024-09-14T04:25:01.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:00 smithi114 ceph-mon[106969]: pgmap v1360: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.1 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 461 op/s; 28261/151392 objects degraded (18.667%); 17089/151392 objects misplaced (11.288%); 11 MiB/s, 170 objects/s recovering 2024-09-14T04:25:01.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:01.164 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:01.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:01.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:01.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:01.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:01.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:01.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:01.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:01.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:01.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:01.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:01.585 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:25:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:25:01] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-09-14T04:25:01.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:01.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:01.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:01.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:01.798 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:01.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:01.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:01 smithi114 ceph-mon[106969]: pgmap v1361: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 464 op/s; 28261/150972 objects degraded (18.719%); 17089/150972 objects misplaced (11.319%); 11 MiB/s, 170 objects/s recovering 2024-09-14T04:25:01.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:01 smithi114 ceph-mon[103029]: pgmap v1361: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 464 op/s; 28261/150972 objects degraded (18.719%); 17089/150972 objects misplaced (11.319%); 11 MiB/s, 170 objects/s recovering 2024-09-14T04:25:01.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:01.975 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:01.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:01.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:02.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:02.084 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:01 smithi160 ceph-mon[88407]: pgmap v1361: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 0 B/s wr, 464 op/s; 28261/150972 objects degraded (18.719%); 17089/150972 objects misplaced (11.319%); 11 MiB/s, 170 objects/s recovering 2024-09-14T04:25:02.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:02.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:02.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:02.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:02.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:02.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:02.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:02.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:02.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:02.547 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:02.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:02.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:02.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:02.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:02.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:02.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:02.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:02.825 INFO:teuthology.orchestra.run.smithi114.stderr:marked out osd.7. 2024-09-14T04:25:02.826 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:25:02.767+0000 7f3162f23640 -1 mon.a@0(leader).osd e350 definitely_dead 0 2024-09-14T04:25:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:02.832 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:02.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:02.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:02.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:02.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:03.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:03.080 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:03.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:03.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/658825658' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-14T04:25:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-14T04:25:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-mon[103029]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-14T04:25:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:03.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/658825658' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-14T04:25:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-14T04:25:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-mon[106969]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-14T04:25:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:02 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:02 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/658825658' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-14T04:25:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:02 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-14T04:25:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:02 smithi160 ceph-mon[88407]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-14T04:25:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:02 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:02 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:03.183 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:03.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:03.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:03.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:03.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:03.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:03.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:03.429 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:03.439 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:25:03.439 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:25:03.439 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:25:03.439 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:25:03.439 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:25:03.439 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:25:03.439 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:25:03.440 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:25:03.440 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:25:03.440 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:25:03.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:03.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:03.455 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:25:03.455 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:25:03.471 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:25:03.471 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:25:03.486 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:25:03.486 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:25:03.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:03.577 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:03.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:03.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:03.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:03.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:03.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:03.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:03.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:03.947 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:03.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:04.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:04.073 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:04.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:04.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:03 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-14T04:25:04.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:03 smithi114 ceph-mon[106969]: osdmap e351: 8 total, 8 up, 5 in 2024-09-14T04:25:04.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:03 smithi114 ceph-mon[106969]: pgmap v1363: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 486 op/s; 27852/149331 objects degraded (18.651%); 16475/149331 objects misplaced (11.033%); 12 MiB/s, 190 objects/s recovering 2024-09-14T04:25:04.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:03 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4115183268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:04.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:03 smithi114 ceph-mon[106969]: osdmap e352: 8 total, 8 up, 5 in 2024-09-14T04:25:04.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:03 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-14T04:25:04.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:03 smithi114 ceph-mon[103029]: osdmap e351: 8 total, 8 up, 5 in 2024-09-14T04:25:04.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:03 smithi114 ceph-mon[103029]: pgmap v1363: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 486 op/s; 27852/149331 objects degraded (18.651%); 16475/149331 objects misplaced (11.033%); 12 MiB/s, 190 objects/s recovering 2024-09-14T04:25:04.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:03 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4115183268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:04.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:03 smithi114 ceph-mon[103029]: osdmap e352: 8 total, 8 up, 5 in 2024-09-14T04:25:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:03 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-14T04:25:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:03 smithi160 ceph-mon[88407]: osdmap e351: 8 total, 8 up, 5 in 2024-09-14T04:25:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:03 smithi160 ceph-mon[88407]: pgmap v1363: 89 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 486 op/s; 27852/149331 objects degraded (18.651%); 16475/149331 objects misplaced (11.033%); 12 MiB/s, 190 objects/s recovering 2024-09-14T04:25:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:03 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4115183268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:03 smithi160 ceph-mon[88407]: osdmap e352: 8 total, 8 up, 5 in 2024-09-14T04:25:04.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:04.195 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:04.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:04.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:04.318 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:04.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:04.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:04.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:04.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:04.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:04.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:04.565 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:04.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:04.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:04.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:04.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:04.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:04.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:04.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:04.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:04.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:04.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:04.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:05.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:05.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:05.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:05.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[103029]: 13.c starting backfill to osd.0 from (0'0,0'0] MAX to 350'8889 2024-09-14T04:25:05.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[103029]: 13.a continuing backfill to osd.3 from (329'1900,350'8738] MIN to 350'8738 2024-09-14T04:25:05.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[103029]: 13.a starting backfill to osd.6 from (0'0,0'0] MAX to 350'8738 2024-09-14T04:25:05.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[103029]: 13.c continuing backfill to osd.4 from (329'2000,350'8889] MIN to 350'8889 2024-09-14T04:25:05.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 27852/149331 objects degraded (18.651%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:05.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[103029]: osdmap e353: 8 total, 8 up, 5 in 2024-09-14T04:25:05.102 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[106969]: 13.c starting backfill to osd.0 from (0'0,0'0] MAX to 350'8889 2024-09-14T04:25:05.102 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[106969]: 13.a continuing backfill to osd.3 from (329'1900,350'8738] MIN to 350'8738 2024-09-14T04:25:05.102 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[106969]: 13.a starting backfill to osd.6 from (0'0,0'0] MAX to 350'8738 2024-09-14T04:25:05.102 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[106969]: 13.c continuing backfill to osd.4 from (329'2000,350'8889] MIN to 350'8889 2024-09-14T04:25:05.102 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 27852/149331 objects degraded (18.651%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:05.102 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:04 smithi114 ceph-mon[106969]: osdmap e353: 8 total, 8 up, 5 in 2024-09-14T04:25:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:04 smithi160 ceph-mon[88407]: 13.c starting backfill to osd.0 from (0'0,0'0] MAX to 350'8889 2024-09-14T04:25:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:04 smithi160 ceph-mon[88407]: 13.a continuing backfill to osd.3 from (329'1900,350'8738] MIN to 350'8738 2024-09-14T04:25:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:04 smithi160 ceph-mon[88407]: 13.a starting backfill to osd.6 from (0'0,0'0] MAX to 350'8738 2024-09-14T04:25:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:04 smithi160 ceph-mon[88407]: 13.c continuing backfill to osd.4 from (329'2000,350'8889] MIN to 350'8889 2024-09-14T04:25:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 27852/149331 objects degraded (18.651%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:05.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:04 smithi160 ceph-mon[88407]: osdmap e353: 8 total, 8 up, 5 in 2024-09-14T04:25:05.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:05.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:05.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:05.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:05.339 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:05.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:05.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:05.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:05.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:05.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:05.596 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:05.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:05.698 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:05.735 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:05.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:05.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:05.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:05.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:05.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:05.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:05.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:05.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:06.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:06.114 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:06.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:06.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[103029]: 13.e starting backfill to osd.0 from (0'0,0'0] MAX to 350'8577 2024-09-14T04:25:06.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[103029]: 13.8 starting backfill to osd.0 from (0'0,0'0] MAX to 350'8785 2024-09-14T04:25:06.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[103029]: 13.1 starting backfill to osd.3 from (0'0,0'0] MAX to 350'9029 2024-09-14T04:25:06.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[103029]: 13.7 starting backfill to osd.5 from (0'0,0'0] MAX to 350'8940 2024-09-14T04:25:06.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[103029]: 13.f starting backfill to osd.4 from (0'0,0'0] MAX to 350'8912 2024-09-14T04:25:06.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[103029]: pgmap v1366: 89 pgs: 1 activating+remapped, 3 remapped+peering, 8 peering, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 69 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 336 op/s; 18827/145326 objects degraded (12.955%); 15649/145326 objects misplaced (10.768%); 9.2 MiB/s, 146 objects/s recovering 2024-09-14T04:25:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[106969]: 13.e starting backfill to osd.0 from (0'0,0'0] MAX to 350'8577 2024-09-14T04:25:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[106969]: 13.8 starting backfill to osd.0 from (0'0,0'0] MAX to 350'8785 2024-09-14T04:25:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[106969]: 13.1 starting backfill to osd.3 from (0'0,0'0] MAX to 350'9029 2024-09-14T04:25:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[106969]: 13.7 starting backfill to osd.5 from (0'0,0'0] MAX to 350'8940 2024-09-14T04:25:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[106969]: 13.f starting backfill to osd.4 from (0'0,0'0] MAX to 350'8912 2024-09-14T04:25:06.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:05 smithi114 ceph-mon[106969]: pgmap v1366: 89 pgs: 1 activating+remapped, 3 remapped+peering, 8 peering, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 69 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 336 op/s; 18827/145326 objects degraded (12.955%); 15649/145326 objects misplaced (10.768%); 9.2 MiB/s, 146 objects/s recovering 2024-09-14T04:25:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:05 smithi160 ceph-mon[88407]: 13.e starting backfill to osd.0 from (0'0,0'0] MAX to 350'8577 2024-09-14T04:25:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:05 smithi160 ceph-mon[88407]: 13.8 starting backfill to osd.0 from (0'0,0'0] MAX to 350'8785 2024-09-14T04:25:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:05 smithi160 ceph-mon[88407]: 13.1 starting backfill to osd.3 from (0'0,0'0] MAX to 350'9029 2024-09-14T04:25:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:05 smithi160 ceph-mon[88407]: 13.7 starting backfill to osd.5 from (0'0,0'0] MAX to 350'8940 2024-09-14T04:25:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:05 smithi160 ceph-mon[88407]: 13.f starting backfill to osd.4 from (0'0,0'0] MAX to 350'8912 2024-09-14T04:25:06.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:05 smithi160 ceph-mon[88407]: pgmap v1366: 89 pgs: 1 activating+remapped, 3 remapped+peering, 8 peering, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 69 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 336 op/s; 18827/145326 objects degraded (12.955%); 15649/145326 objects misplaced (10.768%); 9.2 MiB/s, 146 objects/s recovering 2024-09-14T04:25:06.217 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:06.255 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:06.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:06.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:06.400 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:06.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:06.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:06.526 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:06.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:06.628 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:06.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:06.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:06.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:06.807 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:06.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:06.908 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:06.932 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:06.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:07.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:07.060 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:07.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:07.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:06 smithi114 ceph-mon[106969]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-14T04:25:07.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:06 smithi114 ceph-mon[103029]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-14T04:25:07.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:06 smithi160 ceph-mon[88407]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-14T04:25:07.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:07.190 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:07.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:07.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:07.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:07.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:07.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:07.462 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:07.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:07.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:07.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:07.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:07.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:07.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:07.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:07.832 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:07.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:07.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:07.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:07.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:07.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:08.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:08.120 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:08.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:08.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:08.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:08.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:08.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:08.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:08.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:08.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:08.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:08.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:08.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:08.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:08.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:08.762 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:08.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:08.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:08.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:08.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:25:08.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:08.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:08 smithi114 ceph-mon[106969]: pgmap v1367: 89 pgs: 1 activating+remapped, 3 remapped+peering, 8 peering, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 69 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 313 op/s; 18827/145326 objects degraded (12.955%); 15649/145326 objects misplaced (10.768%); 9.2 MiB/s, 146 objects/s recovering 2024-09-14T04:25:08.930 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:08.930 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:08.930 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:08 smithi114 ceph-mon[103029]: pgmap v1367: 89 pgs: 1 activating+remapped, 3 remapped+peering, 8 peering, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 69 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 313 op/s; 18827/145326 objects degraded (12.955%); 15649/145326 objects misplaced (10.768%); 9.2 MiB/s, 146 objects/s recovering 2024-09-14T04:25:08.930 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:08.930 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:08.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:08.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:09.008 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:25:09.008 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:25:09.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:09.081 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:09.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:08 smithi160 ceph-mon[88407]: pgmap v1367: 89 pgs: 1 activating+remapped, 3 remapped+peering, 8 peering, 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 69 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 313 op/s; 18827/145326 objects degraded (12.955%); 15649/145326 objects misplaced (10.768%); 9.2 MiB/s, 146 objects/s recovering 2024-09-14T04:25:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:09.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:09.229 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:09.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:09.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:09.359 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:09.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:09.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:09.491 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:09.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:09.593 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:09.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:09.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:09.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:09.682 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:25:09.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:09.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:09.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:09.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:09 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1820936891' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:25:09.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:25:09.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 18827/145326 objects degraded (12.955%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:09 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1820936891' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:25:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:25:09.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 18827/145326 objects degraded (12.955%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:09.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:09.913 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:09.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:09.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:09.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-14T04:25:10.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:10.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:10.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:10.131 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 4, 5] out_osds: [2, 1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:25:10.131 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:25:10.131 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-14T04:25:10.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:25:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:09 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1820936891' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:25:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:25:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 18827/145326 objects degraded (12.955%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:10.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:10.208 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:10.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:10.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:10.309 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:10.351 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:10.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:10.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:10.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:10.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:10.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:10.621 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:10.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:25:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:25:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:25:10.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:10.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:10.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:10.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:10.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:10.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:10.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:10 smithi114 ceph-mon[103029]: pgmap v1368: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 501 op/s; 24250/139821 objects degraded (17.344%); 25282/139821 objects misplaced (18.082%); 14 MiB/s, 218 objects/s recovering 2024-09-14T04:25:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:10 smithi114 ceph-mon[103029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-14T04:25:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:10 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:10 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:25:10.990 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:10 smithi114 ceph-mon[103029]: osdmap e354: 8 total, 8 up, 5 in 2024-09-14T04:25:10.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:10 smithi114 ceph-mon[106969]: pgmap v1368: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 501 op/s; 24250/139821 objects degraded (17.344%); 25282/139821 objects misplaced (18.082%); 14 MiB/s, 218 objects/s recovering 2024-09-14T04:25:10.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:10 smithi114 ceph-mon[106969]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-14T04:25:10.990 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:10 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:10.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:10 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:25:10.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:10 smithi114 ceph-mon[106969]: osdmap e354: 8 total, 8 up, 5 in 2024-09-14T04:25:11.012 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:11.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:11.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:11.158 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:11.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:10 smithi160 ceph-mon[88407]: pgmap v1368: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 501 op/s; 24250/139821 objects degraded (17.344%); 25282/139821 objects misplaced (18.082%); 14 MiB/s, 218 objects/s recovering 2024-09-14T04:25:11.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:10 smithi160 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-14T04:25:11.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:10 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:11.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:10 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:25:11.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:10 smithi160 ceph-mon[88407]: osdmap e354: 8 total, 8 up, 5 in 2024-09-14T04:25:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:11.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:11.209 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:11.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:11.291 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:11.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:11.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:11.426 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:11.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:11.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:11.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:11.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:11.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:25:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:25:11] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-14T04:25:11.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:11.693 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:11.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:11.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:11.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:11.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:11.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:11.954 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:11.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:12.004 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:12.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:12.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:12.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:12.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:12.220 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:12.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:12.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:12.349 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:12.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:12.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:12.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:12.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:12.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:12.619 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:12.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:12.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:12.744 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:12.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:12.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:12.846 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:12.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:12.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:12.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:13.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:13.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:13.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:13.032 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:12 smithi114 ceph-mon[103029]: pgmap v1370: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 391 op/s; 24250/139347 objects degraded (17.403%); 25282/139347 objects misplaced (18.143%); 11 MiB/s, 172 objects/s recovering 2024-09-14T04:25:13.033 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:12 smithi114 ceph-mon[106969]: pgmap v1370: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 391 op/s; 24250/139347 objects degraded (17.403%); 25282/139347 objects misplaced (18.143%); 11 MiB/s, 172 objects/s recovering 2024-09-14T04:25:13.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:13.138 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:13.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:12 smithi160 ceph-mon[88407]: pgmap v1370: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 391 op/s; 24250/139347 objects degraded (17.403%); 25282/139347 objects misplaced (18.143%); 11 MiB/s, 172 objects/s recovering 2024-09-14T04:25:13.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:25:13.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:13.263 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:13.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:13.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:13.393 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:13.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:13.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:13.520 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:13.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:13.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:13.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:13.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:13.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:13.787 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:13.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:13.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:13.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:13.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:14.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:14.089 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:14.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:14.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:14.231 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:14.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:14.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:14.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:14.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:14.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:14.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:14.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:14.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:14.637 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:14.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:14.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:14.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:14.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:14.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:14.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:14.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:14.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:14.954 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:14 smithi114 ceph-mon[106969]: pgmap v1371: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 320 op/s; 23547/137337 objects degraded (17.145%); 24818/137337 objects misplaced (18.071%); 11 MiB/s, 169 objects/s recovering 2024-09-14T04:25:14.954 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 23547/137337 objects degraded (17.145%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:14.954 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:14 smithi114 ceph-mon[103029]: pgmap v1371: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 320 op/s; 23547/137337 objects degraded (17.145%); 24818/137337 objects misplaced (18.071%); 11 MiB/s, 169 objects/s recovering 2024-09-14T04:25:14.955 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 23547/137337 objects degraded (17.145%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:15.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:15.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:15.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:15.128 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:25:15.128 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:25:15.154 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:25:15.155 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:25:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:14 smithi160 ceph-mon[88407]: pgmap v1371: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 320 op/s; 23547/137337 objects degraded (17.145%); 24818/137337 objects misplaced (18.071%); 11 MiB/s, 169 objects/s recovering 2024-09-14T04:25:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 23547/137337 objects degraded (17.145%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:15.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:15.180 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:25:15.180 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:25:15.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:15.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:15.203 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:25:15.203 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:25:15.226 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:25:15.227 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:25:15.255 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:25:15.255 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:25:15.284 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:25:15.285 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:25:15.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:15.306 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:25:15.306 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:25:15.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:15.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:15.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:15.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:15.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:15.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:15.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:15.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:15.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:15.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:15.715 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:15.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:15.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:15.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:15.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:15.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/802936568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:15.939 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/802936568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:15.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:15.986 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:15.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:16.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:16.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:16.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/802936568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:16.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:16.254 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:16.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:16.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:16.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:16.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:16.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:16.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:16.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:16.625 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:16.649 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:16.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:16.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:16.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:16.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:16.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:16.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:16.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:17.006 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:16 smithi114 ceph-mon[106969]: pgmap v1372: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 466 op/s; 23033/131325 objects degraded (17.539%); 23625/131325 objects misplaced (17.990%); 13 MiB/s, 210 objects/s recovering 2024-09-14T04:25:17.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:16 smithi114 ceph-mon[103029]: pgmap v1372: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 466 op/s; 23033/131325 objects degraded (17.539%); 23625/131325 objects misplaced (17.990%); 13 MiB/s, 210 objects/s recovering 2024-09-14T04:25:17.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:17.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:17.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:17.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:17.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:16 smithi160 ceph-mon[88407]: pgmap v1372: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 466 op/s; 23033/131325 objects degraded (17.539%); 23625/131325 objects misplaced (17.990%); 13 MiB/s, 210 objects/s recovering 2024-09-14T04:25:17.165 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:17.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:17.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:17.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:17.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:17.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:17.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:17.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:17.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:17.568 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:17.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:17.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:17.670 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:17.704 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:17.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:17.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:17.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:17.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:17.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:17.970 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:17.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:18.048 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:17 smithi114 ceph-mon[106969]: pgmap v1373: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 466 op/s; 23033/131325 objects degraded (17.539%); 23625/131325 objects misplaced (17.990%); 13 MiB/s, 210 objects/s recovering 2024-09-14T04:25:18.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:17 smithi114 ceph-mon[103029]: pgmap v1373: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 466 op/s; 23033/131325 objects degraded (17.539%); 23625/131325 objects misplaced (17.990%); 13 MiB/s, 210 objects/s recovering 2024-09-14T04:25:18.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:18.125 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:18.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:17 smithi160 ceph-mon[88407]: pgmap v1373: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.7 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 466 op/s; 23033/131325 objects degraded (17.539%); 23625/131325 objects misplaced (17.990%); 13 MiB/s, 210 objects/s recovering 2024-09-14T04:25:18.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:18.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:18.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:18.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:18.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:25:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:18.401 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:18.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:18.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:18.538 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:18.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:18.617 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:18.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:18.651 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:25:18.653 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:25:18.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:18.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:18.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:18.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:18.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:18.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:18 smithi114 ceph-mon[103029]: from='client.47015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:25:18.899 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:18 smithi114 ceph-mon[106969]: from='client.47015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:25:18.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:18.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:18.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:19.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:19.069 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:19.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:19.106 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:25:19.106 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:25:19.106 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:25:19.106 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:46612 12.2 12:4a477a48:::benchmark_data_smithi114_253692_object46611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:17:47.222463+0000", 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 451.61395217400002, 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.87429041600000001, 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 46612 2024-09-14T04:25:19.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222463+0000", 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222463+0000", 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.109 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222465+0000", 2024-09-14T04:25:19.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5099999999999999e-06 2024-09-14T04:25:19.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.109 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.109 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.109 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222486+0000", 2024-09-14T04:25:19.109 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1673000000000001e-05 2024-09-14T04:25:19.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222502+0000", 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6079e-05 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222510+0000", 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.17e-06 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222577+0000", 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7605000000000001e-05 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:47.222597+0000", 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9324999999999999e-05 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.092490+0000", 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.86989392600000004 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.092608+0000", 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000117557 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,7", 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.092742+0000", 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013379900000000001 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.094278+0000", 2024-09-14T04:25:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015362329999999999 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.096603+0000", 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023254209999999998 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.096680+0000", 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6534999999999998e-05 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.096718+0000", 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8052000000000001e-05 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:48.096754+0000", 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5531000000000003e-05 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.114 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:119401 12.b 12:d0be7710:::benchmark_data_smithi114_253692_object1422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:57.068009+0000", 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 381.76840638200002, 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.41671702599999999, 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119401 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068009+0000", 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068009+0000", 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068009+0000", 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1600000000000001e-07 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068013+0000", 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2839999999999998e-06 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068020+0000", 2024-09-14T04:25:19.117 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6009999999999997e-06 2024-09-14T04:25:19.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068026+0000", 2024-09-14T04:25:19.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3800000000000002e-06 2024-09-14T04:25:19.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.118 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.118 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.118 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.072027+0000", 2024-09-14T04:25:19.118 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0040008229999999997 2024-09-14T04:25:19.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.119 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.119 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.119 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.072133+0000", 2024-09-14T04:25:19.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010624 2024-09-14T04:25:19.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.120 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-14T04:25:19.120 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.072229+0000", 2024-09-14T04:25:19.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5859999999999994e-05 2024-09-14T04:25:19.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.120 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.083294+0000", 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011065121000000001 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.110875+0000", 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.027580898999999999 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.484597+0000", 2024-09-14T04:25:19.121 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.373722832 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.484664+0000", 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6778000000000003e-05 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.484726+0000", 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1692000000000005e-05 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.122 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:119403 12.b 12:d65c6516:::benchmark_data_smithi114_253692_object1424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:57.068164+0000", 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 381.76825094100002, 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.422416187, 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:25:19.123 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119403 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068164+0000", 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068164+0000", 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068165+0000", 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5600000000000004e-07 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068170+0000", 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.189e-06 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.125 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068181+0000", 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1097000000000001e-05 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.068188+0000", 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0999999999999998e-06 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.075726+0000", 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0075387659999999997 2024-09-14T04:25:19.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.075848+0000", 2024-09-14T04:25:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012163399999999999 2024-09-14T04:25:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-14T04:25:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.075948+0000", 2024-09-14T04:25:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000100103 2024-09-14T04:25:19.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.100687+0000", 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024738779999999998 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.110894+0000", 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010207292999999999 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.128 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.490494+0000", 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.37959958999999999 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.490534+0000", 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0148000000000003e-05 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.490581+0000", 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6530999999999999e-05 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.129 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:119418 12.b 12:de46c377:::benchmark_data_smithi114_253692_object1439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:57.082269+0000", 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 381.75414614599998, 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.40863736000000001, 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.130 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:25:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:25:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119418 2024-09-14T04:25:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.082269+0000", 2024-09-14T04:25:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.082269+0000", 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.082271+0000", 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4419999999999999e-06 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.082275+0000", 2024-09-14T04:25:19.132 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7140000000000001e-06 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.082286+0000", 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0756e-05 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.082291+0000", 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1340000000000003e-06 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.083329+0000", 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010374749999999999 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.083402+0000", 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3299000000000004e-05 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,6", 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.096763+0000", 2024-09-14T04:25:19.134 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013361141999999999 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.106105+0000", 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0093421479999999998 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.111015+0000", 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00490962 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.135 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.490789+0000", 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.37977442 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.490858+0000", 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8862000000000006e-05 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:57.490907+0000", 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8347999999999999e-05 2024-09-14T04:25:19.136 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.137 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.137 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.137 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29700 13.3 13:c2e92b0e:::benchmark_data_smithi114_288325_object29699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.137 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.588118+0000", 2024-09-14T04:25:19.137 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.248297116, 2024-09-14T04:25:19.137 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.074433771, 2024-09-14T04:25:19.137 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.138 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.138 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:25:19.138 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.138 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.138 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.138 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29700 2024-09-14T04:25:19.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.138 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588118+0000", 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588118+0000", 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588120+0000", 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.778e-06 2024-09-14T04:25:19.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588152+0000", 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1919999999999999e-05 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588162+0000", 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0176999999999999e-05 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.140 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588171+0000", 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7609999999999996e-06 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588226+0000", 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5238999999999997e-05 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588236+0000", 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4919999999999999e-06 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.141 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.656606+0000", 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0683703579999999 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.658327+0000", 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017215209999999999 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662499+0000", 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041715069999999996 2024-09-14T04:25:19.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662552+0000", 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3018e-05 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29706 13.3 13:c6337d6e:::benchmark_data_smithi114_288325_object29705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.588194+0000", 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.24822179200001, 2024-09-14T04:25:19.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1080256479999999, 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29706 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588194+0000", 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588194+0000", 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588194+0000", 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0399999999999996e-07 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588214+0000", 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9947999999999999e-05 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588222+0000", 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4610000000000002e-06 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588227+0000", 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6080000000000003e-06 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588261+0000", 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.3414999999999998e-05 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.588267+0000", 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6229999999999999e-06 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.658541+0000", 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.070274111 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.147 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for missing object", 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.658631+0000", 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0186999999999996e-05 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662035+0000", 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0034038279999999998 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for degraded object", 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662093+0000", 2024-09-14T04:25:19.148 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8050999999999997e-05 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.696107+0000", 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.034013623 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.696219+0000", 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011218900000000001 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29726 13.3 13:c1546d62:::benchmark_data_smithi114_288325_object29725:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.596841+0000", 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.239574167, 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.065928867, 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29726 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.150 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.596841+0000", 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.596841+0000", 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.596843+0000", 2024-09-14T04:25:19.151 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.77e-06 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.596872+0000", 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9352999999999998e-05 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.596886+0000", 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3730000000000001e-05 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.596894+0000", 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5429999999999996e-06 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.599052+0000", 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021583499999999999 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.599059+0000", 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.109e-06 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.658672+0000", 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0596125030000001 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.658830+0000", 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000158498 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.658950+0000", 2024-09-14T04:25:19.154 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011965500000000001 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662106+0000", 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031567140000000001 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662155+0000", 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8375999999999999e-05 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.155 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662598+0000", 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00044317899999999998 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662662+0000", 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3992000000000007e-05 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662770+0000", 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000108095 2024-09-14T04:25:19.156 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29728 13.3 13:c38fa3e6:::benchmark_data_smithi114_288325_object29727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.597700+0000", 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.23871497499999, 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0653362040000001, 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.157 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29728 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.597700+0000", 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.597700+0000", 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.597702+0000", 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.341e-06 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.597725+0000", 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3204999999999999e-05 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.597795+0000", 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0531000000000006e-05 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.597804+0000", 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3960000000000006e-06 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.599074+0000", 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0012703390000000001 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.599078+0000", 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4989999999999999e-06 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.659889+0000", 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0608116700000001 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.660038+0000", 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014869800000000001 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.660216+0000", 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017788999999999999 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662463+0000", 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022475749999999999 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662831+0000", 2024-09-14T04:25:19.162 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000367573 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662884+0000", 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3421000000000002e-05 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.662936+0000", 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1966999999999999e-05 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.163 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.663037+0000", 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010009900000000001 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29738 13.3 13:cef14f72:::benchmark_data_smithi114_288325_object29737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.602903+0000", 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.233512426, 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0645606869999999, 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.164 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29738 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.602903+0000", 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.165 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.602903+0000", 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.602905+0000", 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.829e-06 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.602931+0000", 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5797999999999999e-05 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.602950+0000", 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9242e-05 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.602960+0000", 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0189e-05 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.603025+0000", 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4681e-05 2024-09-14T04:25:19.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.603033+0000", 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8949999999999998e-06 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.663124+0000", 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0600917240000001 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.663258+0000", 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013332200000000001 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.663409+0000", 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000151612 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667201+0000", 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003792251 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.169 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667265+0000", 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3545000000000002e-05 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667285+0000", 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9621999999999999e-05 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667343+0000", 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8387999999999997e-05 2024-09-14T04:25:19.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.171 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.171 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.171 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667464+0000", 2024-09-14T04:25:19.171 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012058899999999999 2024-09-14T04:25:19.171 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.171 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.171 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29747 13.3 13:c95eaf80:::benchmark_data_smithi114_288325_object29746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.608627+0000", 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.227788183, 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.059193345, 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.172 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29747 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.608627+0000", 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.608627+0000", 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.173 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.608629+0000", 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.035e-06 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.608657+0000", 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8005999999999999e-05 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.608677+0000", 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9511e-05 2024-09-14T04:25:19.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.608686+0000", 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4609999999999997e-06 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.609646+0000", 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00095958800000000004 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.609657+0000", 2024-09-14T04:25:19.175 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0949e-05 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.664035+0000", 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.05437877 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.664166+0000", 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013089899999999999 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.664327+0000", 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016023799999999999 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667475+0000", 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031487350000000002 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667524+0000", 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8610999999999999e-05 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667628+0000", 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000103818 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667683+0000", 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5028999999999998e-05 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667820+0000", 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013769499999999999 2024-09-14T04:25:19.178 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29751 13.3 13:cf6cb810:::benchmark_data_smithi114_288325_object29750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.610759+0000", 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.225655902, 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.057253649, 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.179 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29751 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.610759+0000", 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.610759+0000", 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.180 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.610761+0000", 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7689999999999999e-06 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.610786+0000", 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4978999999999999e-05 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.610818+0000", 2024-09-14T04:25:19.181 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1571e-05 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.610837+0000", 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9618000000000001e-05 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.610899+0000", 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1247000000000005e-05 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.182 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.610905+0000", 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3239999999999999e-06 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.665225+0000", 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.054319775 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.665357+0000", 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013239400000000001 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.183 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.665492+0000", 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013524000000000001 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667482+0000", 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019895009999999999 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667574+0000", 2024-09-14T04:25:19.184 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1842000000000006e-05 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667880+0000", 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030615599999999997 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667929+0000", 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9327999999999999e-05 2024-09-14T04:25:19.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.668013+0000", 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3905000000000004e-05 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29753 13.3 13:c5edd5ca:::benchmark_data_smithi114_288325_object29752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.611316+0000", 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.225099661, 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.058166948, 2024-09-14T04:25:19.186 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29753 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.611316+0000", 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.611316+0000", 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.611317+0000", 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.04e-06 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.188 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.611334+0000", 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7105e-05 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.611344+0000", 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0484e-05 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.611360+0000", 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5699999999999999e-05 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.189 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.611416+0000", 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5736000000000003e-05 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.611428+0000", 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2699e-05 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.666158+0000", 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.054729442 2024-09-14T04:25:19.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.666281+0000", 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000123356 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.666392+0000", 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011039099999999999 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.667833+0000", 2024-09-14T04:25:19.191 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0014409169999999999 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.668048+0000", 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000215233 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.669338+0000", 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001290469 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.192 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.669389+0000", 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0362000000000003e-05 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.669483+0000", 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4013999999999998e-05 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29779 13.3 13:ce177906:::benchmark_data_smithi114_288325_object29778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.193 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.623976+0000", 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.21243964799999, 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0479113440000001, 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29779 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.194 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.623976+0000", 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.623976+0000", 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.623977+0000", 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6700000000000001e-06 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.195 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.623995+0000", 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7685000000000001e-05 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.624013+0000", 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7852000000000001e-05 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.624019+0000", 2024-09-14T04:25:19.196 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8930000000000002e-06 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.624082+0000", 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2830000000000007e-05 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.624089+0000", 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8499999999999994e-06 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.197 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.668074+0000", 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0439843529999999 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.668190+0000", 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011614 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.668329+0000", 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013857 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.198 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.670604+0000", 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022758219999999998 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.671699+0000", 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0010944069999999999 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.671759+0000", 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0149000000000003e-05 2024-09-14T04:25:19.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.671816+0000", 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7309000000000003e-05 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.671887+0000", 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0814000000000003e-05 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.200 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29800 13.3 13:cc0f5a7b:::benchmark_data_smithi114_288325_object29799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.636445+0000", 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.19997073299999, 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.037955438, 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29800 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.201 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.636445+0000", 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.636445+0000", 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.202 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.636447+0000", 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3980000000000002e-06 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.636471+0000", 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.444e-05 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.636490+0000", 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8369e-05 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.203 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.636499+0000", 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.721e-06 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.636562+0000", 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3419999999999999e-05 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.636572+0000", 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7790000000000007e-06 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.204 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.669511+0000", 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0329388779999999 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.669641+0000", 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013016300000000001 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.205 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.669795+0000", 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015440300000000001 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.671970+0000", 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002175293 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.671989+0000", 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.806e-05 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.206 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.674302+0000", 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023137320000000002 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.674338+0000", 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5969000000000001e-05 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.674400+0000", 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1812999999999999e-05 2024-09-14T04:25:19.207 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29822 13.3 13:cbd0b8c3:::benchmark_data_smithi114_288325_object29821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.652796+0000", 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.18361955099999, 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0246972969999999, 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.208 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29822 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.652796+0000", 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.652796+0000", 2024-09-14T04:25:19.209 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.652797+0000", 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.629e-06 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.652827+0000", 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9167e-05 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.210 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.652839+0000", 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2269000000000001e-05 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.652846+0000", 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3749999999999997e-06 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.652909+0000", 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3007000000000002e-05 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.652920+0000", 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0458e-05 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.670657+0000", 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0177376650000001 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.670791+0000", 2024-09-14T04:25:19.212 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000133984 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.670937+0000", 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000145616 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.674246+0000", 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033089220000000002 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.213 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.674262+0000", 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5965e-05 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677319+0000", 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0030574399999999998 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677384+0000", 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4369999999999995e-05 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.214 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677493+0000", 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010943 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29884 13.3 13:ceea81d9:::benchmark_data_smithi114_288325_object29883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.689570+0000", 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.14684507800001, 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98838191200000003, 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.215 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29884 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.689570+0000", 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.216 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.689570+0000", 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.689573+0000", 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3599999999999999e-06 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.689601+0000", 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8229999999999999e-05 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.217 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.689619+0000", 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8260000000000001e-05 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.689628+0000", 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3370000000000006e-06 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.689708+0000", 2024-09-14T04:25:19.218 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.928e-05 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.689724+0000", 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6137000000000002e-05 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.672033+0000", 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98230961699999997 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.219 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.672169+0000", 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000135043 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.672314+0000", 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014565100000000001 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.675987+0000", 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0036723960000000001 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.220 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677547+0000", 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015602420000000001 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677799+0000", 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025193200000000001 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677860+0000", 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1626999999999998e-05 2024-09-14T04:25:19.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677952+0000", 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1799999999999995e-05 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29887 13.3 13:c2080111:::benchmark_data_smithi114_288325_object29886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.691274+0000", 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.145141182, 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98685777500000005, 2024-09-14T04:25:19.222 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29887 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.691274+0000", 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.691274+0000", 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.691276+0000", 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0159999999999998e-06 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.691304+0000", 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7591000000000001e-05 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.691326+0000", 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2288000000000001e-05 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.691336+0000", 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0135999999999999e-05 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.691411+0000", 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5258000000000003e-05 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.691423+0000", 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1158000000000001e-05 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.673170+0000", 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.98174735099999999 2024-09-14T04:25:19.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.673292+0000", 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000121631 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.673428+0000", 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013684400000000001 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677615+0000", 2024-09-14T04:25:19.227 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041863669999999999 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677632+0000", 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7144000000000001e-05 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.678001+0000", 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00036913199999999999 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.228 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.678049+0000", 2024-09-14T04:25:19.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8062000000000002e-05 2024-09-14T04:25:19.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.229 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.229 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.678132+0000", 2024-09-14T04:25:19.229 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.2797e-05 2024-09-14T04:25:19.229 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.229 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.229 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29909 13.3 13:caf41169:::benchmark_data_smithi114_288325_object29908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.708892+0000", 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.127523365, 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.97230001399999999, 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.230 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29909 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.708892+0000", 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.708892+0000", 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.708894+0000", 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5689999999999999e-06 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.708916+0000", 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2546999999999999e-05 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.708931+0000", 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4459e-05 2024-09-14T04:25:19.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.708938+0000", 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.892e-06 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.708989+0000", 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0439999999999998e-05 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.233 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.708996+0000", 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6299999999999998e-06 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.674450+0000", 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.96545368200000004 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.674610+0000", 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015994300000000001 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.675006+0000", 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000395623 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677754+0000", 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002748037 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677964+0000", 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021014100000000001 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.681047+0000", 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003082998 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.681102+0000", 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.524e-05 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.681192+0000", 2024-09-14T04:25:19.236 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9813000000000002e-05 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29925 13.3 13:c76219dd:::benchmark_data_smithi114_288325_object29924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.725318+0000", 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.111097524, 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.95606621599999997, 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.237 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29925 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.725318+0000", 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.725318+0000", 2024-09-14T04:25:19.238 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.725319+0000", 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6330000000000001e-06 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.725338+0000", 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8782000000000001e-05 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.239 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.725351+0000", 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2799999999999999e-05 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.725357+0000", 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1090000000000003e-06 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.725445+0000", 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7540999999999994e-05 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.240 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.725456+0000", 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0906e-05 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.676031+0000", 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.95057581700000005 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.676181+0000", 2024-09-14T04:25:19.241 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014963199999999999 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.676372+0000", 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019138299999999999 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.677969+0000", 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015965549999999999 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.680997+0000", 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00302849 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.681239+0000", 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000241444 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.681294+0000", 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5231e-05 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.243 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.681384+0000", 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.9893000000000001e-05 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:29928 13.3 13:c9232889:::benchmark_data_smithi114_288325_object29927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:22.728839+0000", 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 176.10757599900001, 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.95435669999999995, 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:25:19.244 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 29928 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.728839+0000", 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.728839+0000", 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.728841+0000", 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4109999999999999e-06 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.728861+0000", 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9908e-05 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.728871+0000", 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0529e-05 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.728877+0000", 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4770000000000002e-06 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.728929+0000", 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2547999999999999e-05 2024-09-14T04:25:19.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:22.728937+0000", 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4599999999999997e-06 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.678162+0000", 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.94922558499999998 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.678299+0000", 2024-09-14T04:25:19.248 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013630900000000001 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 0,4", 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.678467+0000", 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016819099999999999 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.681010+0000", 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002543727 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.249 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.681496+0000", 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00048558299999999999 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.683045+0000", 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015487770000000001 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.683097+0000", 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2556000000000003e-05 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:25:19.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:25:19.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:23.683196+0000", 2024-09-14T04:25:19.251 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8639000000000002e-05 2024-09-14T04:25:19.251 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.251 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.251 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.251 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:25:19.251 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:25:19.251 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:25:19.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:19.252 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:18 smithi160 ceph-mon[88407]: from='client.47015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:25:19.274 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:19.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:19.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:19.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:19.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:19.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:19.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:19.573 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:19.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:19.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:19.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:19.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:19.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:19.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:19.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:19.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:19.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:19 smithi114 ceph-mon[106969]: pgmap v1374: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 537 op/s; 21812/123687 objects degraded (17.635%); 22153/123687 objects misplaced (17.911%); 14 MiB/s, 238 objects/s recovering 2024-09-14T04:25:19.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 23033/131325 objects degraded (17.539%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:19.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:19 smithi114 ceph-mon[103029]: pgmap v1374: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 537 op/s; 21812/123687 objects degraded (17.635%); 22153/123687 objects misplaced (17.911%); 14 MiB/s, 238 objects/s recovering 2024-09-14T04:25:19.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 23033/131325 objects degraded (17.539%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:19.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:19.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:19.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:20.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:20.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:20.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:20.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:19 smithi160 ceph-mon[88407]: pgmap v1374: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 537 op/s; 21812/123687 objects degraded (17.635%); 22153/123687 objects misplaced (17.911%); 14 MiB/s, 238 objects/s recovering 2024-09-14T04:25:20.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 23033/131325 objects degraded (17.539%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:20.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:20.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:20.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:20.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-14T04:25:20.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:20.421 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:20.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:20.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:20.584 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:20.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:25:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:25:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:25:20.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:20.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:20.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:20.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:20.788 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:25:20.812 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:20.885 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:20.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:20.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:21.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:21.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:21.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:21.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:21.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:21.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:21.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:21.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:21.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:21.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:21.468 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1373,"stamp":"2024-09-14T04:25:17.147537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2867664869,"num_objects":43775,"num_object_clones":0,"num_object_copies":131325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23033,"num_objects_misplaced":23625,"num_objects_unfound":0,"num_objects_dirty":43775,"num_whiteouts":0,"num_read":293,"num_read_kb":748,"num_write":146589,"num_write_kb":6085628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10973,"num_bytes_recovered":716638205,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93791,"ondisk_log_size":93791,"up":267,"acting":264,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":11275592,"kb_used_data":6112032,"kb_used_omap":19,"kb_used_meta":5163308,"kb_avail":457450168,"statfs":{"total":479975178240,"available":468428972032,"internally_reserved":0,"allocated":6258720768,"data_stored":6250501859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20312,"internal_metadata":5287227560},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-305856512,"num_objects":-4667,"num_object_clones":0,"num_object_copies":-14001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4206,"num_objects_misplaced":7976,"num_objects_unfound":0,"num_objects_dirty":-4667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2102,"num_bytes_recovered":137494550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002898"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":536,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692162+0000","last_change":"2024-09-14T04:25:03.993352+0000","last_active":"2024-09-14T04:25:09.692162+0000","last_peered":"2024-09-14T04:25:09.692162+0000","last_clean":"2024-09-14T04:25:09.692162+0000","last_became_active":"2024-09-14T04:25:03.993134+0000","last_became_peered":"2024-09-14T04:25:03.993134+0000","last_unstale":"2024-09-14T04:25:09.692162+0000","last_undegraded":"2024-09-14T04:25:09.692162+0000","last_fullsized":"2024-09-14T04:25:09.692162+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:02:25.184180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":345,"reported_epoch":353,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-14T04:25:04.833901+0000","last_change":"2024-09-14T04:25:03.845207+0000","last_active":"2024-09-14T04:25:04.833901+0000","last_peered":"2024-09-14T04:25:04.833901+0000","last_clean":"2024-09-14T04:22:32.391862+0000","last_became_active":"2024-09-14T04:25:03.844974+0000","last_became_peered":"2024-09-14T04:25:03.844974+0000","last_unstale":"2024-09-14T04:25:04.833901+0000","last_undegraded":"2024-09-14T04:25:03.829324+0000","last_fullsized":"2024-09-14T04:25:04.833901+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:06:30.968032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":2}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":49,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.829288+0000","last_change":"2024-09-14T04:25:03.863433+0000","last_active":"2024-09-14T04:25:04.829288+0000","last_peered":"2024-09-14T04:25:04.829288+0000","last_clean":"2024-09-14T04:25:04.829288+0000","last_became_active":"2024-09-14T04:25:03.863153+0000","last_became_peered":"2024-09-14T04:25:03.863153+0000","last_unstale":"2024-09-14T04:25:04.829288+0000","last_undegraded":"2024-09-14T04:25:04.829288+0000","last_fullsized":"2024-09-14T04:25:04.829288+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:36:30.030688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":344,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.693884+0000","last_change":"2024-09-14T04:25:03.857103+0000","last_active":"2024-09-14T04:25:09.693884+0000","last_peered":"2024-09-14T04:25:09.693884+0000","last_clean":"2024-09-14T04:25:09.693884+0000","last_became_active":"2024-09-14T04:25:03.856394+0000","last_became_peered":"2024-09-14T04:25:03.856394+0000","last_unstale":"2024-09-14T04:25:09.693884+0000","last_undegraded":"2024-09-14T04:25:09.693884+0000","last_fullsized":"2024-09-14T04:25:09.693884+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:57:36.649422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":208,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692605+0000","last_change":"2024-09-14T04:23:09.283577+0000","last_active":"2024-09-14T04:25:09.692605+0000","last_peered":"2024-09-14T04:25:09.692605+0000","last_clean":"2024-09-14T04:25:09.692605+0000","last_became_active":"2024-09-14T04:23:09.283289+0000","last_became_peered":"2024-09-14T04:23:09.283289+0000","last_unstale":"2024-09-14T04:25:09.692605+0000","last_undegraded":"2024-09-14T04:25:09.692605+0000","last_fullsized":"2024-09-14T04:25:09.692605+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:30:21.593844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":326,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.825603+0000","last_change":"2024-09-14T04:25:03.850982+0000","last_active":"2024-09-14T04:25:04.825603+0000","last_peered":"2024-09-14T04:25:04.825603+0000","last_clean":"2024-09-14T04:25:04.825603+0000","last_became_active":"2024-09-14T04:25:03.850623+0000","last_became_peered":"2024-09-14T04:25:03.850623+0000","last_unstale":"2024-09-14T04:25:04.825603+0000","last_undegraded":"2024-09-14T04:25:04.825603+0000","last_fullsized":"2024-09-14T04:25:04.825603+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:01:05.260156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":349,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692473+0000","last_change":"2024-09-14T04:24:47.620166+0000","last_active":"2024-09-14T04:25:09.692473+0000","last_peered":"2024-09-14T04:25:09.692473+0000","last_clean":"2024-09-14T04:25:09.692473+0000","last_became_active":"2024-09-14T04:24:47.619973+0000","last_became_peered":"2024-09-14T04:24:47.619973+0000","last_unstale":"2024-09-14T04:25:09.692473+0000","last_undegraded":"2024-09-14T04:25:09.692473+0000","last_fullsized":"2024-09-14T04:25:09.692473+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:52:36.038585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":368,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.832490+0000","last_change":"2024-09-14T04:22:08.477318+0000","last_active":"2024-09-14T04:25:04.832490+0000","last_peered":"2024-09-14T04:25:04.832490+0000","last_clean":"2024-09-14T04:25:04.832490+0000","last_became_active":"2024-09-14T04:22:08.467703+0000","last_became_peered":"2024-09-14T04:22:08.467703+0000","last_unstale":"2024-09-14T04:25:04.832490+0000","last_undegraded":"2024-09-14T04:25:04.832490+0000","last_fullsized":"2024-09-14T04:25:04.832490+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:26:43.870476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"354'9360","reported_seq":16427,"reported_epoch":354,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-14T04:25:13.734018+0000","last_change":"2024-09-14T04:25:05.057233+0000","last_active":"2024-09-14T04:25:13.734018+0000","last_peered":"2024-09-14T04:25:13.734018+0000","last_clean":"2024-09-14T04:25:02.769458+0000","last_became_active":"2024-09-14T04:25:05.015517+0000","last_became_peered":"2024-09-14T04:25:05.015517+0000","last_unstale":"2024-09-14T04:25:13.734018+0000","last_undegraded":"2024-09-14T04:25:13.734018+0000","last_fullsized":"2024-09-14T04:25:13.734018+0000","mapping_epoch":352,"log_start":"334'3560","ondisk_log_start":"334'3560","created":321,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:30:47.763794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184549376,"num_objects":2816,"num_object_clones":0,"num_object_copies":8448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2816,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9360,"num_write_kb":389632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3498,"num_bytes_recovered":229244928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,7],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":2816}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":352,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.825878+0000","last_change":"2024-09-14T04:25:03.877723+0000","last_active":"2024-09-14T04:25:04.825878+0000","last_peered":"2024-09-14T04:25:04.825878+0000","last_clean":"2024-09-14T04:25:04.825878+0000","last_became_active":"2024-09-14T04:25:03.853683+0000","last_became_peered":"2024-09-14T04:25:03.853683+0000","last_unstale":"2024-09-14T04:25:04.825878+0000","last_undegraded":"2024-09-14T04:25:04.825878+0000","last_fullsized":"2024-09-14T04:25:04.825878+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:09:16.521799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":49,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.827721+0000","last_change":"2024-09-14T04:23:09.282698+0000","last_active":"2024-09-14T04:25:04.827721+0000","last_peered":"2024-09-14T04:25:04.827721+0000","last_clean":"2024-09-14T04:25:04.827721+0000","last_became_active":"2024-09-14T04:23:09.282181+0000","last_became_peered":"2024-09-14T04:23:09.282181+0000","last_unstale":"2024-09-14T04:25:04.827721+0000","last_undegraded":"2024-09-14T04:25:04.827721+0000","last_fullsized":"2024-09-14T04:25:04.827721+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:18:12.880523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.0","version":"354'8997","reported_seq":17058,"reported_epoch":354,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-14T04:25:12.595616+0000","last_change":"2024-09-14T04:25:03.884521+0000","last_active":"2024-09-14T04:25:12.595616+0000","last_peered":"2024-09-14T04:25:12.595616+0000","last_clean":"2024-09-14T04:22:39.594820+0000","last_became_active":"2024-09-14T04:24:47.746246+0000","last_became_peered":"2024-09-14T04:24:47.746246+0000","last_unstale":"2024-09-14T04:25:12.595616+0000","last_undegraded":"2024-09-14T04:25:03.882829+0000","last_fullsized":"2024-09-14T04:24:47.598892+0000","mapping_epoch":347,"log_start":"329'1900","ondisk_log_start":"329'1900","created":321,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":7097,"log_dups_size":0,"ondisk_log_size":7097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180420608,"num_objects":2753,"num_object_clones":0,"num_object_copies":8259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3008,"num_objects_misplaced":2832,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8997,"num_write_kb":376000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":204668928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[1,6],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":255},{"shards":"1,6","objects":1324},{"shards":"1,6,7","objects":1174}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":16,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830792+0000","last_change":"2024-09-14T04:25:03.874868+0000","last_active":"2024-09-14T04:25:04.830792+0000","last_peered":"2024-09-14T04:25:04.830792+0000","last_clean":"2024-09-14T04:25:04.830792+0000","last_became_active":"2024-09-14T04:25:03.857739+0000","last_became_peered":"2024-09-14T04:25:03.857739+0000","last_unstale":"2024-09-14T04:25:04.830792+0000","last_undegraded":"2024-09-14T04:25:04.830792+0000","last_fullsized":"2024-09-14T04:25:04.830792+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:42:21.991253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":325,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692056+0000","last_change":"2024-09-14T04:24:47.613225+0000","last_active":"2024-09-14T04:25:09.692056+0000","last_peered":"2024-09-14T04:25:09.692056+0000","last_clean":"2024-09-14T04:25:09.692056+0000","last_became_active":"2024-09-14T04:24:47.612805+0000","last_became_peered":"2024-09-14T04:24:47.612805+0000","last_unstale":"2024-09-14T04:25:09.692056+0000","last_undegraded":"2024-09-14T04:25:09.692056+0000","last_fullsized":"2024-09-14T04:25:09.692056+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:35.992716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.3","version":"354'9192","reported_seq":9284,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:14.465678+0000","last_change":"2024-09-14T04:22:23.698482+0000","last_active":"2024-09-14T04:25:14.465678+0000","last_peered":"2024-09-14T04:25:14.465678+0000","last_clean":"2024-09-14T04:25:14.465678+0000","last_became_active":"2024-09-14T04:22:23.655107+0000","last_became_peered":"2024-09-14T04:22:23.655107+0000","last_unstale":"2024-09-14T04:25:14.465678+0000","last_undegraded":"2024-09-14T04:25:14.465678+0000","last_fullsized":"2024-09-14T04:25:14.465678+0000","mapping_epoch":328,"log_start":"330'3387","ondisk_log_start":"330'3387","created":321,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:06.322461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180092928,"num_objects":2748,"num_object_clones":0,"num_object_copies":8244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9192,"num_write_kb":382080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":57,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.829129+0000","last_change":"2024-09-14T04:25:03.878325+0000","last_active":"2024-09-14T04:25:04.829129+0000","last_peered":"2024-09-14T04:25:04.829129+0000","last_clean":"2024-09-14T04:25:04.829129+0000","last_became_active":"2024-09-14T04:25:03.861286+0000","last_became_peered":"2024-09-14T04:25:03.861286+0000","last_unstale":"2024-09-14T04:25:04.829129+0000","last_undegraded":"2024-09-14T04:25:04.829129+0000","last_fullsized":"2024-09-14T04:25:04.829129+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:57:29.738051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":359,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830384+0000","last_change":"2024-09-14T04:25:03.843516+0000","last_active":"2024-09-14T04:25:04.830384+0000","last_peered":"2024-09-14T04:25:04.830384+0000","last_clean":"2024-09-14T04:25:04.830384+0000","last_became_active":"2024-09-14T04:25:03.843025+0000","last_became_peered":"2024-09-14T04:25:03.843025+0000","last_unstale":"2024-09-14T04:25:04.830384+0000","last_undegraded":"2024-09-14T04:25:04.830384+0000","last_fullsized":"2024-09-14T04:25:04.830384+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:39:22.348710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"354'9209","reported_seq":9248,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:13.979413+0000","last_change":"2024-09-14T04:22:08.468483+0000","last_active":"2024-09-14T04:25:13.979413+0000","last_peered":"2024-09-14T04:25:13.979413+0000","last_clean":"2024-09-14T04:25:13.979413+0000","last_became_active":"2024-09-14T04:22:08.467808+0000","last_became_peered":"2024-09-14T04:22:08.467808+0000","last_unstale":"2024-09-14T04:25:13.979413+0000","last_undegraded":"2024-09-14T04:25:13.979413+0000","last_fullsized":"2024-09-14T04:25:13.979413+0000","mapping_epoch":324,"log_start":"334'3648","ondisk_log_start":"334'3648","created":321,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5561,"log_dups_size":0,"ondisk_log_size":5561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:22.761402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173604864,"num_objects":2649,"num_object_clones":0,"num_object_copies":7947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9209,"num_write_kb":379456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":39,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.831089+0000","last_change":"2024-09-14T04:22:08.468750+0000","last_active":"2024-09-14T04:25:04.831089+0000","last_peered":"2024-09-14T04:25:04.831089+0000","last_clean":"2024-09-14T04:25:04.831089+0000","last_became_active":"2024-09-14T04:22:08.468217+0000","last_became_peered":"2024-09-14T04:22:08.468217+0000","last_unstale":"2024-09-14T04:25:04.831089+0000","last_undegraded":"2024-09-14T04:25:04.831089+0000","last_fullsized":"2024-09-14T04:25:04.831089+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:06.709182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692779+0000","last_change":"2024-09-14T04:24:47.614937+0000","last_active":"2024-09-14T04:25:09.692779+0000","last_peered":"2024-09-14T04:25:09.692779+0000","last_clean":"2024-09-14T04:25:09.692779+0000","last_became_active":"2024-09-14T04:24:47.614621+0000","last_became_peered":"2024-09-14T04:24:47.614621+0000","last_unstale":"2024-09-14T04:25:09.692779+0000","last_undegraded":"2024-09-14T04:25:09.692779+0000","last_fullsized":"2024-09-14T04:25:09.692779+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:58:52.546923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.5","version":"354'8927","reported_seq":8978,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:12.303903+0000","last_change":"2024-09-14T04:22:08.465521+0000","last_active":"2024-09-14T04:25:12.303903+0000","last_peered":"2024-09-14T04:25:12.303903+0000","last_clean":"2024-09-14T04:25:12.303903+0000","last_became_active":"2024-09-14T04:22:08.465225+0000","last_became_peered":"2024-09-14T04:22:08.465225+0000","last_unstale":"2024-09-14T04:25:12.303903+0000","last_undegraded":"2024-09-14T04:25:12.303903+0000","last_fullsized":"2024-09-14T04:25:12.303903+0000","mapping_epoch":324,"log_start":"334'3443","ondisk_log_start":"334'3443","created":321,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5484,"log_dups_size":0,"ondisk_log_size":5484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:29:28.137105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178454528,"num_objects":2723,"num_object_clones":0,"num_object_copies":8169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8927,"num_write_kb":372800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":572,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.693124+0000","last_change":"2024-09-14T04:13:37.173224+0000","last_active":"2024-09-14T04:25:09.693124+0000","last_peered":"2024-09-14T04:25:09.693124+0000","last_clean":"2024-09-14T04:25:09.693124+0000","last_became_active":"2024-09-14T04:13:37.158630+0000","last_became_peered":"2024-09-14T04:13:37.158630+0000","last_unstale":"2024-09-14T04:25:09.693124+0000","last_undegraded":"2024-09-14T04:25:09.693124+0000","last_fullsized":"2024-09-14T04:25:09.693124+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:41:25.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":409,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692239+0000","last_change":"2024-09-14T04:17:48.083765+0000","last_active":"2024-09-14T04:25:09.692239+0000","last_peered":"2024-09-14T04:25:09.692239+0000","last_clean":"2024-09-14T04:25:09.692239+0000","last_became_active":"2024-09-14T04:17:48.083385+0000","last_became_peered":"2024-09-14T04:17:48.083385+0000","last_unstale":"2024-09-14T04:25:09.692239+0000","last_undegraded":"2024-09-14T04:25:09.692239+0000","last_fullsized":"2024-09-14T04:25:09.692239+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:13:49.268399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.4","version":"354'9229","reported_seq":11073,"reported_epoch":354,"state":"active+recovering+remapped","last_fresh":"2024-09-14T04:25:13.836061+0000","last_change":"2024-09-14T04:25:04.048463+0000","last_active":"2024-09-14T04:25:13.836061+0000","last_peered":"2024-09-14T04:25:13.836061+0000","last_clean":"2024-09-14T04:22:39.672198+0000","last_became_active":"2024-09-14T04:24:48.797694+0000","last_became_peered":"2024-09-14T04:24:48.797694+0000","last_unstale":"2024-09-14T04:25:13.836061+0000","last_undegraded":"2024-09-14T04:25:13.836061+0000","last_fullsized":"2024-09-14T04:25:13.836061+0000","mapping_epoch":348,"log_start":"330'3363","ondisk_log_start":"330'3363","created":321,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5866,"log_dups_size":0,"ondisk_log_size":5866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:20:06.800386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179372032,"num_objects":2737,"num_object_clones":0,"num_object_copies":8211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":74,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9229,"num_write_kb":382912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":58195968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,1],"avail_no_missing":["6","1","2"],"object_location_counts":[{"shards":"1,2,6","objects":37},{"shards":"1,5,6","objects":2700}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":350,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830524+0000","last_change":"2024-09-14T04:24:47.615423+0000","last_active":"2024-09-14T04:25:04.830524+0000","last_peered":"2024-09-14T04:25:04.830524+0000","last_clean":"2024-09-14T04:25:04.830524+0000","last_became_active":"2024-09-14T04:24:47.609944+0000","last_became_peered":"2024-09-14T04:24:47.609944+0000","last_unstale":"2024-09-14T04:25:04.830524+0000","last_undegraded":"2024-09-14T04:25:04.830524+0000","last_fullsized":"2024-09-14T04:25:04.830524+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:30:03.475490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":281,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830289+0000","last_change":"2024-09-14T04:24:47.611089+0000","last_active":"2024-09-14T04:25:04.830289+0000","last_peered":"2024-09-14T04:25:04.830289+0000","last_clean":"2024-09-14T04:25:04.830289+0000","last_became_active":"2024-09-14T04:24:47.610787+0000","last_became_peered":"2024-09-14T04:24:47.610787+0000","last_unstale":"2024-09-14T04:25:04.830289+0000","last_undegraded":"2024-09-14T04:25:04.830289+0000","last_fullsized":"2024-09-14T04:25:04.830289+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:39:19.067086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"354'9214","reported_seq":9265,"reported_epoch":354,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-14T04:25:13.754867+0000","last_change":"2024-09-14T04:25:05.010848+0000","last_active":"2024-09-14T04:25:13.754867+0000","last_peered":"2024-09-14T04:25:13.754867+0000","last_clean":"2024-09-14T04:25:02.695153+0000","last_became_active":"2024-09-14T04:25:04.985047+0000","last_became_peered":"2024-09-14T04:25:04.985047+0000","last_unstale":"2024-09-14T04:25:13.754867+0000","last_undegraded":"2024-09-14T04:25:13.754867+0000","last_fullsized":"2024-09-14T04:25:13.754867+0000","mapping_epoch":352,"log_start":"330'3371","ondisk_log_start":"330'3371","created":321,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5843,"log_dups_size":0,"ondisk_log_size":5843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:21:10.505795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181796864,"num_objects":2774,"num_object_clones":0,"num_object_copies":8322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2774,"num_objects_unfound":0,"num_objects_dirty":2774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":383616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,7],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":2774}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":142,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.825558+0000","last_change":"2024-09-14T04:25:03.865707+0000","last_active":"2024-09-14T04:25:04.825558+0000","last_peered":"2024-09-14T04:25:04.825558+0000","last_clean":"2024-09-14T04:25:04.825558+0000","last_became_active":"2024-09-14T04:25:03.865070+0000","last_became_peered":"2024-09-14T04:25:03.865070+0000","last_unstale":"2024-09-14T04:25:04.825558+0000","last_undegraded":"2024-09-14T04:25:04.825558+0000","last_fullsized":"2024-09-14T04:25:04.825558+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:54:15.436222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":232,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830356+0000","last_change":"2024-09-14T04:24:47.616564+0000","last_active":"2024-09-14T04:25:04.830356+0000","last_peered":"2024-09-14T04:25:04.830356+0000","last_clean":"2024-09-14T04:25:04.830356+0000","last_became_active":"2024-09-14T04:24:47.616196+0000","last_became_peered":"2024-09-14T04:24:47.616196+0000","last_unstale":"2024-09-14T04:25:04.830356+0000","last_undegraded":"2024-09-14T04:25:04.830356+0000","last_fullsized":"2024-09-14T04:25:04.830356+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:38:56.531650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.6","version":"354'9323","reported_seq":9362,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:13.953364+0000","last_change":"2024-09-14T04:22:08.468486+0000","last_active":"2024-09-14T04:25:13.953364+0000","last_peered":"2024-09-14T04:25:13.953364+0000","last_clean":"2024-09-14T04:25:13.953364+0000","last_became_active":"2024-09-14T04:22:08.468193+0000","last_became_peered":"2024-09-14T04:22:08.468193+0000","last_unstale":"2024-09-14T04:25:13.953364+0000","last_undegraded":"2024-09-14T04:25:13.953364+0000","last_fullsized":"2024-09-14T04:25:13.953364+0000","mapping_epoch":324,"log_start":"334'3753","ondisk_log_start":"334'3753","created":321,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:00:41.999518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182386688,"num_objects":2783,"num_object_clones":0,"num_object_copies":8349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9323,"num_write_kb":387392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":350,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.691718+0000","last_change":"2024-09-14T04:25:03.848027+0000","last_active":"2024-09-14T04:25:09.691718+0000","last_peered":"2024-09-14T04:25:09.691718+0000","last_clean":"2024-09-14T04:25:09.691718+0000","last_became_active":"2024-09-14T04:25:03.847621+0000","last_became_peered":"2024-09-14T04:25:03.847621+0000","last_unstale":"2024-09-14T04:25:09.691718+0000","last_undegraded":"2024-09-14T04:25:09.691718+0000","last_fullsized":"2024-09-14T04:25:09.691718+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:52.585026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":385,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692549+0000","last_change":"2024-09-14T04:25:03.856805+0000","last_active":"2024-09-14T04:25:09.692549+0000","last_peered":"2024-09-14T04:25:09.692549+0000","last_clean":"2024-09-14T04:25:09.692549+0000","last_became_active":"2024-09-14T04:25:03.856459+0000","last_became_peered":"2024-09-14T04:25:03.856459+0000","last_unstale":"2024-09-14T04:25:09.692549+0000","last_undegraded":"2024-09-14T04:25:09.692549+0000","last_fullsized":"2024-09-14T04:25:09.692549+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:17:41.568556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.c","version":"354'9172","reported_seq":10677,"reported_epoch":354,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-14T04:25:14.013620+0000","last_change":"2024-09-14T04:25:04.045133+0000","last_active":"2024-09-14T04:25:14.013620+0000","last_peered":"2024-09-14T04:25:14.013620+0000","last_clean":"2024-09-14T04:22:39.603599+0000","last_became_active":"2024-09-14T04:25:03.993653+0000","last_became_peered":"2024-09-14T04:25:03.993653+0000","last_unstale":"2024-09-14T04:25:14.013620+0000","last_undegraded":"2024-09-14T04:25:03.989926+0000","last_fullsized":"2024-09-14T04:25:14.013620+0000","mapping_epoch":351,"log_start":"334'3534","ondisk_log_start":"334'3534","created":321,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5638,"log_dups_size":0,"ondisk_log_size":5638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:48:06.292841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177733632,"num_objects":2712,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2712,"num_objects_misplaced":2712,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":380288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":704,"num_bytes_recovered":46137344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,7,1],"avail_no_missing":["5","1","2"],"object_location_counts":[{"shards":"1,2,5","objects":209},{"shards":"1,5,7","objects":2503}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":186,"reported_epoch":353,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-14T04:25:04.832318+0000","last_change":"2024-09-14T04:25:03.845391+0000","last_active":"2024-09-14T04:25:04.832318+0000","last_peered":"2024-09-14T04:25:04.832318+0000","last_clean":"2024-09-14T04:24:53.687419+0000","last_became_active":"2024-09-14T04:25:03.845126+0000","last_became_peered":"2024-09-14T04:25:03.845126+0000","last_unstale":"2024-09-14T04:25:04.832318+0000","last_undegraded":"2024-09-14T04:25:03.827592+0000","last_fullsized":"2024-09-14T04:25:04.832318+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:03:03.220358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":["6","0","7"],"object_location_counts":[{"shards":"0,6,7","objects":2}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":65,"reported_epoch":353,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-14T04:25:04.828450+0000","last_change":"2024-09-14T04:25:03.859057+0000","last_active":"2024-09-14T04:25:04.828450+0000","last_peered":"2024-09-14T04:25:04.828450+0000","last_clean":"2024-09-14T04:24:53.685610+0000","last_became_active":"2024-09-14T04:25:03.858806+0000","last_became_peered":"2024-09-14T04:25:03.858806+0000","last_unstale":"2024-09-14T04:25:04.828450+0000","last_undegraded":"2024-09-14T04:25:03.833841+0000","last_fullsized":"2024-09-14T04:25:04.828450+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:31:56.024893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":350,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.693909+0000","last_change":"2024-09-14T04:22:08.472386+0000","last_active":"2024-09-14T04:25:09.693909+0000","last_peered":"2024-09-14T04:25:09.693909+0000","last_clean":"2024-09-14T04:25:09.693909+0000","last_became_active":"2024-09-14T04:22:08.472100+0000","last_became_peered":"2024-09-14T04:22:08.472100+0000","last_unstale":"2024-09-14T04:25:09.693909+0000","last_undegraded":"2024-09-14T04:25:09.693909+0000","last_fullsized":"2024-09-14T04:25:09.693909+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:36:03.114984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":332,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.832189+0000","last_change":"2024-09-14T04:25:03.848222+0000","last_active":"2024-09-14T04:25:04.832189+0000","last_peered":"2024-09-14T04:25:04.832189+0000","last_clean":"2024-09-14T04:25:04.832189+0000","last_became_active":"2024-09-14T04:25:03.847917+0000","last_became_peered":"2024-09-14T04:25:03.847917+0000","last_unstale":"2024-09-14T04:25:04.832189+0000","last_undegraded":"2024-09-14T04:25:04.832189+0000","last_fullsized":"2024-09-14T04:25:04.832189+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:24:02.863794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"354'9211","reported_seq":10177,"reported_epoch":354,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-14T04:25:13.825997+0000","last_change":"2024-09-14T04:25:05.031546+0000","last_active":"2024-09-14T04:25:13.825997+0000","last_peered":"2024-09-14T04:25:13.825997+0000","last_clean":"2024-09-14T04:22:39.562383+0000","last_became_active":"2024-09-14T04:25:05.010585+0000","last_became_peered":"2024-09-14T04:25:05.010585+0000","last_unstale":"2024-09-14T04:25:13.825997+0000","last_undegraded":"2024-09-14T04:25:04.971945+0000","last_fullsized":"2024-09-14T04:25:13.825997+0000","mapping_epoch":352,"log_start":"330'3343","ondisk_log_start":"330'3343","created":321,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5868,"log_dups_size":0,"ondisk_log_size":5868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:25:05.632251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184614912,"num_objects":2817,"num_object_clones":0,"num_object_copies":8451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2817,"num_objects_misplaced":2976,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9211,"num_write_kb":384896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,7],"avail_no_missing":["6","2","7"],"object_location_counts":[{"shards":"2,6,7","objects":228},{"shards":"5,6,7","objects":2589}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.691566+0000","last_change":"2024-09-14T04:25:03.851044+0000","last_active":"2024-09-14T04:25:09.691566+0000","last_peered":"2024-09-14T04:25:09.691566+0000","last_clean":"2024-09-14T04:25:09.691566+0000","last_became_active":"2024-09-14T04:25:03.850683+0000","last_became_peered":"2024-09-14T04:25:03.850683+0000","last_unstale":"2024-09-14T04:25:09.691566+0000","last_undegraded":"2024-09-14T04:25:09.691566+0000","last_fullsized":"2024-09-14T04:25:09.691566+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:31:19.587275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":441,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.825407+0000","last_change":"2024-09-14T04:22:23.582781+0000","last_active":"2024-09-14T04:25:04.825407+0000","last_peered":"2024-09-14T04:25:04.825407+0000","last_clean":"2024-09-14T04:25:04.825407+0000","last_became_active":"2024-09-14T04:22:23.581865+0000","last_became_peered":"2024-09-14T04:22:23.581865+0000","last_unstale":"2024-09-14T04:25:04.825407+0000","last_undegraded":"2024-09-14T04:25:04.825407+0000","last_fullsized":"2024-09-14T04:25:04.825407+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:41:46.244094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830926+0000","last_change":"2024-09-14T04:25:03.858592+0000","last_active":"2024-09-14T04:25:04.830926+0000","last_peered":"2024-09-14T04:25:04.830926+0000","last_clean":"2024-09-14T04:25:04.830926+0000","last_became_active":"2024-09-14T04:25:03.858157+0000","last_became_peered":"2024-09-14T04:25:03.858157+0000","last_unstale":"2024-09-14T04:25:04.830926+0000","last_undegraded":"2024-09-14T04:25:04.830926+0000","last_fullsized":"2024-09-14T04:25:04.830926+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:59:59.274364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.e","version":"354'8880","reported_seq":8929,"reported_epoch":354,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-14T04:25:14.007157+0000","last_change":"2024-09-14T04:25:05.109848+0000","last_active":"2024-09-14T04:25:14.007157+0000","last_peered":"2024-09-14T04:25:14.007157+0000","last_clean":"2024-09-14T04:25:02.784195+0000","last_became_active":"2024-09-14T04:25:05.038267+0000","last_became_peered":"2024-09-14T04:25:05.038267+0000","last_unstale":"2024-09-14T04:25:14.007157+0000","last_undegraded":"2024-09-14T04:25:14.007157+0000","last_fullsized":"2024-09-14T04:25:14.007157+0000","mapping_epoch":352,"log_start":"330'3322","ondisk_log_start":"330'3322","created":321,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:59:31.221529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169476096,"num_objects":2586,"num_object_clones":0,"num_object_copies":7758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2586,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":366912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,7],"avail_no_missing":["5","4","7"],"object_location_counts":[{"shards":"4,5,7","objects":2586}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":329,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.691792+0000","last_change":"2024-09-14T04:24:47.612418+0000","last_active":"2024-09-14T04:25:09.691792+0000","last_peered":"2024-09-14T04:25:09.691792+0000","last_clean":"2024-09-14T04:25:09.691792+0000","last_became_active":"2024-09-14T04:24:47.611980+0000","last_became_peered":"2024-09-14T04:24:47.611980+0000","last_unstale":"2024-09-14T04:25:09.691792+0000","last_undegraded":"2024-09-14T04:25:09.691792+0000","last_fullsized":"2024-09-14T04:25:09.691792+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:54.457439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":531,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830187+0000","last_change":"2024-09-14T04:24:47.700444+0000","last_active":"2024-09-14T04:25:04.830187+0000","last_peered":"2024-09-14T04:25:04.830187+0000","last_clean":"2024-09-14T04:25:04.830187+0000","last_became_active":"2024-09-14T04:24:47.700119+0000","last_became_peered":"2024-09-14T04:24:47.700119+0000","last_unstale":"2024-09-14T04:25:04.830187+0000","last_undegraded":"2024-09-14T04:25:04.830187+0000","last_fullsized":"2024-09-14T04:25:04.830187+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:20:28.583061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":303,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830201+0000","last_change":"2024-09-14T04:25:03.860028+0000","last_active":"2024-09-14T04:25:04.830201+0000","last_peered":"2024-09-14T04:25:04.830201+0000","last_clean":"2024-09-14T04:25:04.830201+0000","last_became_active":"2024-09-14T04:25:03.859362+0000","last_became_peered":"2024-09-14T04:25:03.859362+0000","last_unstale":"2024-09-14T04:25:04.830201+0000","last_undegraded":"2024-09-14T04:25:04.830201+0000","last_fullsized":"2024-09-14T04:25:04.830201+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:24:58.388563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.d","version":"354'9203","reported_seq":10600,"reported_epoch":354,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-14T04:25:12.375358+0000","last_change":"2024-09-14T04:25:03.876494+0000","last_active":"2024-09-14T04:25:12.375358+0000","last_peered":"2024-09-14T04:25:12.375358+0000","last_clean":"2024-09-14T04:23:08.209175+0000","last_became_active":"2024-09-14T04:24:49.031117+0000","last_became_peered":"2024-09-14T04:24:49.031117+0000","last_unstale":"2024-09-14T04:25:12.375358+0000","last_undegraded":"2024-09-14T04:25:03.873029+0000","last_fullsized":"2024-09-14T04:25:12.375358+0000","mapping_epoch":348,"log_start":"334'3698","ondisk_log_start":"334'3698","created":321,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5505,"log_dups_size":0,"ondisk_log_size":5505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:31:18.170108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183107608,"num_objects":2795,"num_object_clones":0,"num_object_copies":8385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2795,"num_objects_misplaced":4063,"num_objects_unfound":0,"num_objects_dirty":2795,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9203,"num_write_kb":383873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":665,"num_bytes_recovered":43581440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,1],"avail_no_missing":["4","1","7"],"object_location_counts":[{"shards":"1,3,4","objects":1271},{"shards":"1,4,7","objects":1524}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":51,"reported_epoch":353,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-14T04:25:04.828138+0000","last_change":"2024-09-14T04:24:47.634703+0000","last_active":"2024-09-14T04:25:04.828138+0000","last_peered":"2024-09-14T04:25:04.828138+0000","last_clean":"2024-09-14T04:24:40.568350+0000","last_became_active":"2024-09-14T04:24:47.632408+0000","last_became_peered":"2024-09-14T04:24:47.632408+0000","last_unstale":"2024-09-14T04:25:04.828138+0000","last_undegraded":"2024-09-14T04:24:47.613906+0000","last_fullsized":"2024-09-14T04:25:04.828138+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:17:21.219738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":["5","1","6"],"object_location_counts":[{"shards":"1,5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":253,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.833915+0000","last_change":"2024-09-14T04:22:23.582184+0000","last_active":"2024-09-14T04:25:04.833915+0000","last_peered":"2024-09-14T04:25:04.833915+0000","last_clean":"2024-09-14T04:25:04.833915+0000","last_became_active":"2024-09-14T04:22:23.581776+0000","last_became_peered":"2024-09-14T04:22:23.581776+0000","last_unstale":"2024-09-14T04:25:04.833915+0000","last_undegraded":"2024-09-14T04:25:04.833915+0000","last_fullsized":"2024-09-14T04:25:04.833915+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:28:44.468477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":568,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.691960+0000","last_change":"2024-09-14T04:25:03.993517+0000","last_active":"2024-09-14T04:25:09.691960+0000","last_peered":"2024-09-14T04:25:09.691960+0000","last_clean":"2024-09-14T04:25:09.691960+0000","last_became_active":"2024-09-14T04:25:03.993215+0000","last_became_peered":"2024-09-14T04:25:03.993215+0000","last_unstale":"2024-09-14T04:25:09.691960+0000","last_undegraded":"2024-09-14T04:25:09.691960+0000","last_fullsized":"2024-09-14T04:25:09.691960+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:26:02.028512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"354'9247","reported_seq":9401,"reported_epoch":354,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-14T04:25:13.938994+0000","last_change":"2024-09-14T04:24:47.647708+0000","last_active":"2024-09-14T04:25:13.938994+0000","last_peered":"2024-09-14T04:25:13.938994+0000","last_clean":"2024-09-14T04:22:39.669512+0000","last_became_active":"2024-09-14T04:22:41.984075+0000","last_became_peered":"2024-09-14T04:22:41.984075+0000","last_unstale":"2024-09-14T04:25:13.938994+0000","last_undegraded":"2024-09-14T04:22:41.710998+0000","last_fullsized":"2024-09-14T04:22:41.680914+0000","mapping_epoch":332,"log_start":"334'3602","ondisk_log_start":"334'3602","created":321,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5645,"log_dups_size":0,"ondisk_log_size":5645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:59:30.103474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177012736,"num_objects":2701,"num_object_clones":0,"num_object_copies":8103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":382336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":3342336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"2,3,5","objects":205},{"shards":"3,5","objects":-726}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":49,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.829211+0000","last_change":"2024-09-14T04:22:40.683280+0000","last_active":"2024-09-14T04:25:04.829211+0000","last_peered":"2024-09-14T04:25:04.829211+0000","last_clean":"2024-09-14T04:25:04.829211+0000","last_became_active":"2024-09-14T04:22:40.682628+0000","last_became_peered":"2024-09-14T04:22:40.682628+0000","last_unstale":"2024-09-14T04:25:04.829211+0000","last_undegraded":"2024-09-14T04:25:04.829211+0000","last_fullsized":"2024-09-14T04:25:04.829211+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:17:09.414881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.691872+0000","last_change":"2024-09-14T04:25:03.842410+0000","last_active":"2024-09-14T04:25:09.691872+0000","last_peered":"2024-09-14T04:25:09.691872+0000","last_clean":"2024-09-14T04:25:09.691872+0000","last_became_active":"2024-09-14T04:25:03.842072+0000","last_became_peered":"2024-09-14T04:25:03.842072+0000","last_unstale":"2024-09-14T04:25:09.691872+0000","last_undegraded":"2024-09-14T04:25:09.691872+0000","last_fullsized":"2024-09-14T04:25:09.691872+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:07:31.359573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":526,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.827936+0000","last_change":"2024-09-14T04:24:47.634826+0000","last_active":"2024-09-14T04:25:04.827936+0000","last_peered":"2024-09-14T04:25:04.827936+0000","last_clean":"2024-09-14T04:25:04.827936+0000","last_became_active":"2024-09-14T04:24:47.632181+0000","last_became_peered":"2024-09-14T04:24:47.632181+0000","last_unstale":"2024-09-14T04:25:04.827936+0000","last_undegraded":"2024-09-14T04:25:04.827936+0000","last_fullsized":"2024-09-14T04:25:04.827936+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:08:34.754967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"354'9013","reported_seq":10460,"reported_epoch":354,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-14T04:25:12.600934+0000","last_change":"2024-09-14T04:25:03.976277+0000","last_active":"2024-09-14T04:25:12.600934+0000","last_peered":"2024-09-14T04:25:12.600934+0000","last_clean":"2024-09-14T04:22:39.593680+0000","last_became_active":"2024-09-14T04:25:03.959221+0000","last_became_peered":"2024-09-14T04:25:03.959221+0000","last_unstale":"2024-09-14T04:25:12.600934+0000","last_undegraded":"2024-09-14T04:25:03.914090+0000","last_fullsized":"2024-09-14T04:25:12.600934+0000","mapping_epoch":351,"log_start":"329'1900","ondisk_log_start":"329'1900","created":321,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":7113,"log_dups_size":0,"ondisk_log_size":7113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180944896,"num_objects":2761,"num_object_clones":0,"num_object_copies":8283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2730,"num_objects_misplaced":2792,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":376768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":44433408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[1,4,7],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":284},{"shards":"1,4,7","objects":2477}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":544,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.829879+0000","last_change":"2024-09-14T04:24:47.629307+0000","last_active":"2024-09-14T04:25:04.829879+0000","last_peered":"2024-09-14T04:25:04.829879+0000","last_clean":"2024-09-14T04:25:04.829879+0000","last_became_active":"2024-09-14T04:24:47.628896+0000","last_became_peered":"2024-09-14T04:24:47.628896+0000","last_unstale":"2024-09-14T04:25:04.829879+0000","last_undegraded":"2024-09-14T04:25:04.829879+0000","last_fullsized":"2024-09-14T04:25:04.829879+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:59:02.734701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830154+0000","last_change":"2024-09-14T04:25:03.848058+0000","last_active":"2024-09-14T04:25:04.830154+0000","last_peered":"2024-09-14T04:25:04.830154+0000","last_clean":"2024-09-14T04:25:04.830154+0000","last_became_active":"2024-09-14T04:25:03.847755+0000","last_became_peered":"2024-09-14T04:25:03.847755+0000","last_unstale":"2024-09-14T04:25:04.830154+0000","last_undegraded":"2024-09-14T04:25:04.830154+0000","last_fullsized":"2024-09-14T04:25:04.830154+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:08:18.754638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":576,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.828425+0000","last_change":"2024-09-14T04:25:03.847396+0000","last_active":"2024-09-14T04:25:04.828425+0000","last_peered":"2024-09-14T04:25:04.828425+0000","last_clean":"2024-09-14T04:25:04.828425+0000","last_became_active":"2024-09-14T04:25:03.847117+0000","last_became_peered":"2024-09-14T04:25:03.847117+0000","last_unstale":"2024-09-14T04:25:04.828425+0000","last_undegraded":"2024-09-14T04:25:04.828425+0000","last_fullsized":"2024-09-14T04:25:04.828425+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:05:19.638863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"354'9120","reported_seq":10317,"reported_epoch":354,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-14T04:25:14.004950+0000","last_change":"2024-09-14T04:25:03.860857+0000","last_active":"2024-09-14T04:25:14.004950+0000","last_peered":"2024-09-14T04:25:14.004950+0000","last_clean":"2024-09-14T04:22:39.669923+0000","last_became_active":"2024-09-14T04:22:42.223774+0000","last_became_peered":"2024-09-14T04:22:42.223774+0000","last_unstale":"2024-09-14T04:25:14.004950+0000","last_undegraded":"2024-09-14T04:22:41.725520+0000","last_fullsized":"2024-09-14T04:22:41.680462+0000","mapping_epoch":332,"log_start":"334'3490","ondisk_log_start":"334'3490","created":321,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5630,"log_dups_size":0,"ondisk_log_size":5630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:26:43.272376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177340416,"num_objects":2706,"num_object_clones":0,"num_object_copies":8118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9120,"num_write_kb":378432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":37552128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":212},{"shards":"0,5","objects":-140}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":562,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692623+0000","last_change":"2024-09-14T04:25:03.871509+0000","last_active":"2024-09-14T04:25:09.692623+0000","last_peered":"2024-09-14T04:25:09.692623+0000","last_clean":"2024-09-14T04:25:09.692623+0000","last_became_active":"2024-09-14T04:25:03.864512+0000","last_became_peered":"2024-09-14T04:25:03.864512+0000","last_unstale":"2024-09-14T04:25:09.692623+0000","last_undegraded":"2024-09-14T04:25:09.692623+0000","last_fullsized":"2024-09-14T04:25:09.692623+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:43:11.170414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":372,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830497+0000","last_change":"2024-09-14T04:24:47.618083+0000","last_active":"2024-09-14T04:25:04.830497+0000","last_peered":"2024-09-14T04:25:04.830497+0000","last_clean":"2024-09-14T04:25:04.830497+0000","last_became_active":"2024-09-14T04:24:47.617759+0000","last_became_peered":"2024-09-14T04:24:47.617759+0000","last_unstale":"2024-09-14T04:25:04.830497+0000","last_undegraded":"2024-09-14T04:25:04.830497+0000","last_fullsized":"2024-09-14T04:25:04.830497+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:08:35.614869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":95,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.691610+0000","last_change":"2024-09-14T04:24:47.700922+0000","last_active":"2024-09-14T04:25:09.691610+0000","last_peered":"2024-09-14T04:25:09.691610+0000","last_clean":"2024-09-14T04:25:09.691610+0000","last_became_active":"2024-09-14T04:24:47.700545+0000","last_became_peered":"2024-09-14T04:24:47.700545+0000","last_unstale":"2024-09-14T04:25:09.691610+0000","last_undegraded":"2024-09-14T04:25:09.691610+0000","last_fullsized":"2024-09-14T04:25:09.691610+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:13:00.255688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"354'9080","reported_seq":9762,"reported_epoch":354,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-14T04:25:14.007171+0000","last_change":"2024-09-14T04:25:05.170324+0000","last_active":"2024-09-14T04:25:14.007171+0000","last_peered":"2024-09-14T04:25:14.007171+0000","last_clean":"2024-09-14T04:22:39.592382+0000","last_became_active":"2024-09-14T04:25:05.052897+0000","last_became_peered":"2024-09-14T04:25:05.052897+0000","last_unstale":"2024-09-14T04:25:14.007171+0000","last_undegraded":"2024-09-14T04:25:05.032265+0000","last_fullsized":"2024-09-14T04:25:14.007171+0000","mapping_epoch":352,"log_start":"334'3430","ondisk_log_start":"334'3430","created":321,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:21:57.408864+0000","last_clean_scrub_stamp":"2024-09-14T04:21:57.408864+0000","objects_scrubbed":0,"log_size":5650,"log_dups_size":0,"ondisk_log_size":5650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:37:34.249303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176160768,"num_objects":2688,"num_object_clones":0,"num_object_copies":8064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9080,"num_write_kb":376576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":19398656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,7],"avail_no_missing":["5","2","6"],"object_location_counts":[{"shards":"1,5,6","objects":2487},{"shards":"2,5,6","objects":200},{"shards":"5,6,7","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":418,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.826120+0000","last_change":"2024-09-14T04:24:47.693203+0000","last_active":"2024-09-14T04:25:04.826120+0000","last_peered":"2024-09-14T04:25:04.826120+0000","last_clean":"2024-09-14T04:25:04.826120+0000","last_became_active":"2024-09-14T04:24:47.631512+0000","last_became_peered":"2024-09-14T04:24:47.631512+0000","last_unstale":"2024-09-14T04:25:04.826120+0000","last_undegraded":"2024-09-14T04:25:04.826120+0000","last_fullsized":"2024-09-14T04:25:04.826120+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:54:59.128924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":362,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.855747+0000","last_change":"2024-09-14T04:24:47.611672+0000","last_active":"2024-09-14T04:25:04.855747+0000","last_peered":"2024-09-14T04:25:04.855747+0000","last_clean":"2024-09-14T04:25:04.855747+0000","last_became_active":"2024-09-14T04:24:47.611094+0000","last_became_peered":"2024-09-14T04:24:47.611094+0000","last_unstale":"2024-09-14T04:25:04.855747+0000","last_undegraded":"2024-09-14T04:25:04.855747+0000","last_fullsized":"2024-09-14T04:25:04.855747+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:36:34.515592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":39,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.827885+0000","last_change":"2024-09-14T04:22:08.465532+0000","last_active":"2024-09-14T04:25:04.827885+0000","last_peered":"2024-09-14T04:25:04.827885+0000","last_clean":"2024-09-14T04:25:04.827885+0000","last_became_active":"2024-09-14T04:22:08.464571+0000","last_became_peered":"2024-09-14T04:22:08.464571+0000","last_unstale":"2024-09-14T04:25:04.827885+0000","last_undegraded":"2024-09-14T04:25:04.827885+0000","last_fullsized":"2024-09-14T04:25:04.827885+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:25:23.109822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":132,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.826418+0000","last_change":"2024-09-14T04:24:47.620259+0000","last_active":"2024-09-14T04:25:04.826418+0000","last_peered":"2024-09-14T04:25:04.826418+0000","last_clean":"2024-09-14T04:25:04.826418+0000","last_became_active":"2024-09-14T04:24:47.619797+0000","last_became_peered":"2024-09-14T04:24:47.619797+0000","last_unstale":"2024-09-14T04:25:04.826418+0000","last_undegraded":"2024-09-14T04:25:04.826418+0000","last_fullsized":"2024-09-14T04:25:04.826418+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:24.493567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":56,"reported_epoch":353,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-14T04:25:04.834921+0000","last_change":"2024-09-14T04:25:03.844733+0000","last_active":"2024-09-14T04:25:04.834921+0000","last_peered":"2024-09-14T04:25:04.834921+0000","last_clean":"2024-09-14T03:48:52.622802+0000","last_became_active":"2024-09-14T04:25:03.844487+0000","last_became_peered":"2024-09-14T04:25:03.844487+0000","last_unstale":"2024-09-14T04:25:04.834921+0000","last_undegraded":"2024-09-14T04:25:03.833379+0000","last_fullsized":"2024-09-14T04:25:04.834921+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:34:52.382872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":529,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830454+0000","last_change":"2024-09-14T04:24:47.632162+0000","last_active":"2024-09-14T04:25:04.830454+0000","last_peered":"2024-09-14T04:25:04.830454+0000","last_clean":"2024-09-14T04:25:04.830454+0000","last_became_active":"2024-09-14T04:24:47.631561+0000","last_became_peered":"2024-09-14T04:24:47.631561+0000","last_unstale":"2024-09-14T04:25:04.830454+0000","last_undegraded":"2024-09-14T04:25:04.830454+0000","last_fullsized":"2024-09-14T04:25:04.830454+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:21:47.381009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":224,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.693527+0000","last_change":"2024-09-14T04:25:03.852853+0000","last_active":"2024-09-14T04:25:09.693527+0000","last_peered":"2024-09-14T04:25:09.693527+0000","last_clean":"2024-09-14T04:25:09.693527+0000","last_became_active":"2024-09-14T04:25:03.847615+0000","last_became_peered":"2024-09-14T04:25:03.847615+0000","last_unstale":"2024-09-14T04:25:09.693527+0000","last_undegraded":"2024-09-14T04:25:09.693527+0000","last_fullsized":"2024-09-14T04:25:09.693527+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:09:57.623984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":59,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.831003+0000","last_change":"2024-09-14T04:25:03.863384+0000","last_active":"2024-09-14T04:25:04.831003+0000","last_peered":"2024-09-14T04:25:04.831003+0000","last_clean":"2024-09-14T04:25:04.831003+0000","last_became_active":"2024-09-14T04:25:03.863083+0000","last_became_peered":"2024-09-14T04:25:03.863083+0000","last_unstale":"2024-09-14T04:25:04.831003+0000","last_undegraded":"2024-09-14T04:25:04.831003+0000","last_fullsized":"2024-09-14T04:25:04.831003+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:37:04.456671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":517,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.691973+0000","last_change":"2024-09-14T04:25:03.864958+0000","last_active":"2024-09-14T04:25:09.691973+0000","last_peered":"2024-09-14T04:25:09.691973+0000","last_clean":"2024-09-14T04:25:09.691973+0000","last_became_active":"2024-09-14T04:25:03.864529+0000","last_became_peered":"2024-09-14T04:25:03.864529+0000","last_unstale":"2024-09-14T04:25:09.691973+0000","last_undegraded":"2024-09-14T04:25:09.691973+0000","last_fullsized":"2024-09-14T04:25:09.691973+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:12:25.237701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":589,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.830063+0000","last_change":"2024-09-14T04:22:23.652042+0000","last_active":"2024-09-14T04:25:04.830063+0000","last_peered":"2024-09-14T04:25:04.830063+0000","last_clean":"2024-09-14T04:25:04.830063+0000","last_became_active":"2024-09-14T04:22:23.651622+0000","last_became_peered":"2024-09-14T04:22:23.651622+0000","last_unstale":"2024-09-14T04:25:04.830063+0000","last_undegraded":"2024-09-14T04:25:04.830063+0000","last_fullsized":"2024-09-14T04:25:04.830063+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:15.951118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":529,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.691770+0000","last_change":"2024-09-14T04:24:47.613484+0000","last_active":"2024-09-14T04:25:09.691770+0000","last_peered":"2024-09-14T04:25:09.691770+0000","last_clean":"2024-09-14T04:25:09.691770+0000","last_became_active":"2024-09-14T04:24:47.613215+0000","last_became_peered":"2024-09-14T04:24:47.613215+0000","last_unstale":"2024-09-14T04:25:09.691770+0000","last_undegraded":"2024-09-14T04:25:09.691770+0000","last_fullsized":"2024-09-14T04:25:09.691770+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:35.605770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":369,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.855670+0000","last_change":"2024-09-14T04:24:47.621562+0000","last_active":"2024-09-14T04:25:04.855670+0000","last_peered":"2024-09-14T04:25:04.855670+0000","last_clean":"2024-09-14T04:25:04.855670+0000","last_became_active":"2024-09-14T04:24:47.621093+0000","last_became_peered":"2024-09-14T04:24:47.621093+0000","last_unstale":"2024-09-14T04:25:04.855670+0000","last_undegraded":"2024-09-14T04:25:04.855670+0000","last_fullsized":"2024-09-14T04:25:04.855670+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:20:00.191019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":349,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.691676+0000","last_change":"2024-09-14T04:25:03.856727+0000","last_active":"2024-09-14T04:25:09.691676+0000","last_peered":"2024-09-14T04:25:09.691676+0000","last_clean":"2024-09-14T04:25:09.691676+0000","last_became_active":"2024-09-14T04:25:03.856404+0000","last_became_peered":"2024-09-14T04:25:03.856404+0000","last_unstale":"2024-09-14T04:25:09.691676+0000","last_undegraded":"2024-09-14T04:25:09.691676+0000","last_fullsized":"2024-09-14T04:25:09.691676+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:45:15.950009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":352,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692440+0000","last_change":"2024-09-14T04:24:47.622317+0000","last_active":"2024-09-14T04:25:09.692440+0000","last_peered":"2024-09-14T04:25:09.692440+0000","last_clean":"2024-09-14T04:25:09.692440+0000","last_became_active":"2024-09-14T04:24:47.621902+0000","last_became_peered":"2024-09-14T04:24:47.621902+0000","last_unstale":"2024-09-14T04:25:09.692440+0000","last_undegraded":"2024-09-14T04:25:09.692440+0000","last_fullsized":"2024-09-14T04:25:09.692440+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:38:21.542031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":378,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692140+0000","last_change":"2024-09-14T04:24:47.612687+0000","last_active":"2024-09-14T04:25:09.692140+0000","last_peered":"2024-09-14T04:25:09.692140+0000","last_clean":"2024-09-14T04:25:09.692140+0000","last_became_active":"2024-09-14T04:24:47.612392+0000","last_became_peered":"2024-09-14T04:24:47.612392+0000","last_unstale":"2024-09-14T04:25:09.692140+0000","last_undegraded":"2024-09-14T04:25:09.692140+0000","last_fullsized":"2024-09-14T04:25:09.692140+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:52:51.954808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":348,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.693408+0000","last_change":"2024-09-14T04:24:47.613822+0000","last_active":"2024-09-14T04:25:09.693408+0000","last_peered":"2024-09-14T04:25:09.693408+0000","last_clean":"2024-09-14T04:25:09.693408+0000","last_became_active":"2024-09-14T04:24:47.613315+0000","last_became_peered":"2024-09-14T04:24:47.613315+0000","last_unstale":"2024-09-14T04:25:09.693408+0000","last_undegraded":"2024-09-14T04:25:09.693408+0000","last_fullsized":"2024-09-14T04:25:09.693408+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:22.682884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":142,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.826758+0000","last_change":"2024-09-14T04:25:03.851961+0000","last_active":"2024-09-14T04:25:04.826758+0000","last_peered":"2024-09-14T04:25:04.826758+0000","last_clean":"2024-09-14T04:25:04.826758+0000","last_became_active":"2024-09-14T04:25:03.851627+0000","last_became_peered":"2024-09-14T04:25:03.851627+0000","last_unstale":"2024-09-14T04:25:04.826758+0000","last_undegraded":"2024-09-14T04:25:04.826758+0000","last_fullsized":"2024-09-14T04:25:04.826758+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:15:54.476548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":294,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692037+0000","last_change":"2024-09-14T04:24:47.612523+0000","last_active":"2024-09-14T04:25:09.692037+0000","last_peered":"2024-09-14T04:25:09.692037+0000","last_clean":"2024-09-14T04:25:09.692037+0000","last_became_active":"2024-09-14T04:24:47.612181+0000","last_became_peered":"2024-09-14T04:24:47.612181+0000","last_unstale":"2024-09-14T04:25:09.692037+0000","last_undegraded":"2024-09-14T04:25:09.692037+0000","last_fullsized":"2024-09-14T04:25:09.692037+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:02:11.076040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":16,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692697+0000","last_change":"2024-09-14T04:25:03.842543+0000","last_active":"2024-09-14T04:25:09.692697+0000","last_peered":"2024-09-14T04:25:09.692697+0000","last_clean":"2024-09-14T04:25:09.692697+0000","last_became_active":"2024-09-14T04:25:03.841995+0000","last_became_peered":"2024-09-14T04:25:03.841995+0000","last_unstale":"2024-09-14T04:25:09.692697+0000","last_undegraded":"2024-09-14T04:25:09.692697+0000","last_fullsized":"2024-09-14T04:25:09.692697+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:16:52.559373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":439,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.693052+0000","last_change":"2024-09-14T04:25:03.842534+0000","last_active":"2024-09-14T04:25:09.693052+0000","last_peered":"2024-09-14T04:25:09.693052+0000","last_clean":"2024-09-14T04:25:09.693052+0000","last_became_active":"2024-09-14T04:25:03.841946+0000","last_became_peered":"2024-09-14T04:25:03.841946+0000","last_unstale":"2024-09-14T04:25:09.693052+0000","last_undegraded":"2024-09-14T04:25:09.693052+0000","last_fullsized":"2024-09-14T04:25:09.693052+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:28:25.426804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":57,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.831151+0000","last_change":"2024-09-14T04:24:47.706532+0000","last_active":"2024-09-14T04:25:04.831151+0000","last_peered":"2024-09-14T04:25:04.831151+0000","last_clean":"2024-09-14T04:25:04.831151+0000","last_became_active":"2024-09-14T04:24:47.628189+0000","last_became_peered":"2024-09-14T04:24:47.628189+0000","last_unstale":"2024-09-14T04:25:04.831151+0000","last_undegraded":"2024-09-14T04:25:04.831151+0000","last_fullsized":"2024-09-14T04:25:04.831151+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:29:46.206803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":425,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.693969+0000","last_change":"2024-09-14T04:17:48.081862+0000","last_active":"2024-09-14T04:25:09.693969+0000","last_peered":"2024-09-14T04:25:09.693969+0000","last_clean":"2024-09-14T04:25:09.693969+0000","last_became_active":"2024-09-14T04:17:48.081095+0000","last_became_peered":"2024-09-14T04:17:48.081095+0000","last_unstale":"2024-09-14T04:25:09.693969+0000","last_undegraded":"2024-09-14T04:25:09.693969+0000","last_fullsized":"2024-09-14T04:25:09.693969+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:04:07.654057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":43,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.829372+0000","last_change":"2024-09-14T04:22:08.472912+0000","last_active":"2024-09-14T04:25:04.829372+0000","last_peered":"2024-09-14T04:25:04.829372+0000","last_clean":"2024-09-14T04:25:04.829372+0000","last_became_active":"2024-09-14T04:22:08.463787+0000","last_became_peered":"2024-09-14T04:22:08.463787+0000","last_unstale":"2024-09-14T04:25:04.829372+0000","last_undegraded":"2024-09-14T04:25:04.829372+0000","last_fullsized":"2024-09-14T04:25:04.829372+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:56:56.756448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":342,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.691892+0000","last_change":"2024-09-14T04:22:40.684690+0000","last_active":"2024-09-14T04:25:09.691892+0000","last_peered":"2024-09-14T04:25:09.691892+0000","last_clean":"2024-09-14T04:25:09.691892+0000","last_became_active":"2024-09-14T04:22:40.684352+0000","last_became_peered":"2024-09-14T04:22:40.684352+0000","last_unstale":"2024-09-14T04:25:09.691892+0000","last_undegraded":"2024-09-14T04:25:09.691892+0000","last_fullsized":"2024-09-14T04:25:09.691892+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:11:28.618520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":357,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-14T04:25:09.692520+0000","last_change":"2024-09-14T04:25:03.845027+0000","last_active":"2024-09-14T04:25:09.692520+0000","last_peered":"2024-09-14T04:25:09.692520+0000","last_clean":"2024-09-14T04:25:09.692520+0000","last_became_active":"2024-09-14T04:25:03.844662+0000","last_became_peered":"2024-09-14T04:25:03.844662+0000","last_unstale":"2024-09-14T04:25:09.692520+0000","last_undegraded":"2024-09-14T04:25:09.692520+0000","last_fullsized":"2024-09-14T04:25:09.692520+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:11:20.246338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":122,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.825485+0000","last_change":"2024-09-14T04:24:47.614341+0000","last_active":"2024-09-14T04:25:04.825485+0000","last_peered":"2024-09-14T04:25:04.825485+0000","last_clean":"2024-09-14T04:25:04.825485+0000","last_became_active":"2024-09-14T04:24:47.613341+0000","last_became_peered":"2024-09-14T04:24:47.613341+0000","last_unstale":"2024-09-14T04:25:04.825485+0000","last_undegraded":"2024-09-14T04:25:04.825485+0000","last_fullsized":"2024-09-14T04:25:04.825485+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:29:05.479921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":602,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-14T04:25:04.831114+0000","last_change":"2024-09-14T04:22:40.691159+0000","last_active":"2024-09-14T04:25:04.831114+0000","last_peered":"2024-09-14T04:25:04.831114+0000","last_clean":"2024-09-14T04:25:04.831114+0000","last_became_active":"2024-09-14T04:22:40.687791+0000","last_became_peered":"2024-09-14T04:22:40.687791+0000","last_unstale":"2024-09-14T04:25:04.831114+0000","last_undegraded":"2024-09-14T04:25:04.831114+0000","last_fullsized":"2024-09-14T04:25:04.831114+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:20:22.106964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":2867068952,"num_objects":43749,"num_object_clones":0,"num_object_copies":131247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23023,"num_objects_misplaced":23625,"num_objects_unfound":0,"num_objects_dirty":43749,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146377,"num_write_kb":6083969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10934,"num_bytes_recovered":716046336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10712330240,"data_stored":10712318024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93633,"ondisk_log_size":93633,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1482,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":206158430738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626967,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2404544,"kb_used_data":1214648,"kb_used_omap":1,"kb_used_meta":1189886,"kb_avail":91340608,"statfs":{"total":95995035648,"available":93532782592,"internally_reserved":0,"allocated":1243799552,"data_stored":1242147674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1218443608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823195,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3240172,"kb_used_data":1729508,"kb_used_omap":1,"kb_used_meta":1510462,"kb_avail":90504980,"statfs":{"total":95995035648,"available":92677099520,"internally_reserved":0,"allocated":1771016192,"data_stored":1769379179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1546713432},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917277,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2111036,"kb_used_data":1776884,"kb_used_omap":6,"kb_used_meta":334137,"kb_avail":91634116,"statfs":{"total":95995035648,"available":93833334784,"internally_reserved":0,"allocated":1819529216,"data_stored":1817880367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":342157311},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552576,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1749168,"kb_used_data":786088,"kb_used_omap":4,"kb_used_meta":963067,"kb_avail":91995984,"statfs":{"total":95995035648,"available":94203887616,"internally_reserved":0,"allocated":804954112,"data_stored":803313156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":986181053},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":468151435617,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":416611828087,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597384076,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1770672,"kb_used_data":604904,"kb_used_omap":5,"kb_used_meta":1165754,"kb_avail":91974480,"statfs":{"total":95995035648,"available":94181867520,"internally_reserved":0,"allocated":619421696,"data_stored":617781483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":1193732156},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":607977472,"data_stored":607977472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1070010368,"data_stored":1070006296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1795162112,"data_stored":1795162112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":793841664,"data_stored":793837592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1799229440,"data_stored":1799225368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1755381760,"data_stored":1755381760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1233911808,"data_stored":1233911808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1656815616,"data_stored":1656815616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:25:21.473 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-14T04:25:21.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-14T04:25:21.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:21.591 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:21.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:21.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:21.700 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:25:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:25:21] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-14T04:25:21.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:21.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:21.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:21.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:21.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:21.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:22.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:22.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:22.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:22.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:22.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:22.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:22.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:22.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:22.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:22.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:22.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:22 smithi114 ceph-mon[103029]: pgmap v1375: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 471 op/s; 21812/123126 objects degraded (17.715%); 22153/123126 objects misplaced (17.992%); 12 MiB/s, 208 objects/s recovering 2024-09-14T04:25:22.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:22.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:22 smithi114 ceph-mon[106969]: pgmap v1375: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 471 op/s; 21812/123126 objects degraded (17.715%); 22153/123126 objects misplaced (17.992%); 12 MiB/s, 208 objects/s recovering 2024-09-14T04:25:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:22.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:22.393 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:22.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:22.494 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:22.521 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:22.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:22.533 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:22 smithi160 ceph-mon[88407]: pgmap v1375: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 471 op/s; 21812/123126 objects degraded (17.715%); 22153/123126 objects misplaced (17.992%); 12 MiB/s, 208 objects/s recovering 2024-09-14T04:25:22.533 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:22.533 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:22.624 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:22.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:22.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:22.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:22.771 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:22.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:22.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:22.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:22.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:22.957 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:22.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:23.001 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:23.041 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:23.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:23.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:23.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:23.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:23.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:23.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:23.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:23.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:23.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:23.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:23.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:23.511 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:25:23.511 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:25:23.511 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:23.511 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:23.511 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:23.511 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:25:23.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:25:23.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:23.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:23.512 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:23.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:23.653 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:23.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:25:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:25:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:23.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:23.755 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:23.795 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:23.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:23.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:23.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:23.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:24.024 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:24.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:24.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:24.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:24.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:24.172 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:24.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:24.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:24.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:24.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:24.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:24.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:24.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:24.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:24.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:24.569 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:24.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:24.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:24.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:24.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:24.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:24.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:24.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:24.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:24.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:24.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:24.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:25.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:24 smithi114 ceph-mon[106969]: pgmap v1376: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 508 op/s; 21240/121041 objects degraded (17.548%); 21685/121041 objects misplaced (17.915%); 14 MiB/s, 229 objects/s recovering 2024-09-14T04:25:25.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 21240/121041 objects degraded (17.548%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:25.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:24 smithi114 ceph-mon[103029]: pgmap v1376: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 508 op/s; 21240/121041 objects degraded (17.548%); 21685/121041 objects misplaced (17.915%); 14 MiB/s, 229 objects/s recovering 2024-09-14T04:25:25.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 21240/121041 objects degraded (17.548%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:25.068 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:24 smithi160 ceph-mon[88407]: pgmap v1376: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.5 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 508 op/s; 21240/121041 objects degraded (17.548%); 21685/121041 objects misplaced (17.915%); 14 MiB/s, 229 objects/s recovering 2024-09-14T04:25:25.068 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 21240/121041 objects degraded (17.548%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:25.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:25.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:25.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:25.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:25.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:25.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:25.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:25.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:25.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:25.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:25.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:25.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:25.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:25.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:25.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:25.653 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:25.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:25.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:25.675 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:25:25.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:25.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:25.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:25.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:25.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:25.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:25.929 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3442593110' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:25:25.929 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:25 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:25:25.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3442593110' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:25:25.929 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:25 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:25:26.004 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:26.029 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:26.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:26.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:26.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:26.155 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:26.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:26.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3442593110' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:25:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:25 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:25:26.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:26.295 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:26.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:26.301 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:25:26.301 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:25:26.301 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:25:26.302 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:25:26.303 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:25:26.303 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:25:26.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:26.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:26.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:26.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:26.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:26.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:26.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:26.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:26.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:26.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:26.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:26.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:26.807 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:25:26.807 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:25:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:26 smithi160 ceph-mon[88407]: pgmap v1377: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 628 op/s; 20701/114711 objects degraded (18.046%); 20941/114711 objects misplaced (18.255%); 13 MiB/s, 228 objects/s recovering 2024-09-14T04:25:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:26 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:26 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:25:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:26 smithi160 ceph-mon[88407]: osdmap e355: 8 total, 8 up, 5 in 2024-09-14T04:25:26.809 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:26 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1438534782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:26.852 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:26.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:26.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:26.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:26.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:27.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:27.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:27.119 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:27.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:27.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:26 smithi114 ceph-mon[103029]: pgmap v1377: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 628 op/s; 20701/114711 objects degraded (18.046%); 20941/114711 objects misplaced (18.255%); 13 MiB/s, 228 objects/s recovering 2024-09-14T04:25:27.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:26 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:27.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:26 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:25:27.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:26 smithi114 ceph-mon[103029]: osdmap e355: 8 total, 8 up, 5 in 2024-09-14T04:25:27.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:26 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1438534782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:26 smithi114 ceph-mon[106969]: pgmap v1377: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 628 op/s; 20701/114711 objects degraded (18.046%); 20941/114711 objects misplaced (18.255%); 13 MiB/s, 228 objects/s recovering 2024-09-14T04:25:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:26 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:26 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:25:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:26 smithi114 ceph-mon[106969]: osdmap e355: 8 total, 8 up, 5 in 2024-09-14T04:25:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:26 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1438534782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:27.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:27.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:27.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:27.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:27.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:27.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:27.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:27.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:27.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:27.599 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-14T04:25:27.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:27.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:27.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:27.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:27.788 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:27.835 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:27.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:27.937 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:27.994 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:27.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:27.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:28.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:28.096 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:25:28.119 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:28.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:28.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:28.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:28.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:28.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:28.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:28.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:28.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:28.529 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:28.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:28.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:28.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:28.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:28.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:28.783 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:28.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:28.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:28.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:28 smithi160 ceph-mon[88407]: pgmap v1379: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 553 op/s; 20701/114711 objects degraded (18.046%); 20941/114711 objects misplaced (18.255%); 11 MiB/s, 204 objects/s recovering 2024-09-14T04:25:28.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:28.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:29.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:29.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:28 smithi114 ceph-mon[103029]: pgmap v1379: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 553 op/s; 20701/114711 objects degraded (18.046%); 20941/114711 objects misplaced (18.255%); 11 MiB/s, 204 objects/s recovering 2024-09-14T04:25:29.050 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:28 smithi114 ceph-mon[106969]: pgmap v1379: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 553 op/s; 20701/114711 objects degraded (18.046%); 20941/114711 objects misplaced (18.255%); 11 MiB/s, 204 objects/s recovering 2024-09-14T04:25:29.071 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:29.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:29.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:29.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:29.196 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:29.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:29.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:29.323 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:29.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:29.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:29.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:29.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:29.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:29.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:29.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:29.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:29.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:29.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:29.737 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:29.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:29.817 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 20701/114711 objects degraded (18.046%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:29.838 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:29.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:29.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:29.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:29.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:29.981 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 20701/114711 objects degraded (18.046%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:29.981 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 20701/114711 objects degraded (18.046%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:30.002 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:30.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:30.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:30.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:30.127 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:30.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:30.165 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:25:30.165 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:25:30.229 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:30.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:30.262 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:30.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:30.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:30.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:25:30.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:30.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:25:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:25:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:25:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:30.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:30.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:30.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:30.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:30.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:30.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:30.686 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:30 smithi114 ceph-mon[103029]: pgmap v1380: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 552 op/s; 19664/107109 objects degraded (18.359%); 19944/107109 objects misplaced (18.620%); 6.3 MiB/s, 146 objects/s recovering 2024-09-14T04:25:30.687 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:30 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/444511588' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-14T04:25:30.687 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:30 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-14T04:25:30.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:30 smithi114 ceph-mon[106969]: pgmap v1380: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 552 op/s; 19664/107109 objects degraded (18.359%); 19944/107109 objects misplaced (18.620%); 6.3 MiB/s, 146 objects/s recovering 2024-09-14T04:25:30.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:30 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/444511588' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-14T04:25:30.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:30 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-14T04:25:30.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:30.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 1 pg_num to 11 2024-09-14T04:25:30.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:30.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:30.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:30.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:30.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:30 smithi160 ceph-mon[88407]: pgmap v1380: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 552 op/s; 19664/107109 objects degraded (18.359%); 19944/107109 objects misplaced (18.620%); 6.3 MiB/s, 146 objects/s recovering 2024-09-14T04:25:30.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:30 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/444511588' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-14T04:25:30.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:30 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-14T04:25:30.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:25:30.926 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:30.973 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:30.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:30.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-14T04:25:31.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:31.142 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:31.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:31.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:31.269 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:31.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:31.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:31.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:31.459 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:31.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:31.490 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:25:31.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:31.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:31.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:31.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:31.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:25:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:25:31] "GET /metrics HTTP/1.1" 200 36598 "" "Prometheus/2.51.0" 2024-09-14T04:25:31.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:31.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:31.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:31.777 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:31.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:31.858 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:31 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-14T04:25:31.858 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:31 smithi160 ceph-mon[88407]: osdmap e356: 8 total, 8 up, 5 in 2024-09-14T04:25:31.858 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:31 smithi160 ceph-mon[88407]: pgmap v1382: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 603 op/s; 19664/106545 objects degraded (18.456%); 19944/106545 objects misplaced (18.719%); 5.0 MiB/s, 137 objects/s recovering 2024-09-14T04:25:31.858 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:31 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-14T04:25:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:31.878 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:31.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:31.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:32.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:32.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:32.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:31 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-14T04:25:32.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:31 smithi114 ceph-mon[106969]: osdmap e356: 8 total, 8 up, 5 in 2024-09-14T04:25:32.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:31 smithi114 ceph-mon[106969]: pgmap v1382: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 603 op/s; 19664/106545 objects degraded (18.456%); 19944/106545 objects misplaced (18.719%); 5.0 MiB/s, 137 objects/s recovering 2024-09-14T04:25:32.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:31 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-14T04:25:32.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:31 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-14T04:25:32.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:31 smithi114 ceph-mon[103029]: osdmap e356: 8 total, 8 up, 5 in 2024-09-14T04:25:32.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:31 smithi114 ceph-mon[103029]: pgmap v1382: 89 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.2 GiB data, 9.8 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 603 op/s; 19664/106545 objects degraded (18.456%); 19944/106545 objects misplaced (18.719%); 5.0 MiB/s, 137 objects/s recovering 2024-09-14T04:25:32.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:31 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-14T04:25:32.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:32.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:32.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:32.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:32.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:32.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:32.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:32.423 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:32.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:32.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:32.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:32.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:32.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:32.684 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:32.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:32.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:32.808 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:32.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:32.864 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-14T04:25:32.864 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:32 smithi114 ceph-mon[106969]: osdmap e357: 8 total, 8 up, 5 in 2024-09-14T04:25:32.864 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-14T04:25:32.864 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:32 smithi114 ceph-mon[103029]: osdmap e357: 8 total, 8 up, 5 in 2024-09-14T04:25:32.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:32.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:32.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:32.983 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-14T04:25:32.983 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:32 smithi160 ceph-mon[88407]: osdmap e357: 8 total, 8 up, 5 in 2024-09-14T04:25:33.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:33.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:33.176 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:33.219 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:33.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:33.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:33.366 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:33.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:33.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:33.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:33.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:33.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:33.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:33.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:33.755 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:33.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:33.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:33.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:33.966 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:33.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:34.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:33 smithi114 ceph-mon[103029]: osdmap e358: 8 total, 8 up, 5 in 2024-09-14T04:25:34.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:33 smithi114 ceph-mon[103029]: pgmap v1385: 99 pgs: 10 unknown, 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.1 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 565 op/s; 19161/104538 objects degraded (18.329%); 19604/104538 objects misplaced (18.753%); 5.8 MiB/s, 138 objects/s recovering 2024-09-14T04:25:34.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:33 smithi114 ceph-mon[106969]: osdmap e358: 8 total, 8 up, 5 in 2024-09-14T04:25:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:33 smithi114 ceph-mon[106969]: pgmap v1385: 99 pgs: 10 unknown, 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.1 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 565 op/s; 19161/104538 objects degraded (18.329%); 19604/104538 objects misplaced (18.753%); 5.8 MiB/s, 138 objects/s recovering 2024-09-14T04:25:34.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:34.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:34.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:34.095 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:33 smithi160 ceph-mon[88407]: osdmap e358: 8 total, 8 up, 5 in 2024-09-14T04:25:34.095 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:33 smithi160 ceph-mon[88407]: pgmap v1385: 99 pgs: 10 unknown, 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 72 active+clean; 2.1 GiB data, 9.6 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 565 op/s; 19161/104538 objects degraded (18.329%); 19604/104538 objects misplaced (18.753%); 5.8 MiB/s, 138 objects/s recovering 2024-09-14T04:25:34.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:34.218 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:34.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:34.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:34.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:34.343 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:34.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:34.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:34.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:34.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:34.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:34.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:34.619 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:34.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:34.664 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:25:34.664 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:25:34.720 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:34.743 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:34.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:34.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 19161/104538 objects degraded (18.329%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:34.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 19161/104538 objects degraded (18.329%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:34.845 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:34.871 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:34.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 19161/104538 objects degraded (18.329%), 13 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:34.972 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:35.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:35.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:35.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:35.151 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:35.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:35.253 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:35.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:35.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:35.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:35.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-14T04:25:35.428 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:35.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:35.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:35.515 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:25:35.515 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:25:35.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:35.535 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:25:35.536 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:25:35.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:35.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:35.555 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:25:35.555 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:25:35.576 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:25:35.576 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:25:35.606 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:25:35.607 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:25:35.631 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:25:35.632 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:25:35.653 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:25:35.654 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:25:35.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:35.674 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:25:35.674 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:25:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:35.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:35.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:35.729 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:25:35.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:35.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:35.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:35.910 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:25:35.926 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:35 smithi114 ceph-mon[106969]: pgmap v1386: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.0 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 481 op/s; 18610/98448 objects degraded (18.903%); 18931/98448 objects misplaced (19.229%); 3.8 MiB/s, 97 objects/s recovering 2024-09-14T04:25:35.926 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:35 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1847744832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:35.926 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:35 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1485451721' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:25:35.926 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:35 smithi114 ceph-mon[103029]: pgmap v1386: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.0 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 481 op/s; 18610/98448 objects degraded (18.903%); 18931/98448 objects misplaced (19.229%); 3.8 MiB/s, 97 objects/s recovering 2024-09-14T04:25:35.926 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:35 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1847744832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:35.927 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:35 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1485451721' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:25:35.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:35.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:35.997 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:35.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:36.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:35 smithi160 ceph-mon[88407]: pgmap v1386: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.0 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 481 op/s; 18610/98448 objects degraded (18.903%); 18931/98448 objects misplaced (19.229%); 3.8 MiB/s, 97 objects/s recovering 2024-09-14T04:25:36.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:35 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1847744832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:36.063 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:35 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1485451721' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:25:36.098 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:36.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:36.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:36.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:36.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:36.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:36.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:36.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:36.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:36.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:36.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:36.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:36.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:36.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:36.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:36.696 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:36.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:36.717 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 4, 5] out_osds: [2, 1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:25:36.717 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:25:36.717 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-14T04:25:36.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd in 2 2024-09-14T04:25:36.798 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:36.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:36.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:36.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:36.968 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:36.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:37.013 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:36 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:37.013 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:36 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1485451721' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:25:37.013 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:36 smithi160 ceph-mon[88407]: osdmap e359: 8 total, 8 up, 5 in 2024-09-14T04:25:37.044 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:36 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:37.044 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:36 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1485451721' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:25:37.044 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:36 smithi114 ceph-mon[106969]: osdmap e359: 8 total, 8 up, 5 in 2024-09-14T04:25:37.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:36 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:37.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:36 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1485451721' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:25:37.044 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:36 smithi114 ceph-mon[103029]: osdmap e359: 8 total, 8 up, 5 in 2024-09-14T04:25:37.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:37.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:37.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:37.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:37.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:37.258 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:37.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:37.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:37.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:37.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:37.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:37.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:37.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:37.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:37.680 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:37.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:37.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:37.836 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:37.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:37.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:37 smithi160 ceph-mon[88407]: pgmap v1388: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.0 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 449 op/s; 18610/98448 objects degraded (18.903%); 18931/98448 objects misplaced (19.229%); 3.8 MiB/s, 97 objects/s recovering 2024-09-14T04:25:37.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:37.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:37.968 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:37.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:38.048 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:37 smithi114 ceph-mon[106969]: pgmap v1388: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.0 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 449 op/s; 18610/98448 objects degraded (18.903%); 18931/98448 objects misplaced (19.229%); 3.8 MiB/s, 97 objects/s recovering 2024-09-14T04:25:38.048 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:38.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:37 smithi114 ceph-mon[103029]: pgmap v1388: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.0 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 449 op/s; 18610/98448 objects degraded (18.903%); 18931/98448 objects misplaced (19.229%); 3.8 MiB/s, 97 objects/s recovering 2024-09-14T04:25:38.048 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:38.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:38.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:38.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:38.210 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:38.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:38.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:38.292 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:160169 13.9 e359/332 13:9fbf5af0:::benchmark_data_smithi114_288325_object65104:head v 359'9919, mlcod=330'3056)", 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:38.268871+0000", 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.020838176, 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020867006, 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.268871+0000", 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.268871+0000", 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.268870+0000", 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:25:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.268879+0000", 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2230000000000007e-06 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.268894+0000", 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.521e-05 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.268900+0000", 2024-09-14T04:25:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7069999999999997e-06 2024-09-14T04:25:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:25:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.273724+0000", 2024-09-14T04:25:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048244059999999998 2024-09-14T04:25:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:25:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.273776+0000", 2024-09-14T04:25:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2323e-05 2024-09-14T04:25:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:160176 13.4 13:24ab3d26:::benchmark_data_smithi114_288325_object65111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:38.273924+0000", 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.015785528, 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015893507000000001, 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160176 2024-09-14T04:25:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.273924+0000", 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.273924+0000", 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.273922+0000", 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.273929+0000", 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2699999999999999e-06 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.273943+0000", 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396e-05 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.273949+0000", 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5400000000000003e-06 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274370+0000", 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00042103900000000001 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274512+0000", 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000142282 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,5", 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274665+0000", 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015282800000000001 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.284986+0000", 2024-09-14T04:25:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010321250000000001 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285125+0000", 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013894299999999999 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285279+0000", 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015395400000000001 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:160178 13.f 13:fb9f24af:::benchmark_data_smithi114_288325_object65113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-14T04:25:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:38.274425+0000", 2024-09-14T04:25:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.015284483, 2024-09-14T04:25:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015478353, 2024-09-14T04:25:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:25:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160178 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274425+0000", 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274425+0000", 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274424+0000", 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274428+0000", 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8380000000000004e-06 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274436+0000", 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2800000000000003e-06 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274441+0000", 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2100000000000001e-06 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274486+0000", 2024-09-14T04:25:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4637e-05 2024-09-14T04:25:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:25:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274599+0000", 2024-09-14T04:25:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113087 2024-09-14T04:25:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 4,5,7", 2024-09-14T04:25:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.274718+0000", 2024-09-14T04:25:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119268 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.277427+0000", 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002708097 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.286432+0000", 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090057039999999998 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.286465+0000", 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2387e-05 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:160180 13.4 13:247cd7dc:::benchmark_data_smithi114_288325_object65115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:38.277375+0000", 2024-09-14T04:25:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.012334189000000001, 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012610342, 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 160180 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.277375+0000", 2024-09-14T04:25:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.277375+0000", 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.277374+0000", 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.277380+0000", 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5459999999999997e-06 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.277392+0000", 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2132e-05 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.277398+0000", 2024-09-14T04:25:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3789999999999997e-06 2024-09-14T04:25:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:25:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.278601+0000", 2024-09-14T04:25:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012028500000000001 2024-09-14T04:25:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:25:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.278764+0000", 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016370600000000001 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,1,5", 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.278941+0000", 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176739 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285162+0000", 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0062211760000000001 2024-09-14T04:25:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285410+0000", 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024760800000000002 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285535+0000", 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012485299999999999 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:160181 13.a e359/351 13:55de7d00:::benchmark_data_smithi114_288325_object65116:head v 359'9907, mlcod=341'5887)", 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:38.282709+0000", 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0070000959999999999, 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0073551620000000002, 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.282709+0000", 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.282709+0000", 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.282709+0000", 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.282713+0000", 2024-09-14T04:25:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9979999999999997e-06 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.282718+0000", 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0089999999999998e-06 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.282721+0000", 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0580000000000002e-06 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:25:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.282758+0000", 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6770000000000002e-05 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.282776+0000", 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7864e-05 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:160182 13.9 e359/332)", 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:38.285520+0000", 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0041893579999999998, 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0045929880000000001, 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285520+0000", 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285520+0000", 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285519+0000", 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285527+0000", 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9219999999999997e-06 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285546+0000", 2024-09-14T04:25:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9119000000000001e-05 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.285552+0000", 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.339e-06 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:160157 13.8 e359/352 13:195e76a3:::benchmark_data_smithi114_288325_object65092:head v 359'9948, mlcod=330'2910)", 2024-09-14T04:25:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:38.289698+0000", 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.1031e-05, 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00045618399999999999, 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.289698+0000", 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.289698+0000", 2024-09-14T04:25:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.289697+0000", 2024-09-14T04:25:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:25:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.289705+0000", 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0229999999999992e-06 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.289718+0000", 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2476e-05 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.289723+0000", 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.524e-06 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.289785+0000", 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1964999999999994e-05 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.289808+0000", 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3037e-05 2024-09-14T04:25:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:160166 13.9 e359/332 13:9e931c53:::benchmark_data_smithi114_288325_object65101:head v 359'9918, mlcod=330'3056)", 2024-09-14T04:25:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:38.263233+0000", 2024-09-14T04:25:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.026476367000000001, 2024-09-14T04:25:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.026975381999999999, 2024-09-14T04:25:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:25:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.263233+0000", 2024-09-14T04:25:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.263233+0000", 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.263232+0000", 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.263240+0000", 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5920000000000001e-06 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.263255+0000", 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4054e-05 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.263260+0000", 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8889999999999999e-06 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.263444+0000", 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018321900000000001 2024-09-14T04:25:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:38.263491+0000", 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6919000000000003e-05 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 8 2024-09-14T04:25:38.333 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:25:38.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:38.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:38.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:38.395 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:38.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:38.501 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:38.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:38.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:38.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:38.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:38.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:38.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:38.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:38.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:38.910 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:38.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:39.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:39.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:39.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:39.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:39.161 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:39.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:39.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:39.287 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:39.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:39.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:39.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:39.412 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:39.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:39.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:39.548 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:39.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:39.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:39.674 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:39.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:39.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:39.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:39.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:39.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:25:39.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:39.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:39.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:40.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:40.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:40.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:40.188 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:40.211 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:40.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:40.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:40.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:40.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:40.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:40.380 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:25:40.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:40.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:40.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:40.530 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:40 smithi114 ceph-mon[103029]: pgmap v1389: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 752 op/s; 17479/89730 objects degraded (19.480%); 17758/89730 objects misplaced (19.790%); 6.3 MiB/s, 157 objects/s recovering 2024-09-14T04:25:40.530 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:40 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 17479/89730 objects degraded (19.480%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:40.531 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:40 smithi114 ceph-mon[106969]: pgmap v1389: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 752 op/s; 17479/89730 objects degraded (19.480%); 17758/89730 objects misplaced (19.790%); 6.3 MiB/s, 157 objects/s recovering 2024-09-14T04:25:40.531 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:40 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 17479/89730 objects degraded (19.480%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:40.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:40.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:40.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:25:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:25:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:25:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:40 smithi160 ceph-mon[88407]: pgmap v1389: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 9.1 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 752 op/s; 17479/89730 objects degraded (19.480%); 17758/89730 objects misplaced (19.790%); 6.3 MiB/s, 157 objects/s recovering 2024-09-14T04:25:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:40 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 17479/89730 objects degraded (19.480%), 14 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:40.679 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:40.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:40.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:40.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:40.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:40.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:40.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:40.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:40.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:40.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:41.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:41.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:41.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:41.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:40 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:25:40.838+0000 7f3162f23640 -1 mon.a@0(leader).osd e359 definitely_dead 0 2024-09-14T04:25:41.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:41.157 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:41.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:41.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:41.220 INFO:teuthology.orchestra.run.smithi114.stderr:marked in osd.2. 2024-09-14T04:25:41.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:41.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:41.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:41.392 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:41.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:41.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:41.432 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:41 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3037188014' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-14T04:25:41.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:41 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3037188014' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-14T04:25:41.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:41.539 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:41.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:41.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:41 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3037188014' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-14T04:25:41.662 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:41.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:41.716 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:41.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:41.788 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:41.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:41.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:41.839 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:25:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:25:41] "GET /metrics HTTP/1.1" 200 36595 "" "Prometheus/2.51.0" 2024-09-14T04:25:41.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:41.913 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:41.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:42.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:42.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:42.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:42.139 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:42.164 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:42.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:42.266 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:42.301 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:42.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:42.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:42 smithi114 ceph-mon[103029]: pgmap v1390: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 608 op/s; 17479/89124 objects degraded (19.612%); 17758/89124 objects misplaced (19.925%); 4.2 MiB/s, 116 objects/s recovering 2024-09-14T04:25:42.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3037188014' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-14T04:25:42.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:42 smithi114 ceph-mon[103029]: osdmap e360: 8 total, 8 up, 6 in 2024-09-14T04:25:42.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:42 smithi114 ceph-mon[106969]: pgmap v1390: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 608 op/s; 17479/89124 objects degraded (19.612%); 17758/89124 objects misplaced (19.925%); 4.2 MiB/s, 116 objects/s recovering 2024-09-14T04:25:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3037188014' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-14T04:25:42.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:42 smithi114 ceph-mon[106969]: osdmap e360: 8 total, 8 up, 6 in 2024-09-14T04:25:42.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:42.426 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:42.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:42.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-14T04:25:42.495 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:42 smithi160 ceph-mon[88407]: pgmap v1390: 99 pgs: 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 6 active+recovery_wait+degraded, 3 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 1.8 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 608 op/s; 17479/89124 objects degraded (19.612%); 17758/89124 objects misplaced (19.925%); 4.2 MiB/s, 116 objects/s recovering 2024-09-14T04:25:42.495 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3037188014' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-14T04:25:42.495 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:42 smithi160 ceph-mon[88407]: osdmap e360: 8 total, 8 up, 6 in 2024-09-14T04:25:42.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:42.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:42.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:42.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:42.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:42.770 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:25:42.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:42.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:42.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:42.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:42.922 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:42.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:43.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:43.045 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:43.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:43.137 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-14T04:25:43.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:43.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:43.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:43.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:43.299 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:43.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:43.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[106969]: 13.f starting backfill to osd.2 from (0'0,330'3425] MAX to 359'10228 2024-09-14T04:25:43.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[106969]: osdmap e361: 8 total, 8 up, 6 in 2024-09-14T04:25:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[106969]: 13.1 starting backfill to osd.2 from (0'0,0'0] MAX to 359'10313 2024-09-14T04:25:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[106969]: 13.c starting backfill to osd.2 from (0'0,330'3440] MAX to 359'10123 2024-09-14T04:25:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[106969]: 13.8 starting backfill to osd.2 from (0'0,330'3397] MAX to 359'10069 2024-09-14T04:25:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[106969]: 13.c continuing backfill to osd.4 from (336'4534,359'10123] MIN to 359'10123 2024-09-14T04:25:43.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[106969]: osdmap e362: 8 total, 8 up, 6 in 2024-09-14T04:25:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[103029]: 13.f starting backfill to osd.2 from (0'0,330'3425] MAX to 359'10228 2024-09-14T04:25:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[103029]: osdmap e361: 8 total, 8 up, 6 in 2024-09-14T04:25:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[103029]: 13.1 starting backfill to osd.2 from (0'0,0'0] MAX to 359'10313 2024-09-14T04:25:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[103029]: 13.c starting backfill to osd.2 from (0'0,330'3440] MAX to 359'10123 2024-09-14T04:25:43.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[103029]: 13.8 starting backfill to osd.2 from (0'0,330'3397] MAX to 359'10069 2024-09-14T04:25:43.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[103029]: 13.c continuing backfill to osd.4 from (336'4534,359'10123] MIN to 359'10123 2024-09-14T04:25:43.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:43 smithi114 ceph-mon[103029]: osdmap e362: 8 total, 8 up, 6 in 2024-09-14T04:25:43.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:43.424 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:43.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:43.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:43.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:43.547 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:43.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:43.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:43 smithi160 ceph-mon[88407]: 13.f starting backfill to osd.2 from (0'0,330'3425] MAX to 359'10228 2024-09-14T04:25:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:43 smithi160 ceph-mon[88407]: osdmap e361: 8 total, 8 up, 6 in 2024-09-14T04:25:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:43 smithi160 ceph-mon[88407]: 13.1 starting backfill to osd.2 from (0'0,0'0] MAX to 359'10313 2024-09-14T04:25:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:43 smithi160 ceph-mon[88407]: 13.c starting backfill to osd.2 from (0'0,330'3440] MAX to 359'10123 2024-09-14T04:25:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:43 smithi160 ceph-mon[88407]: 13.8 starting backfill to osd.2 from (0'0,330'3397] MAX to 359'10069 2024-09-14T04:25:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:43 smithi160 ceph-mon[88407]: 13.c continuing backfill to osd.4 from (336'4534,359'10123] MIN to 359'10123 2024-09-14T04:25:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:43 smithi160 ceph-mon[88407]: osdmap e362: 8 total, 8 up, 6 in 2024-09-14T04:25:43.677 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:43.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:43.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:43.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:43.809 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:43.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:43.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:43.945 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:43.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:44.047 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:44.093 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:44.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:44.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:44.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:44.230 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:44.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:44.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:44.364 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:44.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:44.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:44 smithi114 ceph-mon[103029]: pgmap v1393: 99 pgs: 2 remapped+peering, 2 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 490 op/s; 11799/87522 objects degraded (13.481%); 10679/87522 objects misplaced (12.202%); 6.0 MiB/s, 123 objects/s recovering 2024-09-14T04:25:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:44 smithi114 ceph-mon[103029]: 13.9 starting backfill to osd.2 from (0'0,330'3419] MAX to 359'10050 2024-09-14T04:25:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:44 smithi114 ceph-mon[103029]: 13.b starting backfill to osd.2 from (0'0,330'3478] MAX to 359'10204 2024-09-14T04:25:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:44 smithi114 ceph-mon[103029]: 13.4 starting backfill to osd.2 from (0'0,330'3445] MAX to 359'10208 2024-09-14T04:25:44.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:44 smithi114 ceph-mon[103029]: 13.a continuing backfill to osd.3 from (329'1900,359'10014] MIN to 359'10014 2024-09-14T04:25:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:44 smithi114 ceph-mon[106969]: pgmap v1393: 99 pgs: 2 remapped+peering, 2 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 490 op/s; 11799/87522 objects degraded (13.481%); 10679/87522 objects misplaced (12.202%); 6.0 MiB/s, 123 objects/s recovering 2024-09-14T04:25:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:44 smithi114 ceph-mon[106969]: 13.9 starting backfill to osd.2 from (0'0,330'3419] MAX to 359'10050 2024-09-14T04:25:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:44 smithi114 ceph-mon[106969]: 13.b starting backfill to osd.2 from (0'0,330'3478] MAX to 359'10204 2024-09-14T04:25:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:44 smithi114 ceph-mon[106969]: 13.4 starting backfill to osd.2 from (0'0,330'3445] MAX to 359'10208 2024-09-14T04:25:44.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:44 smithi114 ceph-mon[106969]: 13.a continuing backfill to osd.3 from (329'1900,359'10014] MIN to 359'10014 2024-09-14T04:25:44.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:44.489 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:44.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:44 smithi160 ceph-mon[88407]: pgmap v1393: 99 pgs: 2 remapped+peering, 2 active+recovering+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 5 active+recovery_wait+degraded, 2 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 490 op/s; 11799/87522 objects degraded (13.481%); 10679/87522 objects misplaced (12.202%); 6.0 MiB/s, 123 objects/s recovering 2024-09-14T04:25:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:44 smithi160 ceph-mon[88407]: 13.9 starting backfill to osd.2 from (0'0,330'3419] MAX to 359'10050 2024-09-14T04:25:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:44 smithi160 ceph-mon[88407]: 13.b starting backfill to osd.2 from (0'0,330'3478] MAX to 359'10204 2024-09-14T04:25:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:44 smithi160 ceph-mon[88407]: 13.4 starting backfill to osd.2 from (0'0,330'3445] MAX to 359'10208 2024-09-14T04:25:44.560 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:44 smithi160 ceph-mon[88407]: 13.a continuing backfill to osd.3 from (329'1900,359'10014] MIN to 359'10014 2024-09-14T04:25:44.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:44.645 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:44.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:44.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:44.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:44.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:44.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:44.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:44.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:45.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:45.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:45.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:45.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:45.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:45.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:45.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:45.324 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:45.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:45.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:45.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:45.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:45.467 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:45 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 11799/87522 objects degraded (13.481%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:45.471 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:45 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 11799/87522 objects degraded (13.481%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:45.530 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:25:45.531 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:25:45.558 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:25:45.559 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:25:45.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:45.588 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:45 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 11799/87522 objects degraded (13.481%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:25:45.594 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:25:45.594 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:25:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:45.601 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:45.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:45.626 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:25:45.626 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:25:45.654 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:25:45.654 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:25:45.692 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:25:45.692 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:25:45.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:45.725 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:25:45.725 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:25:45.749 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:45.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:45.756 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:25:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:25:45.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:45.894 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:45.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:45.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:46.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:46.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:46.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:46.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:46.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:46.260 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:25:46.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:46.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:46.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:46.345 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:25:46.345 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:25:46.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:46 smithi114 ceph-mon[106969]: pgmap v1395: 99 pgs: 2 remapped+peering, 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 324 op/s; 10080/83880 objects degraded (12.017%); 11455/83880 objects misplaced (13.656%); 11 MiB/s, 191 objects/s recovering 2024-09-14T04:25:46.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1358278402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:46.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2749976888' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:25:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:46 smithi114 ceph-mon[103029]: pgmap v1395: 99 pgs: 2 remapped+peering, 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 324 op/s; 10080/83880 objects degraded (12.017%); 11455/83880 objects misplaced (13.656%); 11 MiB/s, 191 objects/s recovering 2024-09-14T04:25:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1358278402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:46.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2749976888' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:25:46.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:46.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:46.445 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:46.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:46.546 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:46.584 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:46.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:46 smithi160 ceph-mon[88407]: pgmap v1395: 99 pgs: 2 remapped+peering, 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 324 op/s; 10080/83880 objects degraded (12.017%); 11455/83880 objects misplaced (13.656%); 11 MiB/s, 191 objects/s recovering 2024-09-14T04:25:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1358278402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2749976888' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:25:46.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:46.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:46.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:46.812 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:46.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:46.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:46.946 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:46.975 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:46.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:47.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-14T04:25:47.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:47.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:47.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:47.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:47.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:47.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:47.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:47.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:47.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:47 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:25:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:47 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2749976888' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:25:47.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:47 smithi160 ceph-mon[88407]: osdmap e363: 8 total, 8 up, 6 in 2024-09-14T04:25:47.430 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:47.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:47.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:47.543 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:25:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:47.561 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:47.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:47.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:47 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:25:47.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2749976888' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:25:47.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:47 smithi114 ceph-mon[103029]: osdmap e363: 8 total, 8 up, 6 in 2024-09-14T04:25:47.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:47 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:25:47.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:47 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2749976888' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:25:47.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:47 smithi114 ceph-mon[106969]: osdmap e363: 8 total, 8 up, 6 in 2024-09-14T04:25:47.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:47.698 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:47.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:47.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:47.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:47.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:47.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:47.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:47.972 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:47.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:48.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:48.101 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:48.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:48.138 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 4, 5, 2] out_osds: [1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:25:48.138 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:25:48.138 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-14T04:25:48.138 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-09-14T04:25:48.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 0 2024-09-14T04:25:48.203 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:48.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:48.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:48.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:48.381 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:48.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:48.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:48.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:48 smithi160 ceph-mon[88407]: pgmap v1397: 99 pgs: 2 remapped+peering, 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 294 op/s; 10080/83880 objects degraded (12.017%); 11455/83880 objects misplaced (13.656%); 11 MiB/s, 193 objects/s recovering 2024-09-14T04:25:48.483 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:48.520 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:48.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:48.532 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:48.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:25:48.603 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:48 smithi114 ceph-mon[103029]: pgmap v1397: 99 pgs: 2 remapped+peering, 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 294 op/s; 10080/83880 objects degraded (12.017%); 11455/83880 objects misplaced (13.656%); 11 MiB/s, 193 objects/s recovering 2024-09-14T04:25:48.604 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:48 smithi114 ceph-mon[106969]: pgmap v1397: 99 pgs: 2 remapped+peering, 1 active+recovering+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 1.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 294 op/s; 10080/83880 objects degraded (12.017%); 11455/83880 objects misplaced (13.656%); 11 MiB/s, 193 objects/s recovering 2024-09-14T04:25:48.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:48.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:48.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:48.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:48.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:48.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:48.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:48.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:48.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:49.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:49.069 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:49.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:49.171 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:49.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:49.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:49.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:49.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:49.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:49.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:25:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:49.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:49.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:49.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:49.583 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:49.614 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:49.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:49.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:49.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:49.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:49.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:49.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:49.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:49.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:50.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:50.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:50.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:50.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:50.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:50.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:50.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:50.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:50.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:50.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:50.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:50.456 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:50.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:50.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:50.489 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:50 smithi114 ceph-mon[103029]: pgmap v1398: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 84 active+clean; 1.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 467 op/s; 13893/79398 objects degraded (17.498%); 21913/79398 objects misplaced (27.599%); 13 MiB/s, 258 objects/s recovering 2024-09-14T04:25:50.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:50 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 13893/79398 objects degraded (17.498%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T04:25:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:50 smithi114 ceph-mon[106969]: pgmap v1398: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 84 active+clean; 1.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 467 op/s; 13893/79398 objects degraded (17.498%); 21913/79398 objects misplaced (27.599%); 13 MiB/s, 258 objects/s recovering 2024-09-14T04:25:50.490 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:50 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 13893/79398 objects degraded (17.498%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T04:25:50.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:50.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:50.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:50 smithi160 ceph-mon[88407]: pgmap v1398: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 84 active+clean; 1.6 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 467 op/s; 13893/79398 objects degraded (17.498%); 21913/79398 objects misplaced (27.599%); 13 MiB/s, 258 objects/s recovering 2024-09-14T04:25:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:50 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 13893/79398 objects degraded (17.498%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T04:25:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:25:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:25:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:25:50.687 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:50.718 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:50.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:50.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:50.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:50.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:50.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:50.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:50.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:51.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:51.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:51.126 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:51.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:51.226 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:51.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:51.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:51.378 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:51.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:51.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:51.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:51.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:51.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:51.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:51.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:51.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:51.786 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:25:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:25:51] "GET /metrics HTTP/1.1" 200 36595 "" "Prometheus/2.51.0" 2024-09-14T04:25:51.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:51.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:51.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:51.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:51.981 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:51.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:51.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:52.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:52.124 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:52.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:52.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:52.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:52.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:52.286 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.4 primary-affinity to 0 (802) 2024-09-14T04:25:52.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:52.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:52 smithi114 ceph-mon[106969]: pgmap v1399: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 84 active+clean; 1.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 551 op/s; 13014/74274 objects degraded (17.522%); 21580/74274 objects misplaced (29.055%); 10 MiB/s, 210 objects/s recovering 2024-09-14T04:25:52.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:52 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/754966113' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-14T04:25:52.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:52 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-14T04:25:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:52 smithi114 ceph-mon[103029]: pgmap v1399: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 84 active+clean; 1.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 551 op/s; 13014/74274 objects degraded (17.522%); 21580/74274 objects misplaced (29.055%); 10 MiB/s, 210 objects/s recovering 2024-09-14T04:25:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:52 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/754966113' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-14T04:25:52.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:52 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-14T04:25:52.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:52.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:52.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:52 smithi160 ceph-mon[88407]: pgmap v1399: 99 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 84 active+clean; 1.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 551 op/s; 13014/74274 objects degraded (17.522%); 21580/74274 objects misplaced (29.055%); 10 MiB/s, 210 objects/s recovering 2024-09-14T04:25:52.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:52 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/754966113' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-14T04:25:52.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:52 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-14T04:25:52.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:52.505 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:52.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:52.529 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:25:52.529 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:25:52.547 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:25:52.548 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:25:52.567 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:25:52.567 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:25:52.590 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:25:52.590 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:25:52.607 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:25:52.607 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:25:52.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:52.623 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:25:52.624 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:25:52.630 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:52.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:52.644 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:25:52.644 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:25:52.661 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:25:52.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:25:52.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:52.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:52.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:52.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:52.879 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:52.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:52.981 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:53.006 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:53.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:53.021 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:25:53.021 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:25:53.041 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:53.107 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:53.130 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:53.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:53.231 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:53.254 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:53.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:53.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:53.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:53.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:53.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:53.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:53.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:53.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:53.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:53 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-14T04:25:53.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:53 smithi114 ceph-mon[103029]: osdmap e364: 8 total, 8 up, 6 in 2024-09-14T04:25:53.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2104024751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:53.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:53.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:53.583 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:53 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-14T04:25:53.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:53 smithi114 ceph-mon[106969]: osdmap e364: 8 total, 8 up, 6 in 2024-09-14T04:25:53.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2104024751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:53.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:53.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:53.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:53.637 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:53.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:53 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-14T04:25:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:53 smithi160 ceph-mon[88407]: osdmap e364: 8 total, 8 up, 6 in 2024-09-14T04:25:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2104024751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:25:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:25:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:25:53.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:53.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:53.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:53.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:53.887 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:53.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:53.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:54.011 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:54.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:54.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:54.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:54.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:54.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:54.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:54.259 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:54.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:54.387 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:54.410 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:54.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:54.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:54 smithi114 ceph-mon[103029]: pgmap v1401: 99 pgs: 1 remapped+peering, 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 82 active+clean; 1.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 475 op/s; 11571/72453 objects degraded (15.970%); 18604/72453 objects misplaced (25.677%); 6.7 MiB/s, 180 objects/s recovering 2024-09-14T04:25:54.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:54 smithi114 ceph-mon[103029]: osdmap e365: 8 total, 8 up, 6 in 2024-09-14T04:25:54.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:54 smithi114 ceph-mon[103029]: osdmap e366: 8 total, 8 up, 6 in 2024-09-14T04:25:54.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:54 smithi114 ceph-mon[106969]: pgmap v1401: 99 pgs: 1 remapped+peering, 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 82 active+clean; 1.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 475 op/s; 11571/72453 objects degraded (15.970%); 18604/72453 objects misplaced (25.677%); 6.7 MiB/s, 180 objects/s recovering 2024-09-14T04:25:54.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:54 smithi114 ceph-mon[106969]: osdmap e365: 8 total, 8 up, 6 in 2024-09-14T04:25:54.435 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:54 smithi114 ceph-mon[106969]: osdmap e366: 8 total, 8 up, 6 in 2024-09-14T04:25:54.512 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:54.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:54.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:54.556 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:54 smithi160 ceph-mon[88407]: pgmap v1401: 99 pgs: 1 remapped+peering, 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 3 active+recovery_wait+degraded, 82 active+clean; 1.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 475 op/s; 11571/72453 objects degraded (15.970%); 18604/72453 objects misplaced (25.677%); 6.7 MiB/s, 180 objects/s recovering 2024-09-14T04:25:54.556 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:54 smithi160 ceph-mon[88407]: osdmap e365: 8 total, 8 up, 6 in 2024-09-14T04:25:54.556 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:54 smithi160 ceph-mon[88407]: osdmap e366: 8 total, 8 up, 6 in 2024-09-14T04:25:54.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-14T04:25:54.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:54.658 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:54.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:54.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:54.760 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:54.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:54.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:54.866 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:25:54.886 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:54.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:54.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:55.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:55.071 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:55.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:55.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:55.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:55.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:55.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:55.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:55.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:55.441 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:55.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:55.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:55.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:55 smithi114 ceph-mon[103029]: 13.d starting backfill to osd.5 from (341'5993,363'10617] MIN to 363'10618 2024-09-14T04:25:55.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:55 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 11571/72453 objects degraded (15.970%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T04:25:55.555 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:55 smithi114 ceph-mon[106969]: 13.d starting backfill to osd.5 from (341'5993,363'10617] MIN to 363'10618 2024-09-14T04:25:55.555 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:55 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 11571/72453 objects degraded (15.970%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T04:25:55.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:55.595 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:55.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:55.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:25:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:55 smithi160 ceph-mon[88407]: 13.d starting backfill to osd.5 from (341'5993,363'10617] MIN to 363'10618 2024-09-14T04:25:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:55 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 11571/72453 objects degraded (15.970%), 11 pgs degraded (PG_DEGRADED) 2024-09-14T04:25:55.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:55.726 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:55.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:55.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:55.828 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:55.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:55.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:55.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:55.977 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:55.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:56.045 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:25:56.079 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:56.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:56.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:56.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:56.226 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:56.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:56.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:56.345 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:25:56.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:56.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:56.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:56.489 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:56.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:56.539 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:56 smithi160 ceph-mon[88407]: pgmap v1404: 99 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 83 active+clean; 1.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 711 op/s; 12222/66585 objects degraded (18.355%); 18991/66585 objects misplaced (28.521%); 15 MiB/s, 298 objects/s recovering 2024-09-14T04:25:56.540 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:56 smithi160 ceph-mon[88407]: osdmap e367: 8 total, 8 up, 6 in 2024-09-14T04:25:56.540 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:56 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2062523677' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:25:56.591 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:56 smithi114 ceph-mon[103029]: pgmap v1404: 99 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 83 active+clean; 1.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 711 op/s; 12222/66585 objects degraded (18.355%); 18991/66585 objects misplaced (28.521%); 15 MiB/s, 298 objects/s recovering 2024-09-14T04:25:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:56 smithi114 ceph-mon[103029]: osdmap e367: 8 total, 8 up, 6 in 2024-09-14T04:25:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:56 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2062523677' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:25:56.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:56 smithi114 ceph-mon[106969]: pgmap v1404: 99 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 83 active+clean; 1.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 711 op/s; 12222/66585 objects degraded (18.355%); 18991/66585 objects misplaced (28.521%); 15 MiB/s, 298 objects/s recovering 2024-09-14T04:25:56.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:56 smithi114 ceph-mon[106969]: osdmap e367: 8 total, 8 up, 6 in 2024-09-14T04:25:56.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:56 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2062523677' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:25:56.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:56.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:56.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:56.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:56.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:56.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:56.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:56.881 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:25:56.881 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:25:56.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.881 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:168864 13.8 e368/360 13:1d49f240:::benchmark_data_smithi114_288325_object73799:head v 368'10499, mlcod=0'0)", 2024-09-14T04:25:56.881 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:56.867507+0000", 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0090156219999999992, 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0090437520000000004, 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.867507+0000", 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.882 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.867507+0000", 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.867506+0000", 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:56.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.867516+0000", 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0373e-05 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.867538+0000", 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2279000000000001e-05 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.867547+0000", 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2859999999999999e-06 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.867642+0000", 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5705999999999995e-05 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.867669+0000", 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6347e-05 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:56.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:168872 13.c e368/360 13:3c29a613:::benchmark_data_smithi114_288325_object73807:head v 368'10529, mlcod=341'5942)", 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:56.873678+0000", 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0028445319999999999, 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029331579999999999, 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873678+0000", 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:56.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873678+0000", 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873677+0000", 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:56.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873687+0000", 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0438999999999999e-05 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873707+0000", 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9368000000000002e-05 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873714+0000", 2024-09-14T04:25:56.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7190000000000001e-06 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.874187+0000", 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000472304 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.874248+0000", 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1243999999999998e-05 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:56.889 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(13.8 368/360 [PushOp(13:129c7fbe:::benchmark_data_smithi114_288325_object73804:head, version: 336'4616, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(13:129c7fbe:::benchmark_data_smithi114_288325_object73804:head@336'4616, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:56.873729+0000", 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0027935170000000001, 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002956682, 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.890 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873729+0000", 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873729+0000", 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873729+0000", 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873765+0000", 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5899000000000003e-05 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873778+0000", 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2869e-05 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.892 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.873785+0000", 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7670000000000003e-06 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.874852+0000", 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010669589999999999 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:25:56.893 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:56.874876+0000", 2024-09-14T04:25:56.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3773000000000001e-05 2024-09-14T04:25:56.894 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:56.894 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:25:56.894 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:56.894 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:25:56.894 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:25:56.894 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 3 2024-09-14T04:25:56.894 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:25:56.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:56.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:56.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:57.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:57.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:57.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:57.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:57.190 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:57.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:57.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:57.322 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:57.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:57.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:57.424 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:57.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:57.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:57.557 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:57.579 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:57 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:57.579 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:57 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2062523677' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:25:57.579 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:57 smithi160 ceph-mon[88407]: osdmap e368: 8 total, 8 up, 6 in 2024-09-14T04:25:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:57.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:57.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:57.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:57 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:57.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:57 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2062523677' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:25:57.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:57 smithi114 ceph-mon[103029]: osdmap e368: 8 total, 8 up, 6 in 2024-09-14T04:25:57.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:57 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:25:57.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:57 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2062523677' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:25:57.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:57 smithi114 ceph-mon[106969]: osdmap e368: 8 total, 8 up, 6 in 2024-09-14T04:25:57.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-14T04:25:57.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:57.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:57.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:57.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:25:57.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:57.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:25:58.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:58.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:58.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:58.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:58.136 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:25:58.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:25:58.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:58.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:25:58.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:58.299 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:58.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:58.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:58.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:58.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:58.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:58.581 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:58.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:58.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:58.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:25:58 smithi114 ceph-mon[103029]: pgmap v1407: 99 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 83 active+clean; 1.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 488 op/s; 12222/66585 objects degraded (18.355%); 18991/66585 objects misplaced (28.521%); 16 MiB/s, 247 objects/s recovering 2024-09-14T04:25:58.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:25:58 smithi114 ceph-mon[106969]: pgmap v1407: 99 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 83 active+clean; 1.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 488 op/s; 12222/66585 objects degraded (18.355%); 18991/66585 objects misplaced (28.521%); 16 MiB/s, 247 objects/s recovering 2024-09-14T04:25:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:25:58 smithi160 ceph-mon[88407]: pgmap v1407: 99 pgs: 1 activating+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 83 active+clean; 1.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 488 op/s; 12222/66585 objects degraded (18.355%); 18991/66585 objects misplaced (28.521%); 16 MiB/s, 247 objects/s recovering 2024-09-14T04:25:58.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:25:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:58.726 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:58.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:25:58.828 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:25:58.862 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:58.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:25:58.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:58.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:58.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:25:59.099 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:59.136 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:59.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:59.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:59.274 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:59.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:59.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:25:59.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:59.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:25:59.411 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:25:59.411 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:25:59.435 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:25:59.436 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:25:59.458 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:25:59.458 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:25:59.484 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:25:59.484 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:25:59.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:25:59.519 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:25:59.519 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:25:59.540 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:59.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:25:59.549 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:25:59.549 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:25:59.578 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:25:59.578 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:25:59.602 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:25:59.602 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:25:59.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:59.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:59.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:59.722 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:25:59.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:25:59.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:25:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:59.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:25:59.830 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 6, 4, 5, 2] out_osds: [1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:25:59.830 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:25:59.830 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 3, 6, 4, 5, 2] 2024-09-14T04:25:59.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd out 0 2024-09-14T04:25:59.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:25:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:25:59.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:25:59.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:00.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:00.102 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:00.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:00.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:00.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:00.233 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:00.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:00.309 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:26:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:26:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:26:00.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:00.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:00.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:00.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:00.503 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:00.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:00.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:00.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:00 smithi114 ceph-mon[103029]: pgmap v1408: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 1.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 463 op/s; 11835/64296 objects degraded (18.407%); 17639/64296 objects misplaced (27.434%); 20 MiB/s, 326 objects/s recovering 2024-09-14T04:26:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4032165529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:26:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:00 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 11835/64296 objects degraded (18.407%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:00.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:00 smithi114 ceph-mon[106969]: pgmap v1408: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 1.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 463 op/s; 11835/64296 objects degraded (18.407%); 17639/64296 objects misplaced (27.434%); 20 MiB/s, 326 objects/s recovering 2024-09-14T04:26:00.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4032165529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:26:00.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:00 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 11835/64296 objects degraded (18.407%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:00.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:00.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:00 smithi160 ceph-mon[88407]: pgmap v1408: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+degraded+remapped, 1 active+recovery_wait, 6 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 1.3 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 463 op/s; 11835/64296 objects degraded (18.407%); 17639/64296 objects misplaced (27.434%); 20 MiB/s, 326 objects/s recovering 2024-09-14T04:26:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4032165529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:26:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:00 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 11835/64296 objects degraded (18.407%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:00.673 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:00.681 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:00.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:00.777 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:00.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:00.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:00.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:00.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:01.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:01.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:01.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:01.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:01.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:01.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:01.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:01.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:01.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:01.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:01.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:01.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:01.602 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:01.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:01.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:01.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:01.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:01.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:01.758 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:26:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:26:01] "GET /metrics HTTP/1.1" 200 36605 "" "Prometheus/2.51.0" 2024-09-14T04:26:01.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:01.865 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:01.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:01.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:02.005 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:02.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:02.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:02.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:02.180 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:26:02.180 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:26:02.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:02.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:02.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:02.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:02 smithi160 ceph-mon[88407]: pgmap v1409: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 1.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 371 op/s; 10974/59373 objects degraded (18.483%); 17369/59373 objects misplaced (29.254%); 9.8 MiB/s, 157 objects/s recovering 2024-09-14T04:26:02.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:02.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:02.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:02.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:02.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:26:02.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:02.584 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:02.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:02.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:02 smithi114 ceph-mon[103029]: pgmap v1409: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 1.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 371 op/s; 10974/59373 objects degraded (18.483%); 17369/59373 objects misplaced (29.254%); 9.8 MiB/s, 157 objects/s recovering 2024-09-14T04:26:02.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:02 smithi114 ceph-mon[106969]: pgmap v1409: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 1.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 371 op/s; 10974/59373 objects degraded (18.483%); 17369/59373 objects misplaced (29.254%); 9.8 MiB/s, 157 objects/s recovering 2024-09-14T04:26:02.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:02.729 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:02.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:02.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:02.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:02.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:02.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:02.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-14T04:26:02.984 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:02.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:03.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:03.116 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:03.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:03.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:03.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:03.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:03.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:03.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:03.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:03.477 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:26:03.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:03.532 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:03.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:03.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:03.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:03.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:03.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:03.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:03.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:03.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:03.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:03.917 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:03.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:03.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:04.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:04.040 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:04.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:04.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:04.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:04.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:04.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:04.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:04.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:04.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:04.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:04.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:04.591 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:04.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:04.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:04 smithi114 ceph-mon[106969]: pgmap v1410: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 1.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 355 op/s; 10974/58062 objects degraded (18.900%); 17089/58062 objects misplaced (29.432%); 9.5 MiB/s, 152 objects/s recovering 2024-09-14T04:26:04.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:04 smithi114 ceph-mon[103029]: pgmap v1410: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 1.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 355 op/s; 10974/58062 objects degraded (18.900%); 17089/58062 objects misplaced (29.432%); 9.5 MiB/s, 152 objects/s recovering 2024-09-14T04:26:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:04 smithi160 ceph-mon[88407]: pgmap v1410: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 1.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 355 op/s; 10974/58062 objects degraded (18.900%); 17089/58062 objects misplaced (29.432%); 9.5 MiB/s, 152 objects/s recovering 2024-09-14T04:26:04.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:04.727 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:04.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:04.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:04.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:04.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:04.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:04.968 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:05.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:05.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:05.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:05.156 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:05.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:05.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:05.295 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:05.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:05.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:05.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:05.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:05.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:05.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:05.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:05.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:05.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:05 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 10974/58062 objects degraded (18.900%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:05.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:05.693 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:05.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:05.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:05.821 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:05.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:05.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:05 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 10974/58062 objects degraded (18.900%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:05.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:05 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 10974/58062 objects degraded (18.900%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:05.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:05.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:05.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:06.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:06.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:06.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:06.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:06.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:06.234 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:06.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:06.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:06.361 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:06.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:06.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:06.418 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:26:06.419 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:26:06.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:06.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:06.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:06.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:06.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:06.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:06.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:06 smithi160 ceph-mon[88407]: pgmap v1411: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 1.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 579 op/s; 9530/49446 objects degraded (19.274%); 16531/49446 objects misplaced (33.432%); 9.1 MiB/s, 145 objects/s recovering 2024-09-14T04:26:06.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:06.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:06.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:06.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:06.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:06 smithi114 ceph-mon[106969]: pgmap v1411: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 1.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 579 op/s; 9530/49446 objects degraded (19.274%); 16531/49446 objects misplaced (33.432%); 9.1 MiB/s, 145 objects/s recovering 2024-09-14T04:26:06.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:06 smithi114 ceph-mon[103029]: pgmap v1411: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 1.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 579 op/s; 9530/49446 objects degraded (19.274%); 16531/49446 objects misplaced (33.432%); 9.1 MiB/s, 145 objects/s recovering 2024-09-14T04:26:06.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:06.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:06.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:06.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:07.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:07.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:07.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:07.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:07.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:07.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:07.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:07.320 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:07.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:07.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:07.455 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:07.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:07.557 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:07.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:07.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:07.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:07.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:07.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:07.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:07.882 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:07.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:07.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:08.025 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:08.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:08.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:08.163 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:08.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:08.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:08.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:26:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:08.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:08.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:08.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:08.436 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:08.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:08.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:08.568 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:08.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:08.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:08.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:08.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:08.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:08.721 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:26:08.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:08.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:08.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:08 smithi114 ceph-mon[106969]: pgmap v1412: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 1.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 528 op/s; 9530/49446 objects degraded (19.274%); 16531/49446 objects misplaced (33.432%); 8.3 MiB/s, 132 objects/s recovering 2024-09-14T04:26:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:08 smithi114 ceph-mon[103029]: pgmap v1412: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 1.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 528 op/s; 9530/49446 objects degraded (19.274%); 16531/49446 objects misplaced (33.432%); 8.3 MiB/s, 132 objects/s recovering 2024-09-14T04:26:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:08.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:08.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:08.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:09.053 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:08 smithi160 ceph-mon[88407]: pgmap v1412: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 1.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 528 op/s; 9530/49446 objects degraded (19.274%); 16531/49446 objects misplaced (33.432%); 8.3 MiB/s, 132 objects/s recovering 2024-09-14T04:26:09.053 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:09.054 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:09.087 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:09.119 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:09.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:09.221 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:09.262 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:09.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:09.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:09.405 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:09.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:09.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:09.555 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:09.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:09.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:09.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:09.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:09.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:09.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:09.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:09.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:09.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:09.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:10.044 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:10.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:10.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:10.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 9530/49446 objects degraded (19.274%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:10.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 9530/49446 objects degraded (19.274%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 9530/49446 objects degraded (19.274%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:10.166 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:10.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:10.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:10.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:10.334 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:10.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:10.436 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:10.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:10.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:10.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:10.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:10.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:26:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:26:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:26:10.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:10.746 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:10.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:10.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:10.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:10 smithi114 ceph-mon[106969]: pgmap v1413: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 947 MiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 585 op/s; 9065/45507 objects degraded (19.920%); 15979/45507 objects misplaced (35.113%); 8.3 MiB/s, 133 objects/s recovering 2024-09-14T04:26:10.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:10 smithi114 ceph-mon[103029]: pgmap v1413: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 947 MiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 585 op/s; 9065/45507 objects degraded (19.920%); 15979/45507 objects misplaced (35.113%); 8.3 MiB/s, 133 objects/s recovering 2024-09-14T04:26:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:10.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:10.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:10.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:11.025 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:11.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:11.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:10 smithi160 ceph-mon[88407]: pgmap v1413: 99 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 947 MiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 585 op/s; 9065/45507 objects degraded (19.920%); 15979/45507 objects misplaced (35.113%); 8.3 MiB/s, 133 objects/s recovering 2024-09-14T04:26:11.159 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:11.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:11.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:11.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:11.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:11.432 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:11.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:11.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:11.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:11.605 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:11.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:11.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:11.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:11 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/35996313' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:11.694 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:11 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:11.697 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:26:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:26:11] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-14T04:26:11.702 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:26:11.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:11.761 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:11.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:11.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:11.893 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:26:11.893 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:26:11.904 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:11.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:11.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:11 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/35996313' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:11.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:11 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:12.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:12.045 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:12.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:12.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:12.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:11 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/35996313' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:12.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:11 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:12.182 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:12.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:12.284 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:12.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:12.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:12.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:12.460 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:12.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:12.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:12.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-14T04:26:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:12.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:12.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:12.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:12.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:12.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:12.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:12.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:12.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:12 smithi160 ceph-mon[88407]: pgmap v1414: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 805 MiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 711 op/s; 7908/38682 objects degraded (20.444%); 15581/38682 objects misplaced (40.280%); 4.7 MiB/s, 75 objects/s recovering 2024-09-14T04:26:12.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:12 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:26:12.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:12 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:26:12.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:12 smithi160 ceph-mon[88407]: osdmap e369: 8 total, 8 up, 6 in 2024-09-14T04:26:12.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:12.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:13.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:13.034 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:12 smithi114 ceph-mon[103029]: pgmap v1414: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 805 MiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 711 op/s; 7908/38682 objects degraded (20.444%); 15581/38682 objects misplaced (40.280%); 4.7 MiB/s, 75 objects/s recovering 2024-09-14T04:26:13.035 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:12 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:26:13.035 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:12 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:26:13.035 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:12 smithi114 ceph-mon[103029]: osdmap e369: 8 total, 8 up, 6 in 2024-09-14T04:26:13.035 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:12 smithi114 ceph-mon[106969]: pgmap v1414: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 805 MiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 711 op/s; 7908/38682 objects degraded (20.444%); 15581/38682 objects misplaced (40.280%); 4.7 MiB/s, 75 objects/s recovering 2024-09-14T04:26:13.035 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:12 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:26:13.035 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:12 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:26:13.035 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:12 smithi114 ceph-mon[106969]: osdmap e369: 8 total, 8 up, 6 in 2024-09-14T04:26:13.076 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:26:13.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:13.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:13.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:13.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:13.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:13.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:13.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:13.372 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:13.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:13.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:13.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:13.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:13.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:13.644 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:13.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:13.746 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:13.783 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:13.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:13.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:13.927 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:13.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:14.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:14.064 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:14.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:14.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:14.200 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:14.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:14.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:14.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:14.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:14.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:14.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:14.493 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:14.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:14.594 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:14.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:14 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:26:14.559+0000 7f3162f23640 -1 mon.a@0(leader).osd e369 definitely_dead 0 2024-09-14T04:26:14.632 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:14.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:14.703 INFO:teuthology.orchestra.run.smithi114.stderr:marked out osd.0. 2024-09-14T04:26:14.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:14.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:14.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:14.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:14.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:14.891 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:14.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:14.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:14.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:15.010 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:14 smithi160 ceph-mon[88407]: pgmap v1416: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 776 MiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 691 op/s; 7925/37314 objects degraded (21.239%); 15250/37314 objects misplaced (40.869%); 4.8 MiB/s, 76 objects/s recovering 2024-09-14T04:26:15.010 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:14 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4186011603' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-14T04:26:15.010 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:14 smithi160 ceph-mon[88407]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-14T04:26:15.010 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:14 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 7925/37314 objects degraded (21.239%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:15.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:15.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:15.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:26:15.105 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:26:15.105 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:26:15.106 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:14 smithi114 ceph-mon[103029]: pgmap v1416: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 776 MiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 691 op/s; 7925/37314 objects degraded (21.239%); 15250/37314 objects misplaced (40.869%); 4.8 MiB/s, 76 objects/s recovering 2024-09-14T04:26:15.106 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:14 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4186011603' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-14T04:26:15.107 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:14 smithi114 ceph-mon[103029]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-14T04:26:15.107 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:14 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 7925/37314 objects degraded (21.239%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:15.107 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:14 smithi114 ceph-mon[106969]: pgmap v1416: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 86 active+clean; 776 MiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 691 op/s; 7925/37314 objects degraded (21.239%); 15250/37314 objects misplaced (40.869%); 4.8 MiB/s, 76 objects/s recovering 2024-09-14T04:26:15.107 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:14 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4186011603' entity='client.admin' cmd={"prefix": "osd out", "ids": ["0"]} : dispatch 2024-09-14T04:26:15.107 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:14 smithi114 ceph-mon[106969]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-14T04:26:15.107 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:14 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 7925/37314 objects degraded (21.239%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:15.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:15.138 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:15.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:15.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:15.263 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:15.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:15.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:15.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:15.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:15.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:15.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:15.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:15.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:15.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:15.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:15.634 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:15.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:15.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:15.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:15.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:15.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:15.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:15.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:15.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:16.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:16.011 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:16.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:16.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:16.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4186011603' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-14T04:26:16.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:15 smithi114 ceph-mon[103029]: osdmap e370: 8 total, 8 up, 5 in 2024-09-14T04:26:16.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:15 smithi114 ceph-mon[103029]: pgmap v1418: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 587 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 884 op/s; 6439/28209 objects degraded (22.826%); 14673/28209 objects misplaced (52.015%); 6.0 MiB/s, 96 objects/s recovering 2024-09-14T04:26:16.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4186011603' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-14T04:26:16.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:15 smithi114 ceph-mon[106969]: osdmap e370: 8 total, 8 up, 5 in 2024-09-14T04:26:16.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:15 smithi114 ceph-mon[106969]: pgmap v1418: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 587 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 884 op/s; 6439/28209 objects degraded (22.826%); 14673/28209 objects misplaced (52.015%); 6.0 MiB/s, 96 objects/s recovering 2024-09-14T04:26:16.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:16.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:16.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:16.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4186011603' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-14T04:26:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:15 smithi160 ceph-mon[88407]: osdmap e370: 8 total, 8 up, 5 in 2024-09-14T04:26:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:15 smithi160 ceph-mon[88407]: pgmap v1418: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 587 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 884 op/s; 6439/28209 objects degraded (22.826%); 14673/28209 objects misplaced (52.015%); 6.0 MiB/s, 96 objects/s recovering 2024-09-14T04:26:16.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:16.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:16.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:16.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:16.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:16.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:16.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-14T04:26:16.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:16.509 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:16.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:16.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:16.637 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:16.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:16.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:16.710 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:26:16.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:16.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:16.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:16.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:16.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:16.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:16.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:17.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:17.008 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:16 smithi160 ceph-mon[88407]: osdmap e371: 8 total, 8 up, 5 in 2024-09-14T04:26:17.008 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:16 smithi160 ceph-mon[88407]: 13.e starting backfill to osd.2 from (0'0,0'0] MAX to 369'10917 2024-09-14T04:26:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:17.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:17.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:17.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:16 smithi114 ceph-mon[106969]: osdmap e371: 8 total, 8 up, 5 in 2024-09-14T04:26:17.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:16 smithi114 ceph-mon[106969]: 13.e starting backfill to osd.2 from (0'0,0'0] MAX to 369'10917 2024-09-14T04:26:17.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:16 smithi114 ceph-mon[103029]: osdmap e371: 8 total, 8 up, 5 in 2024-09-14T04:26:17.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:16 smithi114 ceph-mon[103029]: 13.e starting backfill to osd.2 from (0'0,0'0] MAX to 369'10917 2024-09-14T04:26:17.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:17.158 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:17.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:17.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:17.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:17.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:17.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:17.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:17.406 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:17.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:17.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:17.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:17.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:17.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:17.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:17.656 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:17.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:17.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:17.784 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:17.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:17.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:17.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:17.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:17.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:17.977 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:17 smithi160 ceph-mon[88407]: 13.5 starting backfill to osd.3 from (0'0,0'0] MAX to 369'11089 2024-09-14T04:26:17.978 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:17 smithi160 ceph-mon[88407]: 13.0 starting backfill to osd.4 from (329'1900,359'9963] MIN to 369'11151 2024-09-14T04:26:17.978 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:17 smithi160 ceph-mon[88407]: osdmap e372: 8 total, 8 up, 5 in 2024-09-14T04:26:17.978 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:17 smithi160 ceph-mon[88407]: 13.3 starting backfill to osd.5 from (0'0,0'0] MAX to 369'11358 2024-09-14T04:26:17.978 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:17 smithi160 ceph-mon[88407]: 3.12 continuing backfill to osd.5 from (0'0,0'0] MIN to 0'0 2024-09-14T04:26:17.978 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:17 smithi160 ceph-mon[88407]: 13.9 continuing backfill to osd.2 from (341'5690,369'11298] MIN to 369'11298 2024-09-14T04:26:17.978 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:17 smithi160 ceph-mon[88407]: pgmap v1421: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 587 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 638 op/s; 6439/28209 objects degraded (22.826%); 14673/28209 objects misplaced (52.015%); 4.2 MiB/s, 67 objects/s recovering 2024-09-14T04:26:18.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:18.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[106969]: 13.5 starting backfill to osd.3 from (0'0,0'0] MAX to 369'11089 2024-09-14T04:26:18.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[106969]: 13.0 starting backfill to osd.4 from (329'1900,359'9963] MIN to 369'11151 2024-09-14T04:26:18.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[106969]: osdmap e372: 8 total, 8 up, 5 in 2024-09-14T04:26:18.012 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[106969]: 13.3 starting backfill to osd.5 from (0'0,0'0] MAX to 369'11358 2024-09-14T04:26:18.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[106969]: 3.12 continuing backfill to osd.5 from (0'0,0'0] MIN to 0'0 2024-09-14T04:26:18.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[106969]: 13.9 continuing backfill to osd.2 from (341'5690,369'11298] MIN to 369'11298 2024-09-14T04:26:18.013 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[106969]: pgmap v1421: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 587 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 638 op/s; 6439/28209 objects degraded (22.826%); 14673/28209 objects misplaced (52.015%); 4.2 MiB/s, 67 objects/s recovering 2024-09-14T04:26:18.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[103029]: 13.5 starting backfill to osd.3 from (0'0,0'0] MAX to 369'11089 2024-09-14T04:26:18.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[103029]: 13.0 starting backfill to osd.4 from (329'1900,359'9963] MIN to 369'11151 2024-09-14T04:26:18.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[103029]: osdmap e372: 8 total, 8 up, 5 in 2024-09-14T04:26:18.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[103029]: 13.3 starting backfill to osd.5 from (0'0,0'0] MAX to 369'11358 2024-09-14T04:26:18.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[103029]: 3.12 continuing backfill to osd.5 from (0'0,0'0] MIN to 0'0 2024-09-14T04:26:18.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[103029]: 13.9 continuing backfill to osd.2 from (341'5690,369'11298] MIN to 369'11298 2024-09-14T04:26:18.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:17 smithi114 ceph-mon[103029]: pgmap v1421: 99 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 86 active+clean; 587 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 638 op/s; 6439/28209 objects degraded (22.826%); 14673/28209 objects misplaced (52.015%); 4.2 MiB/s, 67 objects/s recovering 2024-09-14T04:26:18.037 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:18.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:18.129 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:26:18.129 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:26:18.129 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:26:18.130 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:26:18.139 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:18.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:18.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:18.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:18.168 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:18.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:18.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:18.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:18.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:18.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:18.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:18.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:18.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:18.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:18.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:18.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:18.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:18.666 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:18.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:18.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:18.788 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:18.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:18.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:18 smithi114 ceph-mon[103029]: osdmap e373: 8 total, 8 up, 5 in 2024-09-14T04:26:18.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:18 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1847352286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:26:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:18 smithi114 ceph-mon[106969]: osdmap e373: 8 total, 8 up, 5 in 2024-09-14T04:26:18.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:18 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1847352286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:26:18.890 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:18.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:18.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:18.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:19.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:19.040 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:19.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:19.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:19.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:18 smithi160 ceph-mon[88407]: osdmap e373: 8 total, 8 up, 5 in 2024-09-14T04:26:19.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:18 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1847352286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:26:19.164 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:19.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:19.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:19.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:19.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:19.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:19.398 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:26:19.398 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:26:19.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:19.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:19.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:19.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:19.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:19.582 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:26:19.582 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:26:19.599 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:26:19.599 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:26:19.616 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:26:19.616 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:26:19.644 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:26:19.644 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:26:19.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:19.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:26:19.660 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:26:19.670 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:19.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:19.676 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:26:19.677 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:26:19.772 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:19.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:19.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:19.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:19 smithi114 ceph-mon[106969]: osdmap e374: 8 total, 8 up, 5 in 2024-09-14T04:26:19.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:19 smithi114 ceph-mon[106969]: pgmap v1424: 99 pgs: 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 85 active+clean; 570 MiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 67 op/s; 6502/27393 objects degraded (23.736%); 17774/27393 objects misplaced (64.885%); 2.2 MiB/s, 89 objects/s recovering 2024-09-14T04:26:19.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:19 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 6439/28209 objects degraded (22.826%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:19.869 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:19 smithi114 ceph-mon[103029]: osdmap e374: 8 total, 8 up, 5 in 2024-09-14T04:26:19.869 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:19 smithi114 ceph-mon[103029]: pgmap v1424: 99 pgs: 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 85 active+clean; 570 MiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 67 op/s; 6502/27393 objects degraded (23.736%); 17774/27393 objects misplaced (64.885%); 2.2 MiB/s, 89 objects/s recovering 2024-09-14T04:26:19.873 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:19 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 6439/28209 objects degraded (22.826%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:19.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:19.929 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:19.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:20.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:20.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:20.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:20.081 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:19 smithi160 ceph-mon[88407]: osdmap e374: 8 total, 8 up, 5 in 2024-09-14T04:26:20.082 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:19 smithi160 ceph-mon[88407]: pgmap v1424: 99 pgs: 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 85 active+clean; 570 MiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 67 op/s; 6502/27393 objects degraded (23.736%); 17774/27393 objects misplaced (64.885%); 2.2 MiB/s, 89 objects/s recovering 2024-09-14T04:26:20.082 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:19 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 6439/28209 objects degraded (22.826%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:20.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-14T04:26:20.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:20.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:20.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:20.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:20.306 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:20.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:20.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:26:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:26:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:26:20.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:20.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:20.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:20.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:20.536 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:20.565 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:20.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:20.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:20.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:20.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:20.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:20.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:20.810 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:20.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:20.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:20.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:20.931 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:20 smithi114 ceph-mon[106969]: osdmap e375: 8 total, 8 up, 5 in 2024-09-14T04:26:20.931 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:20 smithi114 ceph-mon[103029]: osdmap e375: 8 total, 8 up, 5 in 2024-09-14T04:26:20.935 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:20.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:20.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:21.011 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:21.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:21.063 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:21.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:21.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:21.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:20 smithi160 ceph-mon[88407]: osdmap e375: 8 total, 8 up, 5 in 2024-09-14T04:26:21.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:21.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:21.188 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:21.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:21.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:21.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:21.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:21.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:21.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:21.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:21.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:21.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:21.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:21.568 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:26:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:26:21] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-14T04:26:21.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:21.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:21.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:21.746 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:26:21.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:21.825 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:21.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:21.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3468225129' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:26:21.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:21 smithi114 ceph-mon[103029]: pgmap v1426: 99 pgs: 2 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 activating+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 83 active+clean; 547 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 142 op/s; 3807/26304 objects degraded (14.473%); 17948/26304 objects misplaced (68.233%); 10 MiB/s, 432 objects/s recovering 2024-09-14T04:26:21.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3468225129' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:26:21.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:21 smithi114 ceph-mon[106969]: pgmap v1426: 99 pgs: 2 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 activating+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 83 active+clean; 547 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 142 op/s; 3807/26304 objects degraded (14.473%); 17948/26304 objects misplaced (68.233%); 10 MiB/s, 432 objects/s recovering 2024-09-14T04:26:21.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:21.960 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:21.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:22.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:22.094 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:22.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:22.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3468225129' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:26:22.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:21 smithi160 ceph-mon[88407]: pgmap v1426: 99 pgs: 2 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 activating+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 83 active+clean; 547 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 142 op/s; 3807/26304 objects degraded (14.473%); 17948/26304 objects misplaced (68.233%); 10 MiB/s, 432 objects/s recovering 2024-09-14T04:26:22.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:22.223 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:22.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:22.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:22.358 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:22.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:22.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:22.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:22.491 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:22.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:22.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:22.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:22.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:22.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:26:22.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:22.757 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:22.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:22.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:22.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:22 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:26:22.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:22 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3468225129' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:26:22.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:22 smithi114 ceph-mon[106969]: osdmap e376: 8 total, 8 up, 5 in 2024-09-14T04:26:22.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:22.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:26:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:22 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:26:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:22 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3468225129' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:26:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:22 smithi114 ceph-mon[103029]: osdmap e376: 8 total, 8 up, 5 in 2024-09-14T04:26:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:26:22.873 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 4, 5, 2] out_osds: [1, 7, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:26:22.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:26:22.874 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-14T04:26:22.874 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:26:22.874 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:26:22.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-14T04:26:22.895 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:22.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:22.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:23.040 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:23.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:23.088 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:23.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:22 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:26:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:22 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3468225129' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:26:23.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:22 smithi160 ceph-mon[88407]: osdmap e376: 8 total, 8 up, 5 in 2024-09-14T04:26:23.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:23.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:26:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:23.168 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:23.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:23.270 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:23.300 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:23.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:23.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:23.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:23.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:23.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:23.566 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:23.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:23.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:23.648 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:26:23.648 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:26:23.648 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-14T04:26:23.649 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:26:23.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:23.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:23.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:23.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:23.837 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:23.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:23.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:23.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:23.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:23.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:24.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:24.093 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:24.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:24.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:23 smithi114 ceph-mon[103029]: pgmap v1428: 99 pgs: 2 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 activating+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 83 active+clean; 545 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 103 op/s; 3779/26220 objects degraded (14.413%); 17948/26220 objects misplaced (68.452%); 6.9 MiB/s, 327 objects/s recovering 2024-09-14T04:26:24.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:23 smithi114 ceph-mon[106969]: pgmap v1428: 99 pgs: 2 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 activating+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 83 active+clean; 545 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 103 op/s; 3779/26220 objects degraded (14.413%); 17948/26220 objects misplaced (68.452%); 6.9 MiB/s, 327 objects/s recovering 2024-09-14T04:26:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:23 smithi160 ceph-mon[88407]: pgmap v1428: 99 pgs: 2 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 activating+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 83 active+clean; 545 MiB data, 7.9 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 103 op/s; 3779/26220 objects degraded (14.413%); 17948/26220 objects misplaced (68.452%); 6.9 MiB/s, 327 objects/s recovering 2024-09-14T04:26:24.195 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:24.227 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:24.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:24.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:24.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:24.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:24.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:24.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:24.481 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:24.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:24.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:26:24.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:24.637 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:24.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:24.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-14T04:26:24.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:24.767 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:24.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:24.869 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:24.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3779/26220 objects degraded (14.413%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:24.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3779/26220 objects degraded (14.413%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:24.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:24.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:25.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:25.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:25.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3779/26220 objects degraded (14.413%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:25.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:25.233 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:25.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:25.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:25.356 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:25.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:25.458 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:25.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:25.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:25.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:25.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:25.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:25.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:25.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:25.751 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:25.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:25.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:25.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:25.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:25.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:25 smithi160 ceph-mon[88407]: pgmap v1429: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 84 active+clean; 476 MiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 258 op/s; 3315/22902 objects degraded (14.475%); 17639/22902 objects misplaced (77.019%); 7.3 MiB/s, 645 objects/s recovering 2024-09-14T04:26:25.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:26.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:26.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:26.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:26.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:26.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:25 smithi114 ceph-mon[106969]: pgmap v1429: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 84 active+clean; 476 MiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 258 op/s; 3315/22902 objects degraded (14.475%); 17639/22902 objects misplaced (77.019%); 7.3 MiB/s, 645 objects/s recovering 2024-09-14T04:26:26.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:25 smithi114 ceph-mon[103029]: pgmap v1429: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 84 active+clean; 476 MiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 258 op/s; 3315/22902 objects degraded (14.475%); 17639/22902 objects misplaced (77.019%); 7.3 MiB/s, 645 objects/s recovering 2024-09-14T04:26:26.142 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:26.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:26.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:26.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:26.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:26.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:26.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:26.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:26.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:26.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:26.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:26.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:26.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:26.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:26.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:26.710 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:26.717 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:26.762 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:26.773 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:26.803 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:26.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:26.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:26.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:26.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:26.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:26.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:27.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:27.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:27.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:27.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:27.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:27.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:27.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:27.325 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:27.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:27.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:27.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:27.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:27.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:27.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:27.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:27.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:27.723 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:27.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:27.825 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:27.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:27.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:27.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:27.981 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:27.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:27.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:28.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:28.105 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:28.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:28.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:28.232 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:28.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:28.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:28.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:28.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:28.474 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:28.501 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:28.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:28.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:28.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:28 smithi114 ceph-mon[103029]: pgmap v1430: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 84 active+clean; 476 MiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 187 op/s; 3315/22902 objects degraded (14.475%); 17639/22902 objects misplaced (77.019%); 5.9 MiB/s, 492 objects/s recovering 2024-09-14T04:26:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:28 smithi114 ceph-mon[106969]: pgmap v1430: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 84 active+clean; 476 MiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 187 op/s; 3315/22902 objects degraded (14.475%); 17639/22902 objects misplaced (77.019%); 5.9 MiB/s, 492 objects/s recovering 2024-09-14T04:26:28.632 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:28.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:28.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:28 smithi160 ceph-mon[88407]: pgmap v1430: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 84 active+clean; 476 MiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 187 op/s; 3315/22902 objects degraded (14.475%); 17639/22902 objects misplaced (77.019%); 5.9 MiB/s, 492 objects/s recovering 2024-09-14T04:26:28.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:28.758 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:28.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:28.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:28.889 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:28.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:28.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:29.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:29.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:29.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:29.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:29.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:29.254 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:29.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:29.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:29.331 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:26:29.331 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:26:29.356 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:26:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:26:29.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:29.380 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:26:29.380 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:26:29.400 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:26:29.400 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:26:29.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:29.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:29.428 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:26:29.428 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:26:29.456 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:26:29.486 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:26:29.487 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:26:29.512 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:26:29.513 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:26:29.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:29.557 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:29.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:29.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:29.688 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:29.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:29.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:29.820 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:29.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:29.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:29.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:29.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:30.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:30.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:30.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:30.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:30.216 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:30.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:30.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:30.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:30.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:30.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:30 smithi114 ceph-mon[106969]: pgmap v1431: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 84 active+clean; 432 MiB data, 7.6 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 194 op/s; 2903/20793 objects degraded (13.961%); 17494/20793 objects misplaced (84.134%); 1.0 MiB/s, 251 objects/s recovering 2024-09-14T04:26:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:30 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3142890556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:26:30.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:30 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2903/20793 objects degraded (13.961%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:30 smithi114 ceph-mon[103029]: pgmap v1431: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 84 active+clean; 432 MiB data, 7.6 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 194 op/s; 2903/20793 objects degraded (13.961%); 17494/20793 objects misplaced (84.134%); 1.0 MiB/s, 251 objects/s recovering 2024-09-14T04:26:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:30 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3142890556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:26:30.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:30 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2903/20793 objects degraded (13.961%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:30.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:30.473 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:30 smithi160 ceph-mon[88407]: pgmap v1431: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 84 active+clean; 432 MiB data, 7.6 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 194 op/s; 2903/20793 objects degraded (13.961%); 17494/20793 objects misplaced (84.134%); 1.0 MiB/s, 251 objects/s recovering 2024-09-14T04:26:30.474 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:30 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3142890556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:26:30.474 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:30 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2903/20793 objects degraded (13.961%), 10 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:30.474 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:26:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:26:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:26:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:30.482 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:30.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:30.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:30.611 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:30.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:30.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:30.740 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:30.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:30.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:30.870 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:30.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:30.972 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:30.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:30.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:31.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:31.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:31.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:31.234 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:31.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:31.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:31.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:31.408 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:31.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:31.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:31.543 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:31.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:31.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:31.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:31.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:31.756 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:26:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:26:31] "GET /metrics HTTP/1.1" 200 36597 "" "Prometheus/2.51.0" 2024-09-14T04:26:31.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:31.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:31.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:31.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:31.934 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:31.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:32.036 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:32.077 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:32.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:32.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:32.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:32.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:32.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:32.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:32.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:32.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:32.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:32 smithi114 ceph-mon[106969]: pgmap v1432: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 84 active+clean; 335 MiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 338 op/s; 2019/16143 objects degraded (12.507%); 15709/16143 objects misplaced (97.312%); 2.7 MiB/s, 430 objects/s recovering 2024-09-14T04:26:32.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:32 smithi114 ceph-mon[103029]: pgmap v1432: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 84 active+clean; 335 MiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 338 op/s; 2019/16143 objects degraded (12.507%); 15709/16143 objects misplaced (97.312%); 2.7 MiB/s, 430 objects/s recovering 2024-09-14T04:26:32.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:32 smithi160 ceph-mon[88407]: pgmap v1432: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 84 active+clean; 335 MiB data, 7.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 338 op/s; 2019/16143 objects degraded (12.507%); 15709/16143 objects misplaced (97.312%); 2.7 MiB/s, 430 objects/s recovering 2024-09-14T04:26:32.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:32.476 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:32.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:32.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:32.622 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:32.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:32.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:32.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:32.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:32.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:32.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:32.877 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:32.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:32.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:32.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:33.008 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:33.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:33.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:33.137 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:33.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:33.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:33.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:33.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:33.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:33.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:33.411 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:33.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:33.513 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:33.548 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:33.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:33.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:33.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:33.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:33.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:33.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:33.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:33.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:33.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:33.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:34.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:34.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:34.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:34.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:34.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:34.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:34.238 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:26:34.255 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:34 smithi114 ceph-mon[106969]: pgmap v1433: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 84 active+clean; 326 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 309 op/s; 1871/15699 objects degraded (11.918%); 15709/15699 objects misplaced (100.064%); 2.4 MiB/s, 377 objects/s recovering 2024-09-14T04:26:34.255 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:34 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2172892810' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:34.255 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:34 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:34.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:34 smithi114 ceph-mon[103029]: pgmap v1433: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 84 active+clean; 326 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 309 op/s; 1871/15699 objects degraded (11.918%); 15709/15699 objects misplaced (100.064%); 2.4 MiB/s, 377 objects/s recovering 2024-09-14T04:26:34.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:34 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2172892810' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:34.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:34 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:34.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:34.341 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:34.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:34.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:34.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:34.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:34.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:34.579 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:34 smithi160 ceph-mon[88407]: pgmap v1433: 99 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 3 active+remapped+backfill_wait, 84 active+clean; 326 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 309 op/s; 1871/15699 objects degraded (11.918%); 15709/15699 objects misplaced (100.064%); 2.4 MiB/s, 377 objects/s recovering 2024-09-14T04:26:34.580 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:34 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2172892810' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:34.580 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:34 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:26:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:34.600 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:34.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:34.628 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:26:34.628 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-14T04:26:34.628 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:26:34.702 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:34.729 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:34.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:34.831 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:34.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:34.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:34.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:35.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:35.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:35.121 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:26:35.134 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:35.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:35.198 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:26:35.198 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:26:35.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:35.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:35.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:35.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:35 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:26:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:35 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:26:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:35 smithi114 ceph-mon[106969]: osdmap e377: 8 total, 8 up, 5 in 2024-09-14T04:26:35.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:35 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1871/15699 objects degraded (11.918%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:35 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:26:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:35 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:26:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:35 smithi114 ceph-mon[103029]: osdmap e377: 8 total, 8 up, 5 in 2024-09-14T04:26:35.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:35 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1871/15699 objects degraded (11.918%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:35.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:35.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:35.412 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:35.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:35.514 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:35.541 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:35.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:35.637 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:35 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:26:35.637 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:35 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:26:35.637 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:35 smithi160 ceph-mon[88407]: osdmap e377: 8 total, 8 up, 5 in 2024-09-14T04:26:35.638 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:35 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1871/15699 objects degraded (11.918%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:35.643 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:35.671 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:35.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:35.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:35.808 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:35.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:35.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:35.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:35.936 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:35.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:35.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:36.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:36.067 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:36.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:36.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:36.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:36.198 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:36.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:36.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:36.332 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:36.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:36 smithi114 ceph-mon[103029]: pgmap v1435: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 165 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 497 op/s; 1034/7971 objects degraded (12.972%); 14864/7971 objects misplaced (186.476%); 3.8 MiB/s, 386 objects/s recovering 2024-09-14T04:26:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:36 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:36.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:36 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:36 smithi114 ceph-mon[106969]: pgmap v1435: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 165 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 497 op/s; 1034/7971 objects degraded (12.972%); 14864/7971 objects misplaced (186.476%); 3.8 MiB/s, 386 objects/s recovering 2024-09-14T04:26:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:36 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:36.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:36 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:36.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:36.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-14T04:26:36.469 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:36.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:36.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:36.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:36.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:36 smithi160 ceph-mon[88407]: pgmap v1435: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 165 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 497 op/s; 1034/7971 objects degraded (12.972%); 14864/7971 objects misplaced (186.476%); 3.8 MiB/s, 386 objects/s recovering 2024-09-14T04:26:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:36 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:36 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:36.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:36.729 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:36.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:36.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:36.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:36.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:36.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:36.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:36.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:37.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:37.114 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:37.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:37.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:37.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:37.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:37.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:37.381 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:37.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:37.394 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:37.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:37.514 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:37.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:37.615 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:37.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:37.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:37.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:37.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:26:37.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:37.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:37.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:37.914 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:37.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:38.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:38.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:38.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:38.073 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:38.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:38.180 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:38.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:38.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:38.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:38.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:38.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:38.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:38.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:38.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:38.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:38.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:38.582 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:38.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:38.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:38.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:38.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:38.725 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 95062 objects 2024-09-14T04:26:38.725 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :175.763 2024-09-14T04:26:38.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:38.828 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:38.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:38.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:38 smithi114 ceph-mon[106969]: pgmap v1436: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 165 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 497 op/s; 1034/7971 objects degraded (12.972%); 14864/7971 objects misplaced (186.476%); 3.8 MiB/s, 386 objects/s recovering 2024-09-14T04:26:38.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:38.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:38 smithi114 ceph-mon[103029]: pgmap v1436: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 165 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 497 op/s; 1034/7971 objects degraded (12.972%); 14864/7971 objects misplaced (186.476%); 3.8 MiB/s, 386 objects/s recovering 2024-09-14T04:26:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:38.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:38.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:38.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:38.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:38.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:39.053 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:39.075 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:39.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:38 smithi160 ceph-mon[88407]: pgmap v1436: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 165 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 497 op/s; 1034/7971 objects degraded (12.972%); 14864/7971 objects misplaced (186.476%); 3.8 MiB/s, 386 objects/s recovering 2024-09-14T04:26:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:26:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:39.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:39.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:39.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:39.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:39.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:39.320 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:39.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:39.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:39.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:39.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:39.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:39.569 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:39.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:39.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:39.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:39.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:39.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:39.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:39.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:39.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:39.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:39.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:39.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:40.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:40.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:40.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:39 smithi114 ceph-mon[103029]: osdmap e378: 8 total, 8 up, 5 in 2024-09-14T04:26:40.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:39 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1034/7971 objects degraded (12.972%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:39 smithi114 ceph-mon[106969]: osdmap e378: 8 total, 8 up, 5 in 2024-09-14T04:26:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:39 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1034/7971 objects degraded (12.972%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:39 smithi160 ceph-mon[88407]: osdmap e378: 8 total, 8 up, 5 in 2024-09-14T04:26:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:39 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1034/7971 objects degraded (12.972%), 9 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:40.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:40.190 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:40.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:40.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:40.313 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:40.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:40.353 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_8 2024-09-14T04:26:40.354 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_8 unique_pool_8 --yes-i-really-really-mean-it 2024-09-14T04:26:40.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:40.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:40.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:40.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:40.560 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:40.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:26:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:26:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:26:40.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:40.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:40 smithi114 ceph-mon[103029]: pgmap v1438: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 111 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 447 op/s; 507/5388 objects degraded (9.410%); 14693/5388 objects misplaced (272.699%); 2.6 MiB/s, 242 objects/s recovering 2024-09-14T04:26:40.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:40 smithi114 ceph-mon[103029]: osdmap e379: 8 total, 8 up, 5 in 2024-09-14T04:26:40.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:40.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:40.786 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:40.809 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:40.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:40.911 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:40.932 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:40.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:41.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:41.056 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:41.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:41.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:40 smithi114 ceph-mon[106969]: pgmap v1438: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 111 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 447 op/s; 507/5388 objects degraded (9.410%); 14693/5388 objects misplaced (272.699%); 2.6 MiB/s, 242 objects/s recovering 2024-09-14T04:26:41.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:40 smithi114 ceph-mon[106969]: osdmap e379: 8 total, 8 up, 5 in 2024-09-14T04:26:41.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:40 smithi160 ceph-mon[88407]: pgmap v1438: 99 pgs: 1 active+recovery_wait+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 111 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 447 op/s; 507/5388 objects degraded (9.410%); 14693/5388 objects misplaced (272.699%); 2.6 MiB/s, 242 objects/s recovering 2024-09-14T04:26:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:40 smithi160 ceph-mon[88407]: osdmap e379: 8 total, 8 up, 5 in 2024-09-14T04:26:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:41.179 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:41.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:41.265 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:26:41.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:41.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:41.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:41.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:41.426 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:41.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:41.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:41.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:41.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:41.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:41.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:41.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:41.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:41.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:41.798 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:41.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:41.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:26:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:26:41] "GET /metrics HTTP/1.1" 200 36594 "" "Prometheus/2.51.0" 2024-09-14T04:26:41.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:41.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:41.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:41.984 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 4, 5, 2] out_osds: [1, 7, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:26:41.984 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:26:42.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:42.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:42.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:42.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:42.171 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:42.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:42.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:42.299 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:42.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:42.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:42.401 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:42.422 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:42.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:42.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:42.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:42.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:42.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:42.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:42.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:42.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:42.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:42.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:42 smithi114 ceph-mon[103029]: pgmap v1440: 99 pgs: 1 activating, 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 3.0 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 641 op/s; 43/198 objects degraded (21.717%); 13721/198 objects misplaced (6929.798%); 5.9 MiB/s, 455 objects/s recovering 2024-09-14T04:26:42.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:42 smithi114 ceph-mon[106969]: pgmap v1440: 99 pgs: 1 activating, 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 3.0 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 641 op/s; 43/198 objects degraded (21.717%); 13721/198 objects misplaced (6929.798%); 5.9 MiB/s, 455 objects/s recovering 2024-09-14T04:26:42.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:42.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:42 smithi160 ceph-mon[88407]: pgmap v1440: 99 pgs: 1 activating, 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 3.0 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 641 op/s; 43/198 objects degraded (21.717%); 13721/198 objects misplaced (6929.798%); 5.9 MiB/s, 455 objects/s recovering 2024-09-14T04:26:42.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:42.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:43.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:43.051 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:43.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:43.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:43.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:43.175 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:43.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:43.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:43.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:43.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:43.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:43.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:43.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:43.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:43.551 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:43.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:43.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:43.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:43.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:43.677 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:43.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:43.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:43.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:43.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:43.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:43.928 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:43.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:44.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:44.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:44.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:44.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:44.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:44.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:44.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:44.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:44.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:44.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:44.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:44.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:44.553 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:44.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:44.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:44.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:44.677 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:44.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:44.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:44.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:44.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:44 smithi114 ceph-mon[106969]: pgmap v1441: 99 pgs: 1 activating, 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 328 op/s; 860/81 objects degraded (1061.728%); 9585/81 objects misplaced (11833.333%); 2.6 MiB/s, 222 objects/s recovering 2024-09-14T04:26:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:44 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 860/81 objects degraded (1061.728%), 4 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:44 smithi114 ceph-mon[103029]: pgmap v1441: 99 pgs: 1 activating, 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 328 op/s; 860/81 objects degraded (1061.728%); 9585/81 objects misplaced (11833.333%); 2.6 MiB/s, 222 objects/s recovering 2024-09-14T04:26:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:44 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 860/81 objects degraded (1061.728%), 4 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:44.903 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:44.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:44.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:45.027 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:45.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:45.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:45.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:45.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:44 smithi160 ceph-mon[88407]: pgmap v1441: 99 pgs: 1 activating, 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 84 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 328 op/s; 860/81 objects degraded (1061.728%); 9585/81 objects misplaced (11833.333%); 2.6 MiB/s, 222 objects/s recovering 2024-09-14T04:26:45.159 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:44 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 860/81 objects degraded (1061.728%), 4 pgs degraded (PG_DEGRADED) 2024-09-14T04:26:45.181 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:45.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:45.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:45.284 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:45.306 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:45.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:45.408 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:45.430 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:45.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:45.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:45.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:45.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:45.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:45.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:45.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:45.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:45.806 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:45.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:45.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:45.930 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:45.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:45.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:46.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:46.058 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:46.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:46.159 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:46.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:46.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:46.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:46.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:46.305 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:46.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:46.407 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:46.433 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:46.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:46.535 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:46.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:46.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:46.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:46.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:46.686 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:46.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:46.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:46.810 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:46.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:46.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:46.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:46.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:46.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:47.030 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:46 smithi114 ceph-mon[106969]: pgmap v1442: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 328 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 2.6 MiB/s, 222 objects/s recovering 2024-09-14T04:26:47.030 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:46 smithi114 ceph-mon[103029]: pgmap v1442: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 328 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 2.6 MiB/s, 222 objects/s recovering 2024-09-14T04:26:47.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:47.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:47.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:47.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:46 smithi160 ceph-mon[88407]: pgmap v1442: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 328 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 2.6 MiB/s, 222 objects/s recovering 2024-09-14T04:26:47.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:47.188 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:47.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:47.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:47.312 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:47.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:47.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:47.441 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:47.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:47.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:47.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:47.567 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:47.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:47.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:47.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:47.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:47.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:47.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:47.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:47.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:47.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:47.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:48.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:48.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:48.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:48.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:48.194 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:48.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:48.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:48.319 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:48.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:48.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:48.445 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:48.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:48.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:48.571 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:48.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:48.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:48.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:48.697 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:48.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:48.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:48.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:48.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:48.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:48 smithi114 ceph-mon[103029]: pgmap v1443: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 289 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 2.5 MiB/s, 209 objects/s recovering 2024-09-14T04:26:48.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:48 smithi114 ceph-mon[106969]: pgmap v1443: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 289 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 2.5 MiB/s, 209 objects/s recovering 2024-09-14T04:26:48.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:48.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:48.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:49.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:49.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:49.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:49.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:48 smithi160 ceph-mon[88407]: pgmap v1443: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 289 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 2.5 MiB/s, 209 objects/s recovering 2024-09-14T04:26:49.179 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:49.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:49.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:49.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:49.329 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:49.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:49.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:49.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:49.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:49.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:49.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:49.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:49.706 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:49.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:49.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:49.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:49.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:49.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:49.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:49.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:49.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:50.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:50.083 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:50.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:50.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:49 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 860/78 objects degraded (1102.564%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:26:50.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:49 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 860/78 objects degraded (1102.564%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:26:50.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:49 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 860/78 objects degraded (1102.564%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-14T04:26:50.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:50.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:50.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:50.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:50.336 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:50.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:50.438 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:50.462 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:50.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:50.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:50.589 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:50.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:26:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:26:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:26:50.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:50.717 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:50.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:50.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:50.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:50.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:50.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:50.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:50.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:50.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:50.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:51.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:51.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:51.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:51.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:50 smithi114 ceph-mon[106969]: pgmap v1444: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 176 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 2.1 MiB/s, 260 objects/s recovering 2024-09-14T04:26:51.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:50 smithi114 ceph-mon[103029]: pgmap v1444: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 176 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 2.1 MiB/s, 260 objects/s recovering 2024-09-14T04:26:51.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:50 smithi160 ceph-mon[88407]: pgmap v1444: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 176 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 2.1 MiB/s, 260 objects/s recovering 2024-09-14T04:26:51.198 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:51.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:51.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:51.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:51.352 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:51.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:51.454 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:51.475 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:51.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:51.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:51.605 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:51.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:51.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:51.733 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:51.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:51.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:51.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:51.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:51.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:51.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:26:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:26:51] "GET /metrics HTTP/1.1" 200 36594 "" "Prometheus/2.51.0" 2024-09-14T04:26:51.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:51.994 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:51.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:52.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:52.122 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:52.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:52.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:52.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:52.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:52.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:52.373 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:52.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:52.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:52.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:52.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:52.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:52.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:52.625 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:52.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:52.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:52.752 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:52.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:52.854 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:52.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:52.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:52.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:53.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:53.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:53.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:53.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:52 smithi114 ceph-mon[106969]: pgmap v1445: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 28 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 100 KiB/s, 199 objects/s recovering 2024-09-14T04:26:53.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:52 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:53.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:52 smithi114 ceph-mon[103029]: pgmap v1445: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 28 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 100 KiB/s, 199 objects/s recovering 2024-09-14T04:26:53.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:53.131 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:53.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:52 smithi160 ceph-mon[88407]: pgmap v1445: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 28 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 100 KiB/s, 199 objects/s recovering 2024-09-14T04:26:53.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:52 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:26:53.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:53.255 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:53.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:53.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:53.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:53.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:53.485 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:53.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:53.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:53.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:53.631 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:53.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:53.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:53.755 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:53.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:53.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:53.882 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:53.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:53.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:54.007 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:54.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:54.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:54.130 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:54.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:54.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:54.254 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:54.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:54.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:54.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:54.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:54.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:54.506 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:54.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:54.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:54.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:54.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:54.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:54.732 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:54.754 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:54.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:54.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:54.878 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:54.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:54.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:54.980 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:55.002 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:55.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:55.104 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:55.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:54 smithi114 ceph-mon[103029]: pgmap v1446: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 3 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 48 KiB/s, 203 objects/s recovering 2024-09-14T04:26:55.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:54 smithi114 ceph-mon[106969]: pgmap v1446: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 3 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 48 KiB/s, 203 objects/s recovering 2024-09-14T04:26:55.131 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:55.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:54 smithi160 ceph-mon[88407]: pgmap v1446: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 3 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 48 KiB/s, 203 objects/s recovering 2024-09-14T04:26:55.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:55.260 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:55.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:55.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:55.385 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:55.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:55.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:55.513 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:55.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:55.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:55.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:55.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:55.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:55.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:55.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:55.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:55.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:55.891 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:55.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:55.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:56.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:56.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:56.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:56.143 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:56.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:56.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:56.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:56.270 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:56.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:56.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:26:56.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:56.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:56.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:26:56.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:56.522 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:56.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:56.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:56.645 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:56.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:56.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:56.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:56.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:56.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:56.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:56.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:56.994 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:57.021 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:57.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:57.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:56 smithi114 ceph-mon[103029]: pgmap v1447: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 0 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 157 objects/s recovering 2024-09-14T04:26:57.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:56 smithi114 ceph-mon[106969]: pgmap v1447: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 0 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 157 objects/s recovering 2024-09-14T04:26:57.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:57.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:57.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:56 smithi160 ceph-mon[88407]: pgmap v1447: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 0 op/s; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 157 objects/s recovering 2024-09-14T04:26:57.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:57.271 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:57.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:57.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:57.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:57.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:57.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:57.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:57.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:57.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:57.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:57.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:57.648 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:57.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:57.750 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:57.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:57.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:57.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:26:57.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:57.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:57.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:26:58.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:58.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:58.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:58.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:58.153 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:58.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:58.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:58.280 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:58.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:58.382 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:58.406 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:58.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:58.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:26:58.529 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:58.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:26:58.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:58.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:58.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:58.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:58.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:58.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:58.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:58.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:58.902 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:58.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:26:59.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:26:59.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:59.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:26:59.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:58 smithi114 ceph-mon[106969]: pgmap v1448: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 157 objects/s recovering 2024-09-14T04:26:59.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:58 smithi114 ceph-mon[103029]: pgmap v1448: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 157 objects/s recovering 2024-09-14T04:26:59.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:59.153 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:59.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:58 smithi160 ceph-mon[88407]: pgmap v1448: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 157 objects/s recovering 2024-09-14T04:26:59.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:59.281 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:59.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:59.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:26:59.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:26:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:59.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:26:59.511 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:59.533 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:59.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:59.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:26:59.656 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:59.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:26:59.758 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:26:59.779 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:59.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:59.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:26:59.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:26:59.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:26:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:26:59.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:00.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:00.021 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:26:59 smithi114 ceph-mon[106969]: pgmap v1449: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 265 objects/s recovering 2024-09-14T04:27:00.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:26:59 smithi114 ceph-mon[103029]: pgmap v1449: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 265 objects/s recovering 2024-09-14T04:27:00.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:00.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:00.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:00.058 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:26:59 smithi160 ceph-mon[88407]: pgmap v1449: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 265 objects/s recovering 2024-09-14T04:27:00.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:00.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:00.157 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:00.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:00.238 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:00.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:00.283 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:00.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:00.289 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:27:00.289 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:27:00.313 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:27:00.313 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:27:00.322 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:27:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:27:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:27:00.332 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:27:00.333 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:27:00.349 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:27:00.350 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:27:00.365 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:27:00.366 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:27:00.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:00.394 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:27:00.394 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:27:00.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:00.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:00.421 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:27:00.421 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:27:00.448 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:27:00.448 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:27:00.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:00.533 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:00.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:00.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:00.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:00.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:00.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:00.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:00.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:00.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:00.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:00.905 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:00.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:01.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:01.028 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:01.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:01.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:01.152 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:01.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:01.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:01.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:01.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:01 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4227957859' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:27:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:27:01.309 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:01.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:01.322 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:01.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4227957859' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:01.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:27:01.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:27:01.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:01.323 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:01.324 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:01.324 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4227957859' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:01.324 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:27:01.324 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:27:01.324 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:01.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:01.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:01.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:01.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:01.528 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:01.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:01.599 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:27:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:27:01] "GET /metrics HTTP/1.1" 200 36583 "" "Prometheus/2.51.0" 2024-09-14T04:27:01.630 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:01.652 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:01.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:01.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:01.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:01.775 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:01.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:01.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:01.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:01.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:02.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:02.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:02.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:02.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:02.145 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:02.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:02.248 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:02.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:02.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:02.305 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:02 smithi114 ceph-mon[106969]: pgmap v1450: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 196 objects/s recovering 2024-09-14T04:27:02.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:02 smithi114 ceph-mon[103029]: pgmap v1450: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 196 objects/s recovering 2024-09-14T04:27:02.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:02.394 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:02.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:02.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:02.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:02 smithi160 ceph-mon[88407]: pgmap v1450: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 196 objects/s recovering 2024-09-14T04:27:02.415 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:27:02.415 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:27:02.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:27:02.415 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:27:02.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.415 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:54910 12.c 12:380aa551:::benchmark_data_smithi114_253692_object54909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-14T04:27:02.415 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:17:53.600122+0000", 2024-09-14T04:27:02.415 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 548.79642029700005, 2024-09-14T04:27:02.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.14566506500000001, 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 54910 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600122+0000", 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600122+0000", 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600124+0000", 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6279999999999999e-06 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600147+0000", 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2909999999999999e-05 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600160+0000", 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3202e-05 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600168+0000", 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7079999999999997e-06 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600224+0000", 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6178999999999999e-05 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600371+0000", 2024-09-14T04:27:02.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014686899999999999 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.600516+0000", 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000144943 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.603830+0000", 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0033139240000000002 2024-09-14T04:27:02.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.603921+0000", 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1420000000000002e-05 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.745695+0000", 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.14177393699999999 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.745747+0000", 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2148999999999998e-05 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:53.745788+0000", 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0195999999999999e-05 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:59150 12.3 12:cd0072a9:::benchmark_data_smithi114_253692_object59149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:17:56.926213+0000", 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 545.47032933100002, 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.19487368399999999, 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:27:02.423 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 59150 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926213+0000", 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926213+0000", 2024-09-14T04:27:02.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926215+0000", 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3799999999999999e-06 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926236+0000", 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0849999999999999e-05 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926248+0000", 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1831000000000001e-05 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926255+0000", 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2729999999999997e-06 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926315+0000", 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0115000000000003e-05 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926446+0000", 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013073300000000001 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.926563+0000", 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011694 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.928752+0000", 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002189346 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.929480+0000", 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00072826499999999997 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.120965+0000", 2024-09-14T04:27:02.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.191485022 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.121030+0000", 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4620000000000001e-05 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.121087+0000", 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7309000000000003e-05 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.429 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:59166 12.3 12:c2097a96:::benchmark_data_smithi114_253692_object59165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:17:56.962037+0000", 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 545.43450596100001, 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.159174488, 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:27:02.430 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 59166 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962037+0000", 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962037+0000", 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962039+0000", 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8479999999999999e-06 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962062+0000", 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3563000000000001e-05 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962077+0000", 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4324999999999999e-05 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962084+0000", 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2320000000000004e-06 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962141+0000", 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6993000000000001e-05 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.962288+0000", 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014693099999999999 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.968967+0000", 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0066796920000000001 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.971082+0000", 2024-09-14T04:27:02.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021143569999999999 2024-09-14T04:27:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:56.971135+0000", 2024-09-14T04:27:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3498000000000002e-05 2024-09-14T04:27:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.121132+0000", 2024-09-14T04:27:02.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.149997043 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.121179+0000", 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7141000000000001e-05 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:17:57.121211+0000", 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1865000000000001e-05 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.436 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:103395 12.3 12:c488e3d1:::benchmark_data_smithi114_253692_object103394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.404797+0000", 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 503.99174551499999, 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.343085796, 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:27:02.437 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:27:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103395 2024-09-14T04:27:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404797+0000", 2024-09-14T04:27:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404797+0000", 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404799+0000", 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3859999999999999e-06 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404826+0000", 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7393e-05 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404889+0000", 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.2936999999999997e-05 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.404900+0000", 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0825999999999999e-05 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405289+0000", 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00038889300000000002 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405410+0000", 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012131899999999999 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.405544+0000", 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013358200000000001 2024-09-14T04:27:02.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.409765+0000", 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042217369999999997 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.412481+0000", 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027151839999999998 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.443 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747776+0000", 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.33529530499999999 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747838+0000", 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1815000000000004e-05 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747883+0000", 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5419e-05 2024-09-14T04:27:02.444 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:103415 12.3 12:c72de31a:::benchmark_data_smithi114_253692_object103414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.421345+0000", 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 503.975197971, 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32664568199999999, 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:02.445 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103415 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421345+0000", 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421345+0000", 2024-09-14T04:27:02.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421346+0000", 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2699999999999998e-07 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421367+0000", 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1509999999999999e-05 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421376+0000", 2024-09-14T04:27:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0890000000000006e-06 2024-09-14T04:27:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421383+0000", 2024-09-14T04:27:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3829999999999997e-06 2024-09-14T04:27:02.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421447+0000", 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4517999999999995e-05 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421584+0000", 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000136441 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.421728+0000", 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000144258 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.424854+0000", 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003126438 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.425776+0000", 2024-09-14T04:27:02.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00092118199999999999 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747921+0000", 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.32214522000000001 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747961+0000", 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0049000000000002e-05 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.451 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.747990+0000", 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9666999999999999e-05 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:103428 12.3 12:c8a48380:::benchmark_data_smithi114_253692_object103427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:38.440286+0000", 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 503.95625639000002, 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30781922699999997, 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.452 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103428 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440286+0000", 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440286+0000", 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440289+0000", 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5229999999999998e-06 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440334+0000", 2024-09-14T04:27:02.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5388e-05 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440348+0000", 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3825999999999999e-05 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440354+0000", 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8479999999999997e-06 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440403+0000", 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9422999999999998e-05 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440526+0000", 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000122595 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,7", 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.440629+0000", 2024-09-14T04:27:02.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010343799999999999 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.443293+0000", 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0026637369999999998 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.443766+0000", 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00047331499999999998 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.748020+0000", 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.30425396100000002 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.748073+0000", 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2438000000000002e-05 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:38.748106+0000", 2024-09-14T04:27:02.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2734999999999997e-05 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46463.0:114041 12.c 12:398ecbb4:::benchmark_data_smithi114_253692_object114040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:18:50.692744+0000", 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 491.703798758, 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.17576045100000001, 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.459 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46463", 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/1857775204", 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 114041 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692744+0000", 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692744+0000", 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692745+0000", 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5899999999999995e-07 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692765+0000", 2024-09-14T04:27:02.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9955000000000001e-05 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692773+0000", 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5779999999999999e-06 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692780+0000", 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3169999999999998e-06 2024-09-14T04:27:02.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692838+0000", 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7986999999999999e-05 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.692955+0000", 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000116982 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.693074+0000", 2024-09-14T04:27:02.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000119327 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.697950+0000", 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0048754949999999997 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.700581+0000", 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002631605 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.868407+0000", 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.167826214 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.868463+0000", 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6079999999999998e-05 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:18:50.868504+0000", 2024-09-14T04:27:02.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1051999999999999e-05 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:31605 13.3 e329/328 13:cc044c86:::benchmark_data_smithi114_288325_object31604:head v 329'1972, mlcod=329'1970)", 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:24.685581+0000", 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 277.710962107, 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.239790843, 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685581+0000", 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685581+0000", 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685582+0000", 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3939999999999999e-06 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685609+0000", 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7186e-05 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685626+0000", 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6368999999999999e-05 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685634+0000", 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6559999999999998e-06 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.468 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685731+0000", 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6457000000000002e-05 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.685751+0000", 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0236000000000001e-05 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.469 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687942+0000", 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00219118 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925372+0000", 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.237429365 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:31611 13.9 e329/324 13:927299fd:::benchmark_data_smithi114_288325_object31610:head v 329'1946, mlcod=329'1944)", 2024-09-14T04:27:02.470 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:24.687573+0000", 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 277.70896990099999, 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.24159929999999999, 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687573+0000", 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.471 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687573+0000", 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687574+0000", 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.35e-06 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687668+0000", 2024-09-14T04:27:02.472 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.3508e-05 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687681+0000", 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2840000000000001e-05 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.687689+0000", 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0620000000000001e-06 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925334+0000", 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23764569399999999 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925360+0000", 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5554e-05 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.929134+0000", 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003774374 2024-09-14T04:27:02.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.929172+0000", 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7917999999999998e-05 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:31614 13.5 e329/324 13:ad7120f7:::benchmark_data_smithi114_288325_object31613:head v 329'1959, mlcod=329'1957)", 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:24.688736+0000", 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 277.70780700300003, 2024-09-14T04:27:02.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.24212169, 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.688736+0000", 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.476 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.688736+0000", 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.688737+0000", 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.85e-07 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.688796+0000", 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9463999999999999e-05 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925276+0000", 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.23647989899999999 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925287+0000", 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1161e-05 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925367+0000", 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9553000000000002e-05 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925387+0000", 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0109e-05 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.930803+0000", 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054157509999999999 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.930857+0000", 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4968e-05 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:31644 13.3 e329/328 13:ce3478ec:::benchmark_data_smithi114_288325_object31643:head v 329'1973, mlcod=329'1970)", 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:22:24.702025+0000", 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 277.69451744600002, 2024-09-14T04:27:02.480 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.22626843099999999, 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.702025+0000", 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.481 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.702025+0000", 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.702027+0000", 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.511e-06 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.702065+0000", 2024-09-14T04:27:02.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.8262000000000001e-05 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925266+0000", 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.223200974 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925284+0000", 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8142999999999999e-05 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925452+0000", 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000167718 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.925476+0000", 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3584e-05 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.928209+0000", 2024-09-14T04:27:02.484 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0027334019999999998 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:22:24.928294+0000", 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4837000000000001e-05 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:98930 13.0 e342/338 13:0a3dc646:::benchmark_data_smithi114_288325_object3865:head v 342'6104, mlcod=0'0)", 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:48.639415+0000", 2024-09-14T04:27:02.485 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 193.75712821600001, 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.022609377999999999, 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:48.639415+0000", 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.486 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:48.639415+0000", 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:48.639415+0000", 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6199999999999998e-07 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:48.639419+0000", 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7570000000000002e-06 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:48.639425+0000", 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7020000000000004e-06 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:48.639430+0000", 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5179999999999998e-06 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:48.643712+0000", 2024-09-14T04:27:02.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.004282307 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:48.643736+0000", 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.3394e-05 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:48.661849+0000", 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.018113036999999998 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.489 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:48.662024+0000", 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017520099999999999 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:100948 13.0 e342/338 13:082bb69d:::benchmark_data_smithi114_288325_object5883:head v 342'6233, mlcod=0'0)", 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:51.628649+0000", 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 190.76789340600001, 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.031085964000000001, 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.490 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.628649+0000", 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.628649+0000", 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.628650+0000", 2024-09-14T04:27:02.492 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4600000000000004e-07 2024-09-14T04:27:02.492 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.492 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.492 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.492 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.628656+0000", 2024-09-14T04:27:02.492 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.02e-06 2024-09-14T04:27:02.493 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.493 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.493 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.493 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.628670+0000", 2024-09-14T04:27:02.493 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3723999999999999e-05 2024-09-14T04:27:02.493 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.493 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.493 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.493 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.628676+0000", 2024-09-14T04:27:02.493 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.6059999999999999e-06 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.638109+0000", 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0094325510000000008 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.638136+0000", 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6673e-05 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.659505+0000", 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.021369407999999999 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.659735+0000", 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023023600000000001 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.495 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:100952 13.0 e342/338 13:0445b429:::benchmark_data_smithi114_288325_object5887:head v 342'6234, mlcod=0'0)", 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:23:51.632261+0000", 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 190.764282221, 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.027645652, 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.632261+0000", 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.632261+0000", 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.632262+0000", 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0559999999999999e-06 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.632268+0000", 2024-09-14T04:27:02.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1040000000000003e-06 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.632282+0000", 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4664000000000001e-05 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.632288+0000", 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6099999999999997e-06 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.645461+0000", 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013172695 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.645488+0000", 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7372000000000002e-05 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.659740+0000", 2024-09-14T04:27:02.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.014251987000000001 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:23:51.659906+0000", 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000166164 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:114413 13.0 e343/338 13:041506c4:::benchmark_data_smithi114_288325_object19348:head v 343'7099, mlcod=0'0)", 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:24:12.522785+0000", 2024-09-14T04:27:02.500 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 169.87375766299999, 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.027303099000000001, 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.522785+0000", 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.501 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.522785+0000", 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.522787+0000", 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3829999999999999e-06 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.522794+0000", 2024-09-14T04:27:02.502 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3880000000000002e-06 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.522805+0000", 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0699e-05 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.522817+0000", 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2078e-05 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.527157+0000", 2024-09-14T04:27:02.503 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043400799999999996 2024-09-14T04:27:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.527187+0000", 2024-09-14T04:27:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0474999999999999e-05 2024-09-14T04:27:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.550046+0000", 2024-09-14T04:27:02.504 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.022858346000000002 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.550088+0000", 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.265e-05 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:114416 13.0 e343/338 13:04cc0bf2:::benchmark_data_smithi114_288325_object19351:head v 343'7100, mlcod=0'0)", 2024-09-14T04:27:02.505 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:24:12.525513+0000", 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 169.87102935300001, 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.024601366, 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.525513+0000", 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.525513+0000", 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.525514+0000", 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5400000000000001e-07 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.525519+0000", 2024-09-14T04:27:02.507 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2989999999999996e-06 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.525529+0000", 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8900000000000002e-06 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.525535+0000", 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8309999999999997e-06 2024-09-14T04:27:02.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.532308+0000", 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0067728670000000001 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.532337+0000", 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8804999999999999e-05 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.550093+0000", 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.017756231000000001 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:24:12.550115+0000", 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1789000000000001e-05 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.510 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:168341 13.0 e367/361 13:05605046:::benchmark_data_smithi114_288325_object73276:head v 367'10343, mlcod=336'4466)", 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:25:55.362943+0000", 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 67.033599362000004, 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.028920227, 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.362943+0000", 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.362943+0000", 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.362945+0000", 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1939999999999999e-06 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.512 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.362952+0000", 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1330000000000001e-06 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.362966+0000", 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.4116e-05 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.362973+0000", 2024-09-14T04:27:02.513 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.5979999999999999e-06 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.363170+0000", 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00019667299999999999 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.363220+0000", 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0188999999999997e-05 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.514 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.391676+0000", 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.028455385999999999 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.391864+0000", 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00018793799999999999 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "MOSDPGPush(13.0 367/361 [PushOp(13:026e8c7b:::benchmark_data_smithi114_288325_object73309:head, version: 336'4474, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(13:026e8c7b:::benchmark_data_smithi114_288325_object73309:head@336'4474, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:25:55.377482+0000", 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 67.019060449999998, 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.023436483000000001, 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.377482+0000", 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.377482+0000", 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.377483+0000", 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3599999999999999e-07 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.377509+0000", 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5831000000000001e-05 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.377526+0000", 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7098000000000001e-05 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.377541+0000", 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5387999999999999e-05 2024-09-14T04:27:02.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.396631+0000", 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.019089569000000001 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.396642+0000", 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0893000000000001e-05 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.519 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.400919+0000", 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042772679999999999 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:168355 13.0 e367/361 13:059357fc:::benchmark_data_smithi114_288325_object73290:head v 367'10344, mlcod=336'4468)", 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:25:55.421466+0000", 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 66.975077048000003, 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.028140305000000001, 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.520 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.421466+0000", 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.421466+0000", 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.521 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.421467+0000", 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6020000000000001e-06 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.421476+0000", 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0880000000000001e-06 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.522 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.421498+0000", 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1968e-05 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.421507+0000", 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.6030000000000005e-06 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.428239+0000", 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00673159 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.523 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.428298+0000", 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9762999999999997e-05 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.449460+0000", 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.021161981 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.524 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:55.449606+0000", 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014571000000000001 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.46760.0:170048 13.0 e368/361 13:0906d30f:::benchmark_data_smithi114_288325_object74983:head v 368'10469, mlcod=336'4552)", 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:25:58.546966+0000", 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 63.849576431999999, 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.016012545, 2024-09-14T04:27:02.525 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:58.546966+0000", 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:58.546966+0000", 2024-09-14T04:27:02.526 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:58.546968+0000", 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5230000000000001e-06 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:58.546976+0000", 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9070000000000007e-06 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:02.527 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:58.546994+0000", 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8011000000000001e-05 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:58.547002+0000", 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4999999999999999e-06 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:58.550743+0000", 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037402329999999999 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.528 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:58.550764+0000", 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1843000000000001e-05 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:58.562896+0000", 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.012131183 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:58.562979+0000", 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3344999999999996e-05 2024-09-14T04:27:02.529 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.530 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.530 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.530 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:02.530 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:02.530 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:27:02.530 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:02.551 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:02.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:02.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:02.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:02.677 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:02.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:02.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:02.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:02.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:02.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:02.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:02.923 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:02.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:03.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:03.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:03.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:03.149 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:03.171 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:03.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:03.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:03.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:03.295 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:03.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:03.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:03 smithi114 ceph-mon[103029]: osdmap e380: 8 total, 8 up, 5 in 2024-09-14T04:27:03.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:03 smithi114 ceph-mon[106969]: osdmap e380: 8 total, 8 up, 5 in 2024-09-14T04:27:03.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:03.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:03 smithi160 ceph-mon[88407]: osdmap e380: 8 total, 8 up, 5 in 2024-09-14T04:27:03.420 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:03.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:03.522 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:03.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-14T04:27:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:03.545 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:03.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:03.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:03.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:03.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:03.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:03.794 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:03.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:03.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:03.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:03.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:03.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:04.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:04.046 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:04.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:04.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:04.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:04.170 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:04.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:04.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:27:04.209 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:04 smithi114 ceph-mon[103029]: osdmap e381: 8 total, 8 up, 5 in 2024-09-14T04:27:04.209 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:04 smithi114 ceph-mon[103029]: pgmap v1453: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 163 objects/s recovering 2024-09-14T04:27:04.210 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:04 smithi114 ceph-mon[106969]: osdmap e381: 8 total, 8 up, 5 in 2024-09-14T04:27:04.210 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:04 smithi114 ceph-mon[106969]: pgmap v1453: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 163 objects/s recovering 2024-09-14T04:27:04.272 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:04.294 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:04.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:04.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:04.407 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:04 smithi160 ceph-mon[88407]: osdmap e381: 8 total, 8 up, 5 in 2024-09-14T04:27:04.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:04 smithi160 ceph-mon[88407]: pgmap v1453: 99 pgs: 1 active+recovery_wait+undersized+remapped, 5 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 85 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 860/78 objects degraded (1102.564%); 9585/78 objects misplaced (12288.462%); 0 B/s, 163 objects/s recovering 2024-09-14T04:27:04.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:04.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:04.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:04.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:04.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:04.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:04.667 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:04.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:04.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:04.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:04.793 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:04.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:04.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:04.915 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:04.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:05.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:05.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:05.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:05.141 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:05.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:05.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:05.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:05.287 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:05.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:05.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:05.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:05.412 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:05.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:05.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:05.536 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:05.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:05.561 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:05 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4/78 objects degraded (5.128%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:27:05.562 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:05 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4/78 objects degraded (5.128%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:27:05.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:05 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4/78 objects degraded (5.128%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:27:05.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:05.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:05.675 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:05.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:05.786 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:05.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:05.887 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:05.910 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:05.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:06.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:06.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:06.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:06.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:06.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:06.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:06.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:06.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:06.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:06.281 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:06.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:06.291 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:27:06.291 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_8' removed 2024-09-14T04:27:06.324 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:06 smithi114 ceph-mon[103029]: pgmap v1454: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+remapped, 4 active+recovery_wait+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 86 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 4/78 objects degraded (5.128%); 8729/78 objects misplaced (11191.026%); 0 B/s, 429 objects/s recovering 2024-09-14T04:27:06.325 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3639800640' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:27:06.325 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:06 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:27:06.325 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3944223772' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:27:06.325 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:06 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:27:06.325 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:06 smithi114 ceph-mon[106969]: pgmap v1454: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+remapped, 4 active+recovery_wait+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 86 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 4/78 objects degraded (5.128%); 8729/78 objects misplaced (11191.026%); 0 B/s, 429 objects/s recovering 2024-09-14T04:27:06.325 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3639800640' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:27:06.325 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:06 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:27:06.325 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3944223772' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:27:06.326 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:06 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:27:06.383 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:06.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:06.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:06.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:06.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:06.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:06.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:06.654 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:06.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:06 smithi160 ceph-mon[88407]: pgmap v1454: 99 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+remapped, 4 active+recovery_wait+remapped, 3 active+recovery_wait+degraded, 4 active+remapped+backfill_wait, 86 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 4/78 objects degraded (5.128%); 8729/78 objects misplaced (11191.026%); 0 B/s, 429 objects/s recovering 2024-09-14T04:27:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3639800640' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:27:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:06 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:27:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3944223772' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:27:06.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:06 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:27:06.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:06.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:06.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:06.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:06.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:06.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:06.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:07.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:07.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:07.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:07.128 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:07.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:07.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:07.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:07.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:07.274 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:07.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:07.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:07.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:07.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:07.450 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:07 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:27:07.450 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:07 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:27:07.450 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:07 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:27:07.450 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:07 smithi114 ceph-mon[106969]: osdmap e382: 8 total, 8 up, 5 in 2024-09-14T04:27:07.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:07 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:27:07.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:07 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:27:07.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:07 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:27:07.451 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:07 smithi114 ceph-mon[103029]: osdmap e382: 8 total, 8 up, 5 in 2024-09-14T04:27:07.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:07.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:07.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:07.622 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:07.644 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:07.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:07.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:07 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:27:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:07 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:27:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:07 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:27:07.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:07 smithi160 ceph-mon[88407]: osdmap e382: 8 total, 8 up, 5 in 2024-09-14T04:27:07.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:07.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:07.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:07.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:07.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:07.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:07.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:08.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:08.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:08.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:08.141 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:08.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:08.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:08.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:08.266 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:08.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:08.321 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:27:08.321 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:27:08.334 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:27:08.335 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:27:08.360 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:27:08.361 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:27:08.364 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:08.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:08.385 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:27:08.385 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:27:08.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:08.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:08.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:08.402 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:27:08.402 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:27:08.417 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:27:08.418 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:27:08.433 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:27:08.433 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:27:08.448 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:27:08.449 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:27:08.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:08.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:08.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:08.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:08.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:08.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:08.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:08.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:08.762 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:08.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:08.856 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-14T04:27:08.857 INFO:tasks.radosbench:Beginning radosbench... 2024-09-14T04:27:08.858 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_9 2024-09-14T04:27:08.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_9 16 2024-09-14T04:27:08.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:08.887 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:08.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:08.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:09.012 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:09.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:09.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:09.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:08 smithi114 ceph-mon[103029]: pgmap v1456: 83 pgs: 3 active+recovery_wait+degraded, 80 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 4/78 objects degraded (5.128%) 2024-09-14T04:27:09.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:09.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:09.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2161504957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:09.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:08 smithi114 ceph-mon[106969]: pgmap v1456: 83 pgs: 3 active+recovery_wait+degraded, 80 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 4/78 objects degraded (5.128%) 2024-09-14T04:27:09.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:09.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:09.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2161504957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:09.136 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:09.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:09.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:08 smithi160 ceph-mon[88407]: pgmap v1456: 83 pgs: 3 active+recovery_wait+degraded, 80 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 4/78 objects degraded (5.128%) 2024-09-14T04:27:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2161504957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:09.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:09.259 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:09.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:09.361 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:09.382 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:09.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:09.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:09.506 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:09.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:09.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:09.634 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:09.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:09.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:09.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:09.757 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:09.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:09.859 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:09.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:09.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:09.982 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:10.004 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:10.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:10.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:10.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:10.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:10.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:10.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:27:10.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:10.260 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:10.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:10.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:10.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:10.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:10.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:10.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:10.511 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:10.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:10.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:10.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:10.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:10.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:10.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:27:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:27:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:27:10.698 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:10.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:10.762 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:10.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:10.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:10 smithi114 ceph-mon[103029]: pgmap v1457: 83 pgs: 1 active+recovery_wait+degraded, 82 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 1/78 objects degraded (1.282%) 2024-09-14T04:27:10.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:10 smithi114 ceph-mon[106969]: pgmap v1457: 83 pgs: 1 active+recovery_wait+degraded, 82 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 1/78 objects degraded (1.282%) 2024-09-14T04:27:10.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:10.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:10.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:10.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:11.015 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:11.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:11.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:11.140 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:11.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:10 smithi160 ceph-mon[88407]: pgmap v1457: 83 pgs: 1 active+recovery_wait+degraded, 82 active+clean; 582 KiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 1/78 objects degraded (1.282%) 2024-09-14T04:27:11.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:11.264 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:11.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:11.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:11.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:11.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:11.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:11.512 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:11.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:11.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:11.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:27:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:27:11] "GET /metrics HTTP/1.1" 200 34156 "" "Prometheus/2.51.0" 2024-09-14T04:27:11.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:11.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:11.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:11.738 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:11.761 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:11.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:11.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:11.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:11.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:11.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:11.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:27:11.947 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-14T04:27:11.948 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:11 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-14T04:27:11.986 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:12.008 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:12.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:12.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:12.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:12.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:11 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-14T04:27:12.235 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:12.256 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:12.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:12.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:12.380 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:12.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:12.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:12.504 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:12.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:12.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:12.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:12.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:12.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:12.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:12.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:12.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:12.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:12.874 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:12.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:12.898 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:27:12.898 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:27:12.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:12.998 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:12.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:12.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:13.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:13.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:12 smithi114 ceph-mon[106969]: pgmap v1458: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:13.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:12 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded) 2024-09-14T04:27:13.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:12 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-14T04:27:13.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:12 smithi114 ceph-mon[106969]: osdmap e383: 8 total, 8 up, 5 in 2024-09-14T04:27:13.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:12 smithi114 ceph-mon[103029]: pgmap v1458: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:13.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:12 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded) 2024-09-14T04:27:13.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:12 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-14T04:27:13.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:12 smithi114 ceph-mon[103029]: osdmap e383: 8 total, 8 up, 5 in 2024-09-14T04:27:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:13.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:13.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:12 smithi160 ceph-mon[88407]: pgmap v1458: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:12 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/78 objects degraded (1.282%), 1 pg degraded) 2024-09-14T04:27:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:12 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-14T04:27:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:12 smithi160 ceph-mon[88407]: osdmap e383: 8 total, 8 up, 5 in 2024-09-14T04:27:13.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:13.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:13.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:13.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:13.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:13.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:13.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:13.494 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:13.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:13.596 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:13.617 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:13.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:13.679 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-14T04:27:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:13.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:13.742 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:13.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:13.844 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:13.869 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:13.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:13.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:13.965 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:13 smithi114 ceph-mon[106969]: osdmap e384: 8 total, 8 up, 5 in 2024-09-14T04:27:13.966 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:13 smithi114 ceph-mon[103029]: osdmap e384: 8 total, 8 up, 5 in 2024-09-14T04:27:13.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:13.996 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:13.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:14.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:14.120 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:14.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:13 smithi160 ceph-mon[88407]: osdmap e384: 8 total, 8 up, 5 in 2024-09-14T04:27:14.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:14.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:14.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:14.256 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:14.293 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:14.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:14.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:14.470 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:14.497 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:14.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:14.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:14.625 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:14.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:14.702 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_9' created 2024-09-14T04:27:14.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:14.749 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:14.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:14.827 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:14 smithi114 ceph-mon[106969]: pgmap v1461: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-14T04:27:14.827 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:14 smithi114 ceph-mon[106969]: 1.9 scrub starts 2024-09-14T04:27:14.827 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:14 smithi114 ceph-mon[106969]: 1.5 scrub starts 2024-09-14T04:27:14.828 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:14 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1246812044' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-14T04:27:14.828 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:14 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-14T04:27:14.828 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:14 smithi114 ceph-mon[103029]: pgmap v1461: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-14T04:27:14.828 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:14 smithi114 ceph-mon[103029]: 1.9 scrub starts 2024-09-14T04:27:14.828 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:14 smithi114 ceph-mon[103029]: 1.5 scrub starts 2024-09-14T04:27:14.828 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:14 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1246812044' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-14T04:27:14.828 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:14 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-14T04:27:14.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:14.872 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:14.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:14.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:14.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:15.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:15.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:15.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:15.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:15.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:14 smithi160 ceph-mon[88407]: pgmap v1461: 83 pgs: 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-14T04:27:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:14 smithi160 ceph-mon[88407]: 1.9 scrub starts 2024-09-14T04:27:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:14 smithi160 ceph-mon[88407]: 1.5 scrub starts 2024-09-14T04:27:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:14 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1246812044' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-14T04:27:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:14 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-14T04:27:15.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:15.253 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:15.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:15.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:15.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:15.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:15.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:15.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:15.501 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:15.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:15.603 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:15.629 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:15.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:15.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:15.752 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:15.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:15.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:15.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:15 smithi114 ceph-mon[103029]: 1.4 scrub starts 2024-09-14T04:27:15.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:15 smithi114 ceph-mon[103029]: 1.2 scrub starts 2024-09-14T04:27:15.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:15 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-09-14T04:27:15.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:15 smithi114 ceph-mon[103029]: osdmap e385: 8 total, 8 up, 5 in 2024-09-14T04:27:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:15 smithi114 ceph-mon[106969]: 1.4 scrub starts 2024-09-14T04:27:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:15 smithi114 ceph-mon[106969]: 1.2 scrub starts 2024-09-14T04:27:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:15 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-09-14T04:27:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:15 smithi114 ceph-mon[106969]: osdmap e385: 8 total, 8 up, 5 in 2024-09-14T04:27:15.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:15.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:15.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:16.008 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:16.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:16.110 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:16.133 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:16.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:15 smithi160 ceph-mon[88407]: 1.4 scrub starts 2024-09-14T04:27:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:15 smithi160 ceph-mon[88407]: 1.2 scrub starts 2024-09-14T04:27:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:15 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-09-14T04:27:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:15 smithi160 ceph-mon[88407]: osdmap e385: 8 total, 8 up, 5 in 2024-09-14T04:27:16.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:16.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:16.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:16.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:16.388 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:16.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:16.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:16.516 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:16.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:16.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:16.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:16.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:16.741 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:16.763 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:16.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:16.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:16 smithi114 ceph-mon[103029]: pgmap v1463: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:16.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:16 smithi114 ceph-mon[103029]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:27:16.767 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:16 smithi114 ceph-mon[103029]: osdmap e386: 8 total, 8 up, 5 in 2024-09-14T04:27:16.768 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:16 smithi114 ceph-mon[106969]: pgmap v1463: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:16.768 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:16 smithi114 ceph-mon[106969]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:27:16.768 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:16 smithi114 ceph-mon[106969]: osdmap e386: 8 total, 8 up, 5 in 2024-09-14T04:27:16.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:16.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:16.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:16.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:16.993 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:17.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:17.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:17.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:17.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_9 rados --yes-i-really-mean-it || true 2024-09-14T04:27:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:17.138 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:17.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:17.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:16 smithi160 ceph-mon[88407]: pgmap v1463: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:17.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:16 smithi160 ceph-mon[88407]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-14T04:27:17.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:16 smithi160 ceph-mon[88407]: osdmap e386: 8 total, 8 up, 5 in 2024-09-14T04:27:17.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:17.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:17.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:17.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:17.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:17.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:17.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:17.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:17.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:17.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:17.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:17.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:17.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:17.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:17.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:17.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:17.895 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:17.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:17.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:17.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:18.018 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:18.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:18.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:17 smithi114 ceph-mon[103029]: osdmap e387: 8 total, 8 up, 5 in 2024-09-14T04:27:18.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:17 smithi114 ceph-mon[103029]: pgmap v1466: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:18.091 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:17 smithi114 ceph-mon[106969]: osdmap e387: 8 total, 8 up, 5 in 2024-09-14T04:27:18.091 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:17 smithi114 ceph-mon[106969]: pgmap v1466: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:18.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:18.141 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:18.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:17 smithi160 ceph-mon[88407]: osdmap e387: 8 total, 8 up, 5 in 2024-09-14T04:27:18.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:17 smithi160 ceph-mon[88407]: pgmap v1466: 99 pgs: 16 unknown, 83 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:18.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:18.265 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:18.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:18.366 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:18.388 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:18.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:18.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:18.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:18.516 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:18.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:18.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:18.643 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:18.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:18.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:18.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:18.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:18.869 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:18.885 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:27:18.886 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:27:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:18.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:18.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:18.903 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:27:18.903 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:27:18.919 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:27:18.919 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:27:18.935 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:27:18.936 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:27:18.952 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:27:18.952 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:27:18.968 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:27:18.969 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:27:18.984 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:27:18.984 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:27:18.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:19.000 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:27:19.001 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:27:19.017 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:19.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:19.120 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:19.141 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:19.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:19.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:19.269 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:19.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:19.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:19.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:19 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:19.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:19 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2315950100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:19.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:19 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:19.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2315950100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:19.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:19.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:19.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:19 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:19.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:19 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2315950100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:19.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:19.515 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:19.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:19.617 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:19.638 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:19.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:19.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:19.762 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:19.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:19.863 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:19.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:19.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:19.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:20.012 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:20.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:20.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:20.120 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:27:20.141 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:20.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:20.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:20.265 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:20.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:20.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:20.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:20 smithi114 ceph-mon[103029]: pgmap v1467: 99 pgs: 13 unknown, 86 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:20.363 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:20 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/610480150' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:27:20.363 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:20 smithi114 ceph-mon[106969]: pgmap v1467: 99 pgs: 13 unknown, 86 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:20.363 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:20 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/610480150' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:27:20.367 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:20.388 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:20.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:20.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:20.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:27:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:27:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:27:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:20 smithi160 ceph-mon[88407]: pgmap v1467: 99 pgs: 13 unknown, 86 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:20 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/610480150' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:27:20.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:20.512 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:20.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:20.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:20.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:20.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:20.736 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:20.762 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:20.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:20.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:20.890 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:20.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:20.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:21.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:21.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:21.048 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:21.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:21.139 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:21.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:21.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:21.268 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:21.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:21.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:21 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:27:21.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/610480150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:27:21.304 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:21 smithi114 ceph-mon[106969]: osdmap e388: 8 total, 8 up, 5 in 2024-09-14T04:27:21.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:21 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:27:21.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/610480150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:27:21.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:21 smithi114 ceph-mon[103029]: osdmap e388: 8 total, 8 up, 5 in 2024-09-14T04:27:21.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:21.390 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:21.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:21.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:21 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:27:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/610480150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:27:21.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:21 smithi160 ceph-mon[88407]: osdmap e388: 8 total, 8 up, 5 in 2024-09-14T04:27:21.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:21.514 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:21.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:21.588 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:27:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:27:21] "GET /metrics HTTP/1.1" 200 34156 "" "Prometheus/2.51.0" 2024-09-14T04:27:21.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:21.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:21.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:21.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:21.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:21.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:21.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:21.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:27:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:21.889 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:21.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:21.942 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:27:21.942 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:27:21.942 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:27:21.942 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:27:21.942 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:27:21.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:22.017 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:22.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:22.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:22.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:22.142 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:22.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:22.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:22.270 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:22.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:22.272 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:22 smithi114 ceph-mon[103029]: pgmap v1469: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:22.272 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:22 smithi114 ceph-mon[106969]: pgmap v1469: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:22.372 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:22.394 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:22.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:22.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:22 smithi160 ceph-mon[88407]: pgmap v1469: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:22.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:22.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:22.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:22.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:22.644 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:22.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:22.746 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:22.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:22.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:22.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:22.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:22.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:22.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:22.998 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:23.024 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:23.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:23.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:27:23.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:23.151 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:23.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:23.253 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:23.278 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:23.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:23.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:23.405 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:23.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:23.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:23.535 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:23.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:23.637 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:23.661 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:23.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:23.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-14T04:27:23.762 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:23.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:23.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:23.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:23.922 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:23.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:24.001 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:24.002 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:24.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:24.002 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:24.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:24.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:24.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:24.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:24.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:24.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:24.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:24.295 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:24.296 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:24.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:24.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:24.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:24.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:24.426 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:24.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:24.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:24.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:24.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:24.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:24.672 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:24.677 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:24.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:24.679 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rados' on pool 'unique_pool_9' 2024-09-14T04:27:24.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:24.806 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:24.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:24.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:24.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:24 smithi114 ceph-mon[106969]: pgmap v1470: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:24.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:24 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1268773296' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:27:24.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:24 smithi114 ceph-mon[103029]: pgmap v1470: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:24.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:24 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1268773296' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:27:24.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:24.929 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:24.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:25.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:25.051 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:25.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:25.153 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:24 smithi160 ceph-mon[88407]: pgmap v1470: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:24 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1268773296' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-14T04:27:25.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:25.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:25.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:25.304 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:25.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:25.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:25.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:25.429 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:25.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:25.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:25.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:25.553 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:25.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:25.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:25.676 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:25.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:25.778 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:25.801 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:25.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:25.857 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:25 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1268773296' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:27:25.857 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:25 smithi114 ceph-mon[106969]: osdmap e389: 8 total, 8 up, 5 in 2024-09-14T04:27:25.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:25 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1268773296' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:27:25.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:25 smithi114 ceph-mon[103029]: osdmap e389: 8 total, 8 up, 5 in 2024-09-14T04:27:25.903 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:25.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:25.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:26.026 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:26.048 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:26.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:26.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:25 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1268773296' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-14T04:27:26.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:25 smithi160 ceph-mon[88407]: osdmap e389: 8 total, 8 up, 5 in 2024-09-14T04:27:26.171 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:26.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:26.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:26.295 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:26.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:26.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:26.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:26.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:26.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:26.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:26.542 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:26.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:26.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:26.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:26.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:26.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:26.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:26.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:26.933 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:26.954 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:26.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:26.967 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:26 smithi114 ceph-mon[106969]: pgmap v1472: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:26.967 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:26 smithi114 ceph-mon[106969]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:27:26.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:26 smithi114 ceph-mon[103029]: pgmap v1472: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:26.967 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:26 smithi114 ceph-mon[103029]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:27:27.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:27.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:27.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:27.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:26 smithi160 ceph-mon[88407]: pgmap v1472: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:26 smithi160 ceph-mon[88407]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-14T04:27:27.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:27.200 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:27.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:27.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:27.324 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:27.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:27.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:27.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:27.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:27.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:27.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:27.572 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:27.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:27.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:27.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:27.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:27.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:27.820 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:27.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:27.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:27.942 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:27.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:28.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:28.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:28.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:28.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:28.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:28.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:28.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:28.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:28.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:28.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:28.438 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:28.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:28.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:28.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:28.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:28.658 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:27:28 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a[77001]: logger=cleanup t=2024-09-14T04:27:28.19313964Z level=info msg="Completed cleanup jobs" duration=12.847126ms 2024-09-14T04:27:28.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:28.683 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:28.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:28.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:28.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:28.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:28.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:28.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:27:28.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:27:28.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:27:28.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:27:28.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:27:28.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:27:28.851 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:27:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:27:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:27:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:27:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:27:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:27:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:27:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:27:28.852 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:27:28.853 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:27:28.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:28.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:28.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:28.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:29.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:29.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:29.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:28 smithi114 ceph-mon[106969]: pgmap v1473: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:29.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:28 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:29.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:28 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2863164864' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:29.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:28 smithi114 ceph-mon[103029]: pgmap v1473: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:29.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:28 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:29.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:28 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2863164864' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:29.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:29.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:28 smithi160 ceph-mon[88407]: pgmap v1473: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:28 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:28 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2863164864' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:29.179 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:29.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:29.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:29.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:29.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:29.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:29.425 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:29.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:29.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:29.549 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:29.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:29.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:29.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:29.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:29.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:29.797 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:29.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:29.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:29.921 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:29.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:30.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:30.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:30.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:30.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:30.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:30.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:30.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:30.290 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:30.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:30.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:30.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:30.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:30.487 DEBUG:teuthology.orchestra.run.smithi114:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_9 bench 90 write' 2024-09-14T04:27:30.488 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_9 2024-09-14T04:27:30.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:27:30.491 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-14T04:27:30.491 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-14T04:27:30.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:30.540 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:hints = 1 2024-09-14T04:27:30.540 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-14T04:27:30.540 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object prefix: benchmark_data_smithi114_329146 2024-09-14T04:27:30.540 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:27:30.540 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 0 0 0 0 0 0 - 0 2024-09-14T04:27:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:30.566 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:30.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:27:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:27:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:27:30.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:30.706 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:30.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:30.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:30.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:30.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:30.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:30.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:30.982 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:30.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:31.049 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:30 smithi114 ceph-mon[106969]: pgmap v1474: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:31.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:30 smithi114 ceph-mon[103029]: pgmap v1474: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:31.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:31.123 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:31.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:30 smithi160 ceph-mon[88407]: pgmap v1474: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:31.225 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:31.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:31.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:31.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:31.396 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:31.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:31.498 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:31.535 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:31.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:31.540 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 1 16 3701 3685 230.3 230.312 0.00655214 0.00432163 2024-09-14T04:27:31.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:27:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:27:31] "GET /metrics HTTP/1.1" 200 36501 "" "Prometheus/2.51.0" 2024-09-14T04:27:31.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:31.663 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:27:31.664 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:27:31.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:31.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:31.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:31.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:31.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:27:31.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:31.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:31.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:32.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:32.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:32.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:32.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:32.214 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:32.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:32.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:32.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:32.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:32.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:32.484 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:32.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:32.540 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 2 16 7219 7203 225.078 219.875 0.00366301 0.00443582 2024-09-14T04:27:32.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:32.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-14T04:27:32.620 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:32.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:32.695 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:32.705 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:32 smithi114 ceph-mon[103029]: pgmap v1475: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:32.726 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:27:32.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:32.766 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:32.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:32.867 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:32.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:32.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:32.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:32 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4033617830' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:27:32.970 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:32 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:27:32.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:32 smithi114 ceph-mon[106969]: pgmap v1475: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:32.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:32 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4033617830' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:27:32.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:32 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:27:33.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:33.043 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:33.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:33.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:33.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:32 smithi160 ceph-mon[88407]: pgmap v1475: 99 pgs: 99 active+clean; 582 KiB data, 5.1 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:32 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4033617830' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:27:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:32 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:27:33.179 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:33.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:33.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:33.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:33.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:33.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:33.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:33.452 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:33.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:33.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:33.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 3 16 10744 10728 223.483 220.312 0.00434071 0.00446828 2024-09-14T04:27:33.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:33.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:33.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:33.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:33.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:33.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:33.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:33.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:33.860 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:33.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:33 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:27:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:33 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:27:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:33 smithi114 ceph-mon[103029]: osdmap e390: 8 total, 8 up, 5 in 2024-09-14T04:27:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:33 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:27:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:33 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:27:33.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:33 smithi114 ceph-mon[106969]: osdmap e390: 8 total, 8 up, 5 in 2024-09-14T04:27:33.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:33.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:33.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:34.099 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:34.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:33 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:27:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:33 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:27:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:33 smithi160 ceph-mon[88407]: osdmap e390: 8 total, 8 up, 5 in 2024-09-14T04:27:34.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:34.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:34.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:34.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:34.413 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:34.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:34.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:34.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 4 16 14240 14224 222.233 218.5 0.00360918 0.00449288 2024-09-14T04:27:34.550 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:34.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:34.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:34.686 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:34.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:34.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:34.823 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:34.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:34.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:34 smithi114 ceph-mon[103029]: pgmap v1477: 99 pgs: 99 active+clean; 582 KiB data, 5.3 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:34.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:34 smithi114 ceph-mon[106969]: pgmap v1477: 99 pgs: 99 active+clean; 582 KiB data, 5.3 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:34.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:34.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:34.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:35.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:35.099 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:35.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:34 smithi160 ceph-mon[88407]: pgmap v1477: 99 pgs: 99 active+clean; 582 KiB data, 5.3 GiB used, 442 GiB / 447 GiB avail 2024-09-14T04:27:35.202 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:35.238 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:35.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:35.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:35.375 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:35.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:35.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:35.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:35.511 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:35.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:35.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 5 16 17523 17507 218.82 205.188 0.00434536 0.00456442 2024-09-14T04:27:35.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:35.647 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:35.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:35.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:35.794 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:35.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:35.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:35 smithi114 ceph-mon[103029]: 1.0 scrub starts 2024-09-14T04:27:35.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:35 smithi114 ceph-mon[103029]: 1.0 scrub ok 2024-09-14T04:27:35.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:35 smithi114 ceph-mon[106969]: 1.0 scrub starts 2024-09-14T04:27:35.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:35 smithi114 ceph-mon[106969]: 1.0 scrub ok 2024-09-14T04:27:35.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:35.929 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:35.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:36.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:36.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:36.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:36.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:35 smithi160 ceph-mon[88407]: 1.0 scrub starts 2024-09-14T04:27:36.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:35 smithi160 ceph-mon[88407]: 1.0 scrub ok 2024-09-14T04:27:36.169 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:36.204 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:36.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:36.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:36.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:36.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:36.476 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:36.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:36.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 6 16 20562 20546 214.003 189.938 0.00661734 0.00466684 2024-09-14T04:27:36.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:36.616 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:36.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:36.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:36.755 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:36.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:36.805 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:36 smithi114 ceph-mon[103029]: pgmap v1478: 99 pgs: 99 active+clean; 869 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T04:27:36.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:36 smithi114 ceph-mon[103029]: 1.a scrub starts 2024-09-14T04:27:36.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:36 smithi114 ceph-mon[103029]: 1.a scrub ok 2024-09-14T04:27:36.806 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:36 smithi114 ceph-mon[106969]: pgmap v1478: 99 pgs: 99 active+clean; 869 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T04:27:36.806 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:36 smithi114 ceph-mon[106969]: 1.a scrub starts 2024-09-14T04:27:36.806 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:36 smithi114 ceph-mon[106969]: 1.a scrub ok 2024-09-14T04:27:36.856 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:36.893 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:36.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:36.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:36.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:37.032 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:37.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:37.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:27:37.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:36 smithi160 ceph-mon[88407]: pgmap v1478: 99 pgs: 99 active+clean; 869 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T04:27:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:36 smithi160 ceph-mon[88407]: 1.a scrub starts 2024-09-14T04:27:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:36 smithi160 ceph-mon[88407]: 1.a scrub ok 2024-09-14T04:27:37.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:37.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:37.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:37.306 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:37.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:37.409 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:37.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:37.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:37.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 7 16 23327 23311 208.117 172.812 0.00685243 0.00480016 2024-09-14T04:27:37.546 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:37.583 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:37.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:37.685 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:37.716 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:37.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:37.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:37.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:37.853 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:37.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:37.896 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:37 smithi114 ceph-mon[103029]: 1.3 scrub starts 2024-09-14T04:27:37.896 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:37 smithi114 ceph-mon[103029]: 1.3 scrub ok 2024-09-14T04:27:37.897 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:37 smithi114 ceph-mon[103029]: pgmap v1479: 99 pgs: 99 active+clean; 869 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T04:27:37.897 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:37.897 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:37.897 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:37 smithi114 ceph-mon[106969]: 1.3 scrub starts 2024-09-14T04:27:37.897 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:37 smithi114 ceph-mon[106969]: 1.3 scrub ok 2024-09-14T04:27:37.897 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:37 smithi114 ceph-mon[106969]: pgmap v1479: 99 pgs: 99 active+clean; 869 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T04:27:37.897 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:37.897 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:37.955 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:37.979 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:37.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:38.081 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:38.115 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:38.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:37 smithi160 ceph-mon[88407]: 1.3 scrub starts 2024-09-14T04:27:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:37 smithi160 ceph-mon[88407]: 1.3 scrub ok 2024-09-14T04:27:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:37 smithi160 ceph-mon[88407]: pgmap v1479: 99 pgs: 99 active+clean; 869 MiB data, 6.9 GiB used, 440 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-14T04:27:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:27:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:38.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:38.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:38.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:38.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:38.389 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:38.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:38.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:38.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:38.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:38.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 8 16 25665 25649 200.366 146.125 0.00688264 0.00498568 2024-09-14T04:27:38.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:38.659 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:27:38.659 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:27:38.659 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:27:38.659 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:27:38.659 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:27:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:27:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:27:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:27:38.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:27:38.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:27:38.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:27:38.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:27:38.660 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:27:38.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:27:38.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:27:38.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:27:38.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:38.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:38.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:38.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:38.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:38.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:38.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:38.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:38.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:38.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:39.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:39.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:39.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:39.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:38 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2151469357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:39.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:38 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2151469357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:38 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2151469357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:27:39.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:39.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:39.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:39.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:39.345 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:39.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:39.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:39.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:39.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:39.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:39.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 9 16 28122 28106 195.164 153.562 0.0124767 0.00511891 2024-09-14T04:27:39.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:39.616 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:39.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:39.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:39.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:39.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:39.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:39.877 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:39.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:39.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:39.999 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:40.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:40.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:39 smithi114 ceph-mon[103029]: 1.7 scrub starts 2024-09-14T04:27:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:39 smithi114 ceph-mon[103029]: 1.7 scrub ok 2024-09-14T04:27:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:39 smithi114 ceph-mon[103029]: pgmap v1480: 99 pgs: 99 active+clean; 1.0 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-14T04:27:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:39 smithi114 ceph-mon[106969]: 1.7 scrub starts 2024-09-14T04:27:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:39 smithi114 ceph-mon[106969]: 1.7 scrub ok 2024-09-14T04:27:40.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:39 smithi114 ceph-mon[106969]: pgmap v1480: 99 pgs: 99 active+clean; 1.0 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-14T04:27:40.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:40.138 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:40.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:40.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:39 smithi160 ceph-mon[88407]: 1.7 scrub starts 2024-09-14T04:27:40.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:39 smithi160 ceph-mon[88407]: 1.7 scrub ok 2024-09-14T04:27:40.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:39 smithi160 ceph-mon[88407]: pgmap v1480: 99 pgs: 99 active+clean; 1.0 GiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-09-14T04:27:40.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:40.275 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:40.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:40.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:40.413 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:40.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:40.516 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:40.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 10 16 30086 30070 187.922 122.75 0.00535022 0.00531583 2024-09-14T04:27:40.551 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:40.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:40.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:27:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:27:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:27:40.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:40.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:40.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:40.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:40.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:40.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:40.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:40.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:41.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:41.074 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:41.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:41.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:41.208 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:41.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:41.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:41.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:41.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:41.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:41.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:41.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 11 16 31973 31957 181.558 117.938 0.00421625 0.00549483 2024-09-14T04:27:41.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:41.617 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:41.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:41.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:41.753 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:27:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:27:41] "GET /metrics HTTP/1.1" 200 36563 "" "Prometheus/2.51.0" 2024-09-14T04:27:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:41.758 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:41.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:41.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:41.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:41.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:42.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:42.021 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:42.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:42.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:42.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:42.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:42.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:42.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:42.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:42.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:42.430 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:42.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:42.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:42.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 12 16 33979 33963 176.876 125.375 0.0070525 0.00564862 2024-09-14T04:27:42.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:42 smithi114 ceph-mon[106969]: pgmap v1481: 99 pgs: 99 active+clean; 1.7 GiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 174 MiB/s wr, 2.78k op/s 2024-09-14T04:27:42.566 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:42 smithi114 ceph-mon[103029]: pgmap v1481: 99 pgs: 99 active+clean; 1.7 GiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 174 MiB/s wr, 2.78k op/s 2024-09-14T04:27:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:42.568 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:42.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:42 smithi160 ceph-mon[88407]: pgmap v1481: 99 pgs: 99 active+clean; 1.7 GiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 174 MiB/s wr, 2.78k op/s 2024-09-14T04:27:42.669 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:42.686 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:27:42.688 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:27:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:42.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:42.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:42.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:42.829 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:42.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:42.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:42.967 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:42.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:43.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:43.089 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:43.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:43.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:43.221 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:43.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:43.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:43.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:43.363 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:43.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:43.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:43 smithi114 ceph-mon[106969]: from='client.47198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:27:43.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:43 smithi114 ceph-mon[103029]: from='client.47198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:27:43.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:43.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:43.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:43.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 13 16 35800 35784 172.024 113.812 0.0146792 0.00580771 2024-09-14T04:27:43.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:43.638 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:43.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:43.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:43 smithi160 ceph-mon[88407]: from='client.47198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:27:43.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:43.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:43.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:43.876 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:43.916 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:43.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:44.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:44.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:44.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:44.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:44.193 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:44.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:44.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:44.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:44.318 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:44.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:44.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:44.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:44.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:44.541 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 14 16 37654 37638 168.012 115.875 0.01017 0.00594636 2024-09-14T04:27:44.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:44.591 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:44.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[103029]: 1.1 scrub starts 2024-09-14T04:27:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[103029]: 1.1 scrub ok 2024-09-14T04:27:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[103029]: pgmap v1482: 99 pgs: 99 active+clean; 1.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 166 MiB/s wr, 2.66k op/s 2024-09-14T04:27:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[103029]: 1.9 scrub starts 2024-09-14T04:27:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[103029]: 1.9 scrub ok 2024-09-14T04:27:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[103029]: 1.6 scrub starts 2024-09-14T04:27:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[103029]: 1.6 scrub ok 2024-09-14T04:27:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[103029]: 1.5 scrub starts 2024-09-14T04:27:44.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[103029]: 1.5 scrub ok 2024-09-14T04:27:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[106969]: 1.1 scrub starts 2024-09-14T04:27:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[106969]: 1.1 scrub ok 2024-09-14T04:27:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[106969]: pgmap v1482: 99 pgs: 99 active+clean; 1.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 166 MiB/s wr, 2.66k op/s 2024-09-14T04:27:44.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[106969]: 1.9 scrub starts 2024-09-14T04:27:44.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[106969]: 1.9 scrub ok 2024-09-14T04:27:44.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[106969]: 1.6 scrub starts 2024-09-14T04:27:44.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[106969]: 1.6 scrub ok 2024-09-14T04:27:44.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[106969]: 1.5 scrub starts 2024-09-14T04:27:44.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:44 smithi114 ceph-mon[106969]: 1.5 scrub ok 2024-09-14T04:27:44.643 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:44 smithi160 ceph-mon[88407]: 1.1 scrub starts 2024-09-14T04:27:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:44 smithi160 ceph-mon[88407]: 1.1 scrub ok 2024-09-14T04:27:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:44 smithi160 ceph-mon[88407]: pgmap v1482: 99 pgs: 99 active+clean; 1.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 166 MiB/s wr, 2.66k op/s 2024-09-14T04:27:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:44 smithi160 ceph-mon[88407]: 1.9 scrub starts 2024-09-14T04:27:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:44 smithi160 ceph-mon[88407]: 1.9 scrub ok 2024-09-14T04:27:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:44 smithi160 ceph-mon[88407]: 1.6 scrub starts 2024-09-14T04:27:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:44 smithi160 ceph-mon[88407]: 1.6 scrub ok 2024-09-14T04:27:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:44 smithi160 ceph-mon[88407]: 1.5 scrub starts 2024-09-14T04:27:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:44 smithi160 ceph-mon[88407]: 1.5 scrub ok 2024-09-14T04:27:44.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:44.719 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:44.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:44.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:44.821 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:44.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:44.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:44.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:44.998 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:44.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:45.100 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:45.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:45.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:45.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:45.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:45.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:45.379 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:45.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:45.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:45.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:45.542 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 15 16 39581 39565 164.835 120.438 0.00626716 0.00606061 2024-09-14T04:27:45.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:45 smithi114 ceph-mon[103029]: 1.4 scrub starts 2024-09-14T04:27:45.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:45 smithi114 ceph-mon[103029]: 1.4 scrub ok 2024-09-14T04:27:45.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:45 smithi114 ceph-mon[103029]: 1.2 scrub starts 2024-09-14T04:27:45.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:45 smithi114 ceph-mon[103029]: 1.2 scrub ok 2024-09-14T04:27:45.543 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:45 smithi114 ceph-mon[106969]: 1.4 scrub starts 2024-09-14T04:27:45.543 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:45 smithi114 ceph-mon[106969]: 1.4 scrub ok 2024-09-14T04:27:45.543 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:45 smithi114 ceph-mon[106969]: 1.2 scrub starts 2024-09-14T04:27:45.543 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:45 smithi114 ceph-mon[106969]: 1.2 scrub ok 2024-09-14T04:27:45.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:45.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:45.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:45.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:45 smithi160 ceph-mon[88407]: 1.4 scrub starts 2024-09-14T04:27:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:45 smithi160 ceph-mon[88407]: 1.4 scrub ok 2024-09-14T04:27:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:45 smithi160 ceph-mon[88407]: 1.2 scrub starts 2024-09-14T04:27:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:45 smithi160 ceph-mon[88407]: 1.2 scrub ok 2024-09-14T04:27:45.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:45.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:45.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:45.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:45.804 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:45.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:45.905 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:45.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:46.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:46.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:46.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:46.172 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:46.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:46.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:46.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:46.343 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:46.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:46.444 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:46.479 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:46.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:46.542 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 16 16 41581 41565 162.344 125 0.0100661 0.00615523 2024-09-14T04:27:46.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:46.585 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:46 smithi114 ceph-mon[103029]: pgmap v1483: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 195 MiB/s wr, 3.12k op/s 2024-09-14T04:27:46.585 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:46 smithi114 ceph-mon[106969]: pgmap v1483: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 195 MiB/s wr, 3.12k op/s 2024-09-14T04:27:46.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:46.627 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:46.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:46 smithi160 ceph-mon[88407]: pgmap v1483: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 195 MiB/s wr, 3.12k op/s 2024-09-14T04:27:46.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:46.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:46.762 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:46.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:46.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:46.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:46.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:47.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:47.035 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:47.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:47.118 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1481,"stamp":"2024-09-14T04:27:41.186052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1825493303,"num_objects":27875,"num_object_clones":0,"num_object_copies":83625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27875,"num_whiteouts":0,"num_read":187,"num_read_kb":476,"num_write":28008,"num_write_kb":1783403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":973001,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29007,"ondisk_log_size":29007,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":10359540,"kb_used_data":4978700,"kb_used_omap":20,"kb_used_meta":5380779,"kb_avail":458366220,"statfs":{"total":479975178240,"available":469367009280,"internally_reserved":0,"allocated":5098188800,"data_stored":5088718778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20935,"internal_metadata":5509918265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1824897386,"num_objects":27849,"num_object_clones":0,"num_object_copies":83547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27849,"num_whiteouts":0,"num_read":-105,"num_read_kb":-271,"num_write":27796,"num_write_kb":1781744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-214678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002621"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":616,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724047+0000","last_change":"2024-09-14T04:25:42.350690+0000","last_active":"2024-09-14T04:27:32.724047+0000","last_peered":"2024-09-14T04:27:32.724047+0000","last_clean":"2024-09-14T04:27:32.724047+0000","last_became_active":"2024-09-14T04:25:42.350363+0000","last_became_peered":"2024-09-14T04:25:42.350363+0000","last_unstale":"2024-09-14T04:27:32.724047+0000","last_undegraded":"2024-09-14T04:27:32.724047+0000","last_fullsized":"2024-09-14T04:27:32.724047+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:11:38.039845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":76,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.726526+0000","last_change":"2024-09-14T04:25:42.287293+0000","last_active":"2024-09-14T04:27:32.726526+0000","last_peered":"2024-09-14T04:27:32.726526+0000","last_clean":"2024-09-14T04:27:32.726526+0000","last_became_active":"2024-09-14T04:25:42.237213+0000","last_became_peered":"2024-09-14T04:25:42.237213+0000","last_unstale":"2024-09-14T04:27:32.726526+0000","last_undegraded":"2024-09-14T04:27:32.726526+0000","last_fullsized":"2024-09-14T04:27:32.726526+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:49:14.895808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":85,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.683002+0000","last_change":"2024-09-14T04:25:03.863433+0000","last_active":"2024-09-14T04:27:24.683002+0000","last_peered":"2024-09-14T04:27:24.683002+0000","last_clean":"2024-09-14T04:27:24.683002+0000","last_became_active":"2024-09-14T04:25:03.863153+0000","last_became_peered":"2024-09-14T04:25:03.863153+0000","last_unstale":"2024-09-14T04:27:24.683002+0000","last_undegraded":"2024-09-14T04:27:24.683002+0000","last_fullsized":"2024-09-14T04:27:24.683002+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:36:30.030688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":441,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.727260+0000","last_change":"2024-09-14T04:26:18.735795+0000","last_active":"2024-09-14T04:27:32.727260+0000","last_peered":"2024-09-14T04:27:32.727260+0000","last_clean":"2024-09-14T04:27:32.727260+0000","last_became_active":"2024-09-14T04:26:18.735453+0000","last_became_peered":"2024-09-14T04:26:18.735453+0000","last_unstale":"2024-09-14T04:27:32.727260+0000","last_undegraded":"2024-09-14T04:27:32.727260+0000","last_fullsized":"2024-09-14T04:27:32.727260+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:11:19.345201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":295,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.725600+0000","last_change":"2024-09-14T04:26:15.728281+0000","last_active":"2024-09-14T04:27:32.725600+0000","last_peered":"2024-09-14T04:27:32.725600+0000","last_clean":"2024-09-14T04:27:32.725600+0000","last_became_active":"2024-09-14T04:26:15.727976+0000","last_became_peered":"2024-09-14T04:26:15.727976+0000","last_unstale":"2024-09-14T04:27:32.725600+0000","last_undegraded":"2024-09-14T04:27:32.725600+0000","last_fullsized":"2024-09-14T04:27:32.725600+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:42.362106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":363,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.684997+0000","last_change":"2024-09-14T04:26:15.737623+0000","last_active":"2024-09-14T04:27:24.684997+0000","last_peered":"2024-09-14T04:27:24.684997+0000","last_clean":"2024-09-14T04:27:24.684997+0000","last_became_active":"2024-09-14T04:26:15.731253+0000","last_became_peered":"2024-09-14T04:26:15.731253+0000","last_unstale":"2024-09-14T04:27:24.684997+0000","last_undegraded":"2024-09-14T04:27:24.684997+0000","last_fullsized":"2024-09-14T04:27:24.684997+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:25:47.213793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":429,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724465+0000","last_change":"2024-09-14T04:25:42.334456+0000","last_active":"2024-09-14T04:27:32.724465+0000","last_peered":"2024-09-14T04:27:32.724465+0000","last_clean":"2024-09-14T04:27:32.724465+0000","last_became_active":"2024-09-14T04:25:42.334177+0000","last_became_peered":"2024-09-14T04:25:42.334177+0000","last_unstale":"2024-09-14T04:27:32.724465+0000","last_undegraded":"2024-09-14T04:27:32.724465+0000","last_fullsized":"2024-09-14T04:27:32.724465+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:12:07.154369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":424,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.684595+0000","last_change":"2024-09-14T04:26:15.744615+0000","last_active":"2024-09-14T04:27:24.684595+0000","last_peered":"2024-09-14T04:27:24.684595+0000","last_clean":"2024-09-14T04:27:24.684595+0000","last_became_active":"2024-09-14T04:26:15.726296+0000","last_became_peered":"2024-09-14T04:26:15.726296+0000","last_unstale":"2024-09-14T04:27:24.684595+0000","last_undegraded":"2024-09-14T04:27:24.684595+0000","last_fullsized":"2024-09-14T04:27:24.684595+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:46:10.656996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.2","version":"390'1673","reported_seq":1686,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:38.893717+0000","last_change":"2024-09-14T04:27:15.715948+0000","last_active":"2024-09-14T04:27:38.893717+0000","last_peered":"2024-09-14T04:27:38.893717+0000","last_clean":"2024-09-14T04:27:38.893717+0000","last_became_active":"2024-09-14T04:27:15.715651+0000","last_became_peered":"2024-09-14T04:27:15.715651+0000","last_unstale":"2024-09-14T04:27:38.893717+0000","last_undegraded":"2024-09-14T04:27:38.893717+0000","last_fullsized":"2024-09-14T04:27:38.893717+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:16:28.089877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109641728,"num_objects":1673,"num_object_clones":0,"num_object_copies":5019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1673,"num_write_kb":107072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":389,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.683101+0000","last_change":"2024-09-14T04:26:15.750280+0000","last_active":"2024-09-14T04:27:24.683101+0000","last_peered":"2024-09-14T04:27:24.683101+0000","last_clean":"2024-09-14T04:27:24.683101+0000","last_became_active":"2024-09-14T04:26:15.736153+0000","last_became_peered":"2024-09-14T04:26:15.736153+0000","last_unstale":"2024-09-14T04:27:24.683101+0000","last_undegraded":"2024-09-14T04:27:24.683101+0000","last_fullsized":"2024-09-14T04:27:24.683101+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:00:25.087633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":85,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682367+0000","last_change":"2024-09-14T04:23:09.282698+0000","last_active":"2024-09-14T04:27:24.682367+0000","last_peered":"2024-09-14T04:27:24.682367+0000","last_clean":"2024-09-14T04:27:24.682367+0000","last_became_active":"2024-09-14T04:23:09.282181+0000","last_became_peered":"2024-09-14T04:23:09.282181+0000","last_unstale":"2024-09-14T04:27:24.682367+0000","last_undegraded":"2024-09-14T04:27:24.682367+0000","last_fullsized":"2024-09-14T04:27:24.682367+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:18:12.880523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"390'1859","reported_seq":1880,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.671882+0000","last_change":"2024-09-14T04:27:15.713659+0000","last_active":"2024-09-14T04:27:39.671882+0000","last_peered":"2024-09-14T04:27:39.671882+0000","last_clean":"2024-09-14T04:27:39.671882+0000","last_became_active":"2024-09-14T04:27:15.713372+0000","last_became_peered":"2024-09-14T04:27:15.713372+0000","last_unstale":"2024-09-14T04:27:39.671882+0000","last_undegraded":"2024-09-14T04:27:39.671882+0000","last_fullsized":"2024-09-14T04:27:39.671882+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1859,"log_dups_size":0,"ondisk_log_size":1859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:54:24.550706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121831424,"num_objects":1859,"num_object_clones":0,"num_object_copies":5577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1859,"num_write_kb":118976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":66,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682793+0000","last_change":"2024-09-14T04:26:15.755751+0000","last_active":"2024-09-14T04:27:24.682793+0000","last_peered":"2024-09-14T04:27:24.682793+0000","last_clean":"2024-09-14T04:27:24.682793+0000","last_became_active":"2024-09-14T04:26:15.745572+0000","last_became_peered":"2024-09-14T04:26:15.745572+0000","last_unstale":"2024-09-14T04:27:24.682793+0000","last_undegraded":"2024-09-14T04:27:24.682793+0000","last_fullsized":"2024-09-14T04:27:24.682793+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:54:48.838252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":398,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.727053+0000","last_change":"2024-09-14T04:24:47.613225+0000","last_active":"2024-09-14T04:27:32.727053+0000","last_peered":"2024-09-14T04:27:32.727053+0000","last_clean":"2024-09-14T04:27:32.727053+0000","last_became_active":"2024-09-14T04:24:47.612805+0000","last_became_peered":"2024-09-14T04:24:47.612805+0000","last_unstale":"2024-09-14T04:27:32.727053+0000","last_undegraded":"2024-09-14T04:27:32.727053+0000","last_fullsized":"2024-09-14T04:27:32.727053+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:35.992716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"390'1823","reported_seq":1844,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.616982+0000","last_change":"2024-09-14T04:27:15.713527+0000","last_active":"2024-09-14T04:27:39.616982+0000","last_peered":"2024-09-14T04:27:39.616982+0000","last_clean":"2024-09-14T04:27:39.616982+0000","last_became_active":"2024-09-14T04:27:15.713014+0000","last_became_peered":"2024-09-14T04:27:15.713014+0000","last_unstale":"2024-09-14T04:27:39.616982+0000","last_undegraded":"2024-09-14T04:27:39.616982+0000","last_fullsized":"2024-09-14T04:27:39.616982+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1823,"log_dups_size":0,"ondisk_log_size":1823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:07:39.018526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119472128,"num_objects":1823,"num_object_clones":0,"num_object_copies":5469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1823,"num_write_kb":116672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":93,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682714+0000","last_change":"2024-09-14T04:25:03.878325+0000","last_active":"2024-09-14T04:27:24.682714+0000","last_peered":"2024-09-14T04:27:24.682714+0000","last_clean":"2024-09-14T04:27:24.682714+0000","last_became_active":"2024-09-14T04:25:03.861286+0000","last_became_peered":"2024-09-14T04:25:03.861286+0000","last_unstale":"2024-09-14T04:27:24.682714+0000","last_undegraded":"2024-09-14T04:27:24.682714+0000","last_fullsized":"2024-09-14T04:27:24.682714+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:57:29.738051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":405,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.684718+0000","last_change":"2024-09-14T04:26:15.726841+0000","last_active":"2024-09-14T04:27:24.684718+0000","last_peered":"2024-09-14T04:27:24.684718+0000","last_clean":"2024-09-14T04:27:24.684718+0000","last_became_active":"2024-09-14T04:26:15.725967+0000","last_became_peered":"2024-09-14T04:26:15.725967+0000","last_unstale":"2024-09-14T04:27:24.684718+0000","last_undegraded":"2024-09-14T04:27:24.684718+0000","last_fullsized":"2024-09-14T04:27:24.684718+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:30:20.743135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1","version":"390'1787","reported_seq":1808,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.683886+0000","last_change":"2024-09-14T04:27:15.712263+0000","last_active":"2024-09-14T04:27:39.683886+0000","last_peered":"2024-09-14T04:27:39.683886+0000","last_clean":"2024-09-14T04:27:39.683886+0000","last_became_active":"2024-09-14T04:27:15.711969+0000","last_became_peered":"2024-09-14T04:27:15.711969+0000","last_unstale":"2024-09-14T04:27:39.683886+0000","last_undegraded":"2024-09-14T04:27:39.683886+0000","last_fullsized":"2024-09-14T04:27:39.683886+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1787,"log_dups_size":0,"ondisk_log_size":1787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:59:28.627517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117112832,"num_objects":1787,"num_object_clones":0,"num_object_copies":5361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1787,"num_write_kb":114368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":75,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682779+0000","last_change":"2024-09-14T04:22:08.468750+0000","last_active":"2024-09-14T04:27:24.682779+0000","last_peered":"2024-09-14T04:27:24.682779+0000","last_clean":"2024-09-14T04:27:24.682779+0000","last_became_active":"2024-09-14T04:22:08.468217+0000","last_became_peered":"2024-09-14T04:22:08.468217+0000","last_unstale":"2024-09-14T04:27:24.682779+0000","last_undegraded":"2024-09-14T04:27:24.682779+0000","last_fullsized":"2024-09-14T04:27:24.682779+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:06.709182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":104,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.726600+0000","last_change":"2024-09-14T04:26:15.732663+0000","last_active":"2024-09-14T04:27:32.726600+0000","last_peered":"2024-09-14T04:27:32.726600+0000","last_clean":"2024-09-14T04:27:32.726600+0000","last_became_active":"2024-09-14T04:26:15.731910+0000","last_became_peered":"2024-09-14T04:26:15.731910+0000","last_unstale":"2024-09-14T04:27:32.726600+0000","last_undegraded":"2024-09-14T04:27:32.726600+0000","last_fullsized":"2024-09-14T04:27:32.726600+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:44.768223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.6","version":"390'1812","reported_seq":1833,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.601990+0000","last_change":"2024-09-14T04:27:15.712649+0000","last_active":"2024-09-14T04:27:39.601990+0000","last_peered":"2024-09-14T04:27:39.601990+0000","last_clean":"2024-09-14T04:27:39.601990+0000","last_became_active":"2024-09-14T04:27:15.711674+0000","last_became_peered":"2024-09-14T04:27:15.711674+0000","last_unstale":"2024-09-14T04:27:39.601990+0000","last_undegraded":"2024-09-14T04:27:39.601990+0000","last_fullsized":"2024-09-14T04:27:39.601990+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1812,"log_dups_size":0,"ondisk_log_size":1812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:25:43.742796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118751232,"num_objects":1812,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1812,"num_write_kb":115968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'100","reported_seq":27,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.726438+0000","last_change":"2024-09-14T04:27:14.686098+0000","last_active":"2024-09-14T04:27:32.726438+0000","last_peered":"2024-09-14T04:27:32.726438+0000","last_clean":"2024-09-14T04:27:32.726438+0000","last_became_active":"2024-09-14T04:27:12.689443+0000","last_became_peered":"2024-09-14T04:27:12.689443+0000","last_unstale":"2024-09-14T04:27:32.726438+0000","last_undegraded":"2024-09-14T04:27:32.726438+0000","last_fullsized":"2024-09-14T04:27:32.726438+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-14T04:27:43.286588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":656,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.726848+0000","last_change":"2024-09-14T04:26:15.751894+0000","last_active":"2024-09-14T04:27:32.726848+0000","last_peered":"2024-09-14T04:27:32.726848+0000","last_clean":"2024-09-14T04:27:32.726848+0000","last_became_active":"2024-09-14T04:26:15.731434+0000","last_became_peered":"2024-09-14T04:26:15.731434+0000","last_unstale":"2024-09-14T04:27:32.726848+0000","last_undegraded":"2024-09-14T04:27:32.726848+0000","last_fullsized":"2024-09-14T04:27:32.726848+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:17.854732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":482,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724125+0000","last_change":"2024-09-14T04:17:48.083765+0000","last_active":"2024-09-14T04:27:32.724125+0000","last_peered":"2024-09-14T04:27:32.724125+0000","last_clean":"2024-09-14T04:27:32.724125+0000","last_became_active":"2024-09-14T04:17:48.083385+0000","last_became_peered":"2024-09-14T04:17:48.083385+0000","last_unstale":"2024-09-14T04:27:32.724125+0000","last_undegraded":"2024-09-14T04:27:32.724125+0000","last_fullsized":"2024-09-14T04:27:32.724125+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:13:49.268399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.7","version":"390'1739","reported_seq":1752,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.228462+0000","last_change":"2024-09-14T04:27:15.714043+0000","last_active":"2024-09-14T04:27:39.228462+0000","last_peered":"2024-09-14T04:27:39.228462+0000","last_clean":"2024-09-14T04:27:39.228462+0000","last_became_active":"2024-09-14T04:27:15.713722+0000","last_became_peered":"2024-09-14T04:27:15.713722+0000","last_unstale":"2024-09-14T04:27:39.228462+0000","last_undegraded":"2024-09-14T04:27:39.228462+0000","last_fullsized":"2024-09-14T04:27:39.228462+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1739,"log_dups_size":0,"ondisk_log_size":1739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:48:47.560579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113967104,"num_objects":1739,"num_object_clones":0,"num_object_copies":5217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1739,"num_write_kb":111296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"76'100","reported_seq":21,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.681895+0000","last_change":"2024-09-14T04:27:12.699187+0000","last_active":"2024-09-14T04:27:24.681895+0000","last_peered":"2024-09-14T04:27:24.681895+0000","last_clean":"2024-09-14T04:27:24.681895+0000","last_became_active":"2024-09-14T04:27:12.688140+0000","last_became_peered":"2024-09-14T04:27:12.688140+0000","last_unstale":"2024-09-14T04:27:24.681895+0000","last_undegraded":"2024-09-14T04:27:24.681895+0000","last_fullsized":"2024-09-14T04:27:24.681895+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":386,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.684833+0000","last_change":"2024-09-14T04:24:47.615423+0000","last_active":"2024-09-14T04:27:24.684833+0000","last_peered":"2024-09-14T04:27:24.684833+0000","last_clean":"2024-09-14T04:27:24.684833+0000","last_became_active":"2024-09-14T04:24:47.609944+0000","last_became_peered":"2024-09-14T04:24:47.609944+0000","last_unstale":"2024-09-14T04:27:24.684833+0000","last_undegraded":"2024-09-14T04:27:24.684833+0000","last_fullsized":"2024-09-14T04:27:24.684833+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:30:03.475490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":70,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724719+0000","last_change":"2024-09-14T04:25:42.247602+0000","last_active":"2024-09-14T04:27:32.724719+0000","last_peered":"2024-09-14T04:27:32.724719+0000","last_clean":"2024-09-14T04:27:32.724719+0000","last_became_active":"2024-09-14T04:25:42.247304+0000","last_became_peered":"2024-09-14T04:25:42.247304+0000","last_unstale":"2024-09-14T04:27:32.724719+0000","last_undegraded":"2024-09-14T04:27:32.724719+0000","last_fullsized":"2024-09-14T04:27:32.724719+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:18:17.260700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"390'1631","reported_seq":1644,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:38.899950+0000","last_change":"2024-09-14T04:27:15.715912+0000","last_active":"2024-09-14T04:27:38.899950+0000","last_peered":"2024-09-14T04:27:38.899950+0000","last_clean":"2024-09-14T04:27:38.899950+0000","last_became_active":"2024-09-14T04:27:15.715613+0000","last_became_peered":"2024-09-14T04:27:15.715613+0000","last_unstale":"2024-09-14T04:27:38.899950+0000","last_undegraded":"2024-09-14T04:27:38.899950+0000","last_fullsized":"2024-09-14T04:27:38.899950+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1631,"log_dups_size":0,"ondisk_log_size":1631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:12:14.385531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106889216,"num_objects":1631,"num_object_clones":0,"num_object_copies":4893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1631,"num_write_kb":104384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":213,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724978+0000","last_change":"2024-09-14T04:25:53.296317+0000","last_active":"2024-09-14T04:27:32.724978+0000","last_peered":"2024-09-14T04:27:32.724978+0000","last_clean":"2024-09-14T04:27:32.724978+0000","last_became_active":"2024-09-14T04:25:53.296039+0000","last_became_peered":"2024-09-14T04:25:53.296039+0000","last_unstale":"2024-09-14T04:27:32.724978+0000","last_undegraded":"2024-09-14T04:27:32.724978+0000","last_fullsized":"2024-09-14T04:27:32.724978+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:26:31.287496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":268,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.685430+0000","last_change":"2024-09-14T04:24:47.616564+0000","last_active":"2024-09-14T04:27:24.685430+0000","last_peered":"2024-09-14T04:27:24.685430+0000","last_clean":"2024-09-14T04:27:24.685430+0000","last_became_active":"2024-09-14T04:24:47.616196+0000","last_became_peered":"2024-09-14T04:24:47.616196+0000","last_unstale":"2024-09-14T04:27:24.685430+0000","last_undegraded":"2024-09-14T04:27:24.685430+0000","last_fullsized":"2024-09-14T04:27:24.685430+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:38:56.531650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"390'1791","reported_seq":1812,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.689005+0000","last_change":"2024-09-14T04:27:15.713653+0000","last_active":"2024-09-14T04:27:39.689005+0000","last_peered":"2024-09-14T04:27:39.689005+0000","last_clean":"2024-09-14T04:27:39.689005+0000","last_became_active":"2024-09-14T04:27:15.713312+0000","last_became_peered":"2024-09-14T04:27:15.713312+0000","last_unstale":"2024-09-14T04:27:39.689005+0000","last_undegraded":"2024-09-14T04:27:39.689005+0000","last_fullsized":"2024-09-14T04:27:39.689005+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1791,"log_dups_size":0,"ondisk_log_size":1791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:16:22.235924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117374976,"num_objects":1791,"num_object_clones":0,"num_object_copies":5373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1791,"num_write_kb":114624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'100","reported_seq":26,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:35.272907+0000","last_change":"2024-09-14T04:27:35.272869+0000","last_active":"2024-09-14T04:27:35.272907+0000","last_peered":"2024-09-14T04:27:35.272907+0000","last_clean":"2024-09-14T04:27:35.272907+0000","last_became_active":"2024-09-14T04:27:12.688026+0000","last_became_peered":"2024-09-14T04:27:12.688026+0000","last_unstale":"2024-09-14T04:27:35.272907+0000","last_undegraded":"2024-09-14T04:27:35.272907+0000","last_fullsized":"2024-09-14T04:27:35.272907+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:35.272785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:35.272785+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:00:40.802976+0000","scrub_duration":0.0029595770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":423,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.723804+0000","last_change":"2024-09-14T04:25:03.848027+0000","last_active":"2024-09-14T04:27:32.723804+0000","last_peered":"2024-09-14T04:27:32.723804+0000","last_clean":"2024-09-14T04:27:32.723804+0000","last_became_active":"2024-09-14T04:25:03.847621+0000","last_became_peered":"2024-09-14T04:25:03.847621+0000","last_unstale":"2024-09-14T04:27:32.723804+0000","last_undegraded":"2024-09-14T04:27:32.723804+0000","last_fullsized":"2024-09-14T04:27:32.723804+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:52.585026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":458,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724556+0000","last_change":"2024-09-14T04:25:03.856805+0000","last_active":"2024-09-14T04:27:32.724556+0000","last_peered":"2024-09-14T04:27:32.724556+0000","last_clean":"2024-09-14T04:27:32.724556+0000","last_became_active":"2024-09-14T04:25:03.856459+0000","last_became_peered":"2024-09-14T04:25:03.856459+0000","last_unstale":"2024-09-14T04:27:32.724556+0000","last_undegraded":"2024-09-14T04:27:32.724556+0000","last_fullsized":"2024-09-14T04:27:32.724556+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:17:41.568556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.f","version":"390'1815","reported_seq":1836,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.690245+0000","last_change":"2024-09-14T04:27:15.713172+0000","last_active":"2024-09-14T04:27:39.690245+0000","last_peered":"2024-09-14T04:27:39.690245+0000","last_clean":"2024-09-14T04:27:39.690245+0000","last_became_active":"2024-09-14T04:27:15.712862+0000","last_became_peered":"2024-09-14T04:27:15.712862+0000","last_unstale":"2024-09-14T04:27:39.690245+0000","last_undegraded":"2024-09-14T04:27:39.690245+0000","last_fullsized":"2024-09-14T04:27:39.690245+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1815,"log_dups_size":0,"ondisk_log_size":1815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:11:50.059459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118947840,"num_objects":1815,"num_object_clones":0,"num_object_copies":5445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1815,"num_write_kb":116160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":263,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:34.456988+0000","last_change":"2024-09-14T04:27:34.456942+0000","last_active":"2024-09-14T04:27:34.456988+0000","last_peered":"2024-09-14T04:27:34.456988+0000","last_clean":"2024-09-14T04:27:34.456988+0000","last_became_active":"2024-09-14T04:26:15.726724+0000","last_became_peered":"2024-09-14T04:26:15.726724+0000","last_unstale":"2024-09-14T04:27:34.456988+0000","last_undegraded":"2024-09-14T04:27:34.456988+0000","last_fullsized":"2024-09-14T04:27:34.456988+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:34.456083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:34.456083+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:01:39.830454+0000","scrub_duration":0.0045693399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":115,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682632+0000","last_change":"2024-09-14T04:25:52.311202+0000","last_active":"2024-09-14T04:27:24.682632+0000","last_peered":"2024-09-14T04:27:24.682632+0000","last_clean":"2024-09-14T04:27:24.682632+0000","last_became_active":"2024-09-14T04:25:42.253580+0000","last_became_peered":"2024-09-14T04:25:42.253580+0000","last_unstale":"2024-09-14T04:27:24.682632+0000","last_undegraded":"2024-09-14T04:27:24.682632+0000","last_fullsized":"2024-09-14T04:27:24.682632+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:59:26.500812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":423,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.727092+0000","last_change":"2024-09-14T04:22:08.472386+0000","last_active":"2024-09-14T04:27:32.727092+0000","last_peered":"2024-09-14T04:27:32.727092+0000","last_clean":"2024-09-14T04:27:32.727092+0000","last_became_active":"2024-09-14T04:22:08.472100+0000","last_became_peered":"2024-09-14T04:22:08.472100+0000","last_unstale":"2024-09-14T04:27:32.727092+0000","last_undegraded":"2024-09-14T04:27:32.727092+0000","last_fullsized":"2024-09-14T04:27:32.727092+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:36:03.114984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":378,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.685570+0000","last_change":"2024-09-14T04:26:15.727747+0000","last_active":"2024-09-14T04:27:24.685570+0000","last_peered":"2024-09-14T04:27:24.685570+0000","last_clean":"2024-09-14T04:27:24.685570+0000","last_became_active":"2024-09-14T04:26:15.726210+0000","last_became_peered":"2024-09-14T04:26:15.726210+0000","last_unstale":"2024-09-14T04:27:24.685570+0000","last_undegraded":"2024-09-14T04:27:24.685570+0000","last_fullsized":"2024-09-14T04:27:24.685570+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:01.720621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.c","version":"390'1762","reported_seq":1783,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.611931+0000","last_change":"2024-09-14T04:27:15.710653+0000","last_active":"2024-09-14T04:27:39.611931+0000","last_peered":"2024-09-14T04:27:39.611931+0000","last_clean":"2024-09-14T04:27:39.611931+0000","last_became_active":"2024-09-14T04:27:15.710386+0000","last_became_peered":"2024-09-14T04:27:15.710386+0000","last_unstale":"2024-09-14T04:27:39.611931+0000","last_undegraded":"2024-09-14T04:27:39.611931+0000","last_fullsized":"2024-09-14T04:27:39.611931+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1762,"log_dups_size":0,"ondisk_log_size":1762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:49:21.884028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115474432,"num_objects":1762,"num_object_clones":0,"num_object_copies":5286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1762,"num_write_kb":112768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"76'100","reported_seq":272,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:36.436358+0000","last_change":"2024-09-14T04:27:36.436313+0000","last_active":"2024-09-14T04:27:36.436358+0000","last_peered":"2024-09-14T04:27:36.436358+0000","last_clean":"2024-09-14T04:27:36.436358+0000","last_became_active":"2024-09-14T04:27:12.685740+0000","last_became_peered":"2024-09-14T04:27:12.685740+0000","last_unstale":"2024-09-14T04:27:36.436358+0000","last_undegraded":"2024-09-14T04:27:36.436358+0000","last_fullsized":"2024-09-14T04:27:36.436358+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:36.436230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:36.436230+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:26:41.485866+0000","scrub_duration":0.0026608669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":111,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.723552+0000","last_change":"2024-09-14T04:26:15.727773+0000","last_active":"2024-09-14T04:27:32.723552+0000","last_peered":"2024-09-14T04:27:32.723552+0000","last_clean":"2024-09-14T04:27:32.723552+0000","last_became_active":"2024-09-14T04:26:15.727477+0000","last_became_peered":"2024-09-14T04:26:15.727477+0000","last_unstale":"2024-09-14T04:27:32.723552+0000","last_undegraded":"2024-09-14T04:27:32.723552+0000","last_fullsized":"2024-09-14T04:27:32.723552+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:09:08.222067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":477,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724596+0000","last_change":"2024-09-14T04:26:15.724485+0000","last_active":"2024-09-14T04:27:32.724596+0000","last_peered":"2024-09-14T04:27:32.724596+0000","last_clean":"2024-09-14T04:27:32.724596+0000","last_became_active":"2024-09-14T04:26:15.723989+0000","last_became_peered":"2024-09-14T04:26:15.723989+0000","last_unstale":"2024-09-14T04:27:32.724596+0000","last_undegraded":"2024-09-14T04:27:32.724596+0000","last_fullsized":"2024-09-14T04:27:32.724596+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:27:04.257761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":77,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724931+0000","last_change":"2024-09-14T04:26:15.753288+0000","last_active":"2024-09-14T04:27:32.724931+0000","last_peered":"2024-09-14T04:27:32.724931+0000","last_clean":"2024-09-14T04:27:32.724931+0000","last_became_active":"2024-09-14T04:26:15.753003+0000","last_became_peered":"2024-09-14T04:26:15.753003+0000","last_unstale":"2024-09-14T04:27:32.724931+0000","last_undegraded":"2024-09-14T04:27:32.724931+0000","last_fullsized":"2024-09-14T04:27:32.724931+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:05:37.332422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.d","version":"390'1530","reported_seq":1543,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:38.899945+0000","last_change":"2024-09-14T04:27:15.716650+0000","last_active":"2024-09-14T04:27:38.899945+0000","last_peered":"2024-09-14T04:27:38.899945+0000","last_clean":"2024-09-14T04:27:38.899945+0000","last_became_active":"2024-09-14T04:27:15.716380+0000","last_became_peered":"2024-09-14T04:27:15.716380+0000","last_unstale":"2024-09-14T04:27:38.899945+0000","last_undegraded":"2024-09-14T04:27:38.899945+0000","last_fullsized":"2024-09-14T04:27:38.899945+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:39:59.769243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100270080,"num_objects":1530,"num_object_clones":0,"num_object_copies":4590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1530,"num_write_kb":97920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"76'100","reported_seq":232,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.723214+0000","last_change":"2024-09-14T04:27:14.622410+0000","last_active":"2024-09-14T04:27:32.723214+0000","last_peered":"2024-09-14T04:27:32.723214+0000","last_clean":"2024-09-14T04:27:32.723214+0000","last_became_active":"2024-09-14T04:27:12.689439+0000","last_became_peered":"2024-09-14T04:27:12.689439+0000","last_unstale":"2024-09-14T04:27:32.723214+0000","last_undegraded":"2024-09-14T04:27:32.723214+0000","last_fullsized":"2024-09-14T04:27:32.723214+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-14T04:27:44.604715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.725180+0000","last_change":"2024-09-14T04:26:15.723403+0000","last_active":"2024-09-14T04:27:32.725180+0000","last_peered":"2024-09-14T04:27:32.725180+0000","last_clean":"2024-09-14T04:27:32.725180+0000","last_became_active":"2024-09-14T04:26:15.723058+0000","last_became_peered":"2024-09-14T04:26:15.723058+0000","last_unstale":"2024-09-14T04:27:32.725180+0000","last_undegraded":"2024-09-14T04:27:32.725180+0000","last_fullsized":"2024-09-14T04:27:32.725180+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:53:41.162301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":76,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.726173+0000","last_change":"2024-09-14T04:26:15.727903+0000","last_active":"2024-09-14T04:27:32.726173+0000","last_peered":"2024-09-14T04:27:32.726173+0000","last_clean":"2024-09-14T04:27:32.726173+0000","last_became_active":"2024-09-14T04:26:15.727050+0000","last_became_peered":"2024-09-14T04:26:15.727050+0000","last_unstale":"2024-09-14T04:27:32.726173+0000","last_undegraded":"2024-09-14T04:27:32.726173+0000","last_fullsized":"2024-09-14T04:27:32.726173+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:27:59.930610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":349,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.683271+0000","last_change":"2024-09-14T04:26:15.737341+0000","last_active":"2024-09-14T04:27:24.683271+0000","last_peered":"2024-09-14T04:27:24.683271+0000","last_clean":"2024-09-14T04:27:24.683271+0000","last_became_active":"2024-09-14T04:26:15.736045+0000","last_became_peered":"2024-09-14T04:26:15.736045+0000","last_unstale":"2024-09-14T04:27:24.683271+0000","last_undegraded":"2024-09-14T04:27:24.683271+0000","last_fullsized":"2024-09-14T04:27:24.683271+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:08:08.143667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.e","version":"390'1780","reported_seq":1801,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.596571+0000","last_change":"2024-09-14T04:27:15.712178+0000","last_active":"2024-09-14T04:27:39.596571+0000","last_peered":"2024-09-14T04:27:39.596571+0000","last_clean":"2024-09-14T04:27:39.596571+0000","last_became_active":"2024-09-14T04:27:15.711715+0000","last_became_peered":"2024-09-14T04:27:15.711715+0000","last_unstale":"2024-09-14T04:27:39.596571+0000","last_undegraded":"2024-09-14T04:27:39.596571+0000","last_fullsized":"2024-09-14T04:27:39.596571+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1780,"log_dups_size":0,"ondisk_log_size":1780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:03:53.968174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116654080,"num_objects":1780,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1780,"num_write_kb":113920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"76'100","reported_seq":16,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682050+0000","last_change":"2024-09-14T04:27:12.688592+0000","last_active":"2024-09-14T04:27:24.682050+0000","last_peered":"2024-09-14T04:27:24.682050+0000","last_clean":"2024-09-14T04:27:24.682050+0000","last_became_active":"2024-09-14T04:27:12.687762+0000","last_became_peered":"2024-09-14T04:27:12.687762+0000","last_unstale":"2024-09-14T04:27:24.682050+0000","last_undegraded":"2024-09-14T04:27:24.682050+0000","last_fullsized":"2024-09-14T04:27:24.682050+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":107,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682738+0000","last_change":"2024-09-14T04:27:06.910535+0000","last_active":"2024-09-14T04:27:24.682738+0000","last_peered":"2024-09-14T04:27:24.682738+0000","last_clean":"2024-09-14T04:27:24.682738+0000","last_became_active":"2024-09-14T04:26:15.729031+0000","last_became_peered":"2024-09-14T04:26:15.729031+0000","last_unstale":"2024-09-14T04:27:24.682738+0000","last_undegraded":"2024-09-14T04:27:24.682738+0000","last_fullsized":"2024-09-14T04:27:24.682738+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:12.734036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":299,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.685058+0000","last_change":"2024-09-14T04:26:15.729742+0000","last_active":"2024-09-14T04:27:24.685058+0000","last_peered":"2024-09-14T04:27:24.685058+0000","last_clean":"2024-09-14T04:27:24.685058+0000","last_became_active":"2024-09-14T04:26:15.728375+0000","last_became_peered":"2024-09-14T04:26:15.728375+0000","last_unstale":"2024-09-14T04:27:24.685058+0000","last_undegraded":"2024-09-14T04:27:24.685058+0000","last_fullsized":"2024-09-14T04:27:24.685058+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:29:31.159401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":70,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.725343+0000","last_change":"2024-09-14T04:25:42.246310+0000","last_active":"2024-09-14T04:27:32.725343+0000","last_peered":"2024-09-14T04:27:32.725343+0000","last_clean":"2024-09-14T04:27:32.725343+0000","last_became_active":"2024-09-14T04:25:42.245997+0000","last_became_peered":"2024-09-14T04:25:42.245997+0000","last_unstale":"2024-09-14T04:27:32.725343+0000","last_undegraded":"2024-09-14T04:27:32.725343+0000","last_fullsized":"2024-09-14T04:27:32.725343+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:01:43.109388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.8","version":"390'1743","reported_seq":1764,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.622336+0000","last_change":"2024-09-14T04:27:15.713610+0000","last_active":"2024-09-14T04:27:39.622336+0000","last_peered":"2024-09-14T04:27:39.622336+0000","last_clean":"2024-09-14T04:27:39.622336+0000","last_became_active":"2024-09-14T04:27:15.713189+0000","last_became_peered":"2024-09-14T04:27:15.713189+0000","last_unstale":"2024-09-14T04:27:39.622336+0000","last_undegraded":"2024-09-14T04:27:39.622336+0000","last_fullsized":"2024-09-14T04:27:39.622336+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1743,"log_dups_size":0,"ondisk_log_size":1743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:10:39.862915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114229248,"num_objects":1743,"num_object_clones":0,"num_object_copies":5229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1743,"num_write_kb":111552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"76'100","reported_seq":272,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:38.451534+0000","last_change":"2024-09-14T04:27:38.451495+0000","last_active":"2024-09-14T04:27:38.451534+0000","last_peered":"2024-09-14T04:27:38.451534+0000","last_clean":"2024-09-14T04:27:38.451534+0000","last_became_active":"2024-09-14T04:27:12.687176+0000","last_became_peered":"2024-09-14T04:27:12.687176+0000","last_unstale":"2024-09-14T04:27:38.451534+0000","last_undegraded":"2024-09-14T04:27:38.451534+0000","last_fullsized":"2024-09-14T04:27:38.451534+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:38.451408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:38.451408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:08.295464+0000","scrub_duration":0.0034959740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":95,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682902+0000","last_change":"2024-09-14T04:25:42.253453+0000","last_active":"2024-09-14T04:27:24.682902+0000","last_peered":"2024-09-14T04:27:24.682902+0000","last_clean":"2024-09-14T04:27:24.682902+0000","last_became_active":"2024-09-14T04:25:42.252529+0000","last_became_peered":"2024-09-14T04:25:42.252529+0000","last_unstale":"2024-09-14T04:27:24.682902+0000","last_undegraded":"2024-09-14T04:27:24.682902+0000","last_fullsized":"2024-09-14T04:27:24.682902+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:19.539802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.725430+0000","last_change":"2024-09-14T04:26:15.723378+0000","last_active":"2024-09-14T04:27:32.725430+0000","last_peered":"2024-09-14T04:27:32.725430+0000","last_clean":"2024-09-14T04:27:32.725430+0000","last_became_active":"2024-09-14T04:26:15.723070+0000","last_became_peered":"2024-09-14T04:26:15.723070+0000","last_unstale":"2024-09-14T04:27:32.725430+0000","last_undegraded":"2024-09-14T04:27:32.725430+0000","last_fullsized":"2024-09-14T04:27:32.725430+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:38:14.395886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":572,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682557+0000","last_change":"2024-09-14T04:26:15.723597+0000","last_active":"2024-09-14T04:27:24.682557+0000","last_peered":"2024-09-14T04:27:24.682557+0000","last_clean":"2024-09-14T04:27:24.682557+0000","last_became_active":"2024-09-14T04:26:15.723289+0000","last_became_peered":"2024-09-14T04:26:15.723289+0000","last_unstale":"2024-09-14T04:27:24.682557+0000","last_undegraded":"2024-09-14T04:27:24.682557+0000","last_fullsized":"2024-09-14T04:27:24.682557+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:55:41.253830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"390'1711","reported_seq":1724,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.234641+0000","last_change":"2024-09-14T04:27:15.712988+0000","last_active":"2024-09-14T04:27:39.234641+0000","last_peered":"2024-09-14T04:27:39.234641+0000","last_clean":"2024-09-14T04:27:39.234641+0000","last_became_active":"2024-09-14T04:27:15.712711+0000","last_became_peered":"2024-09-14T04:27:15.712711+0000","last_unstale":"2024-09-14T04:27:39.234641+0000","last_undegraded":"2024-09-14T04:27:39.234641+0000","last_fullsized":"2024-09-14T04:27:39.234641+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1711,"log_dups_size":0,"ondisk_log_size":1711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:22:26.902016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112132096,"num_objects":1711,"num_object_clones":0,"num_object_copies":5133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1711,"num_write_kb":109504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"76'100","reported_seq":262,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.685330+0000","last_change":"2024-09-14T04:27:12.686759+0000","last_active":"2024-09-14T04:27:24.685330+0000","last_peered":"2024-09-14T04:27:24.685330+0000","last_clean":"2024-09-14T04:27:24.685330+0000","last_became_active":"2024-09-14T04:27:12.686451+0000","last_became_peered":"2024-09-14T04:27:12.686451+0000","last_unstale":"2024-09-14T04:27:24.685330+0000","last_undegraded":"2024-09-14T04:27:24.685330+0000","last_fullsized":"2024-09-14T04:27:24.685330+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":77,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.725015+0000","last_change":"2024-09-14T04:26:15.734625+0000","last_active":"2024-09-14T04:27:32.725015+0000","last_peered":"2024-09-14T04:27:32.725015+0000","last_clean":"2024-09-14T04:27:32.725015+0000","last_became_active":"2024-09-14T04:26:15.734309+0000","last_became_peered":"2024-09-14T04:26:15.734309+0000","last_unstale":"2024-09-14T04:27:32.725015+0000","last_undegraded":"2024-09-14T04:27:32.725015+0000","last_fullsized":"2024-09-14T04:27:32.725015+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:58:57.797342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":58,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.685097+0000","last_change":"2024-09-14T04:26:15.727947+0000","last_active":"2024-09-14T04:27:24.685097+0000","last_peered":"2024-09-14T04:27:24.685097+0000","last_clean":"2024-09-14T04:27:24.685097+0000","last_became_active":"2024-09-14T04:26:15.726608+0000","last_became_peered":"2024-09-14T04:26:15.726608+0000","last_unstale":"2024-09-14T04:27:24.685097+0000","last_undegraded":"2024-09-14T04:27:24.685097+0000","last_fullsized":"2024-09-14T04:27:24.685097+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:33:03.772215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":632,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682634+0000","last_change":"2024-09-14T04:26:15.732599+0000","last_active":"2024-09-14T04:27:24.682634+0000","last_peered":"2024-09-14T04:27:24.682634+0000","last_clean":"2024-09-14T04:27:24.682634+0000","last_became_active":"2024-09-14T04:26:15.725658+0000","last_became_peered":"2024-09-14T04:26:15.725658+0000","last_unstale":"2024-09-14T04:27:24.682634+0000","last_undegraded":"2024-09-14T04:27:24.682634+0000","last_fullsized":"2024-09-14T04:27:24.682634+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:48:20.963473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.a","version":"390'1650","reported_seq":1663,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.212628+0000","last_change":"2024-09-14T04:27:15.713659+0000","last_active":"2024-09-14T04:27:39.212628+0000","last_peered":"2024-09-14T04:27:39.212628+0000","last_clean":"2024-09-14T04:27:39.212628+0000","last_became_active":"2024-09-14T04:27:15.713394+0000","last_became_peered":"2024-09-14T04:27:15.713394+0000","last_unstale":"2024-09-14T04:27:39.212628+0000","last_undegraded":"2024-09-14T04:27:39.212628+0000","last_fullsized":"2024-09-14T04:27:39.212628+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1650,"log_dups_size":0,"ondisk_log_size":1650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:14:07.900514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108134400,"num_objects":1650,"num_object_clones":0,"num_object_copies":4950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1650,"num_write_kb":105600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'100","reported_seq":232,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724622+0000","last_change":"2024-09-14T04:27:14.622439+0000","last_active":"2024-09-14T04:27:32.724622+0000","last_peered":"2024-09-14T04:27:32.724622+0000","last_clean":"2024-09-14T04:27:32.724622+0000","last_became_active":"2024-09-14T04:27:12.687977+0000","last_became_peered":"2024-09-14T04:27:12.687977+0000","last_unstale":"2024-09-14T04:27:32.724622+0000","last_undegraded":"2024-09-14T04:27:32.724622+0000","last_fullsized":"2024-09-14T04:27:32.724622+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-14T04:27:43.602778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":646,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724648+0000","last_change":"2024-09-14T04:26:15.748334+0000","last_active":"2024-09-14T04:27:32.724648+0000","last_peered":"2024-09-14T04:27:32.724648+0000","last_clean":"2024-09-14T04:27:32.724648+0000","last_became_active":"2024-09-14T04:26:15.726174+0000","last_became_peered":"2024-09-14T04:26:15.726174+0000","last_unstale":"2024-09-14T04:27:32.724648+0000","last_undegraded":"2024-09-14T04:27:32.724648+0000","last_fullsized":"2024-09-14T04:27:32.724648+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:25:47.468961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":408,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.685564+0000","last_change":"2024-09-14T04:24:47.618083+0000","last_active":"2024-09-14T04:27:24.685564+0000","last_peered":"2024-09-14T04:27:24.685564+0000","last_clean":"2024-09-14T04:27:24.685564+0000","last_became_active":"2024-09-14T04:24:47.617759+0000","last_became_peered":"2024-09-14T04:24:47.617759+0000","last_unstale":"2024-09-14T04:27:24.685564+0000","last_undegraded":"2024-09-14T04:27:24.685564+0000","last_fullsized":"2024-09-14T04:27:24.685564+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:08:35.614869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":175,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.723643+0000","last_change":"2024-09-14T04:26:15.726983+0000","last_active":"2024-09-14T04:27:32.723643+0000","last_peered":"2024-09-14T04:27:32.723643+0000","last_clean":"2024-09-14T04:27:32.723643+0000","last_became_active":"2024-09-14T04:26:15.726003+0000","last_became_peered":"2024-09-14T04:26:15.726003+0000","last_unstale":"2024-09-14T04:27:32.723643+0000","last_undegraded":"2024-09-14T04:27:32.723643+0000","last_fullsized":"2024-09-14T04:27:32.723643+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:18:15.328679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.b","version":"390'1743","reported_seq":1764,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:39.614527+0000","last_change":"2024-09-14T04:27:15.710534+0000","last_active":"2024-09-14T04:27:39.614527+0000","last_peered":"2024-09-14T04:27:39.614527+0000","last_clean":"2024-09-14T04:27:39.614527+0000","last_became_active":"2024-09-14T04:27:15.710259+0000","last_became_peered":"2024-09-14T04:27:15.710259+0000","last_unstale":"2024-09-14T04:27:39.614527+0000","last_undegraded":"2024-09-14T04:27:39.614527+0000","last_fullsized":"2024-09-14T04:27:39.614527+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":1743,"log_dups_size":0,"ondisk_log_size":1743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:07:54.372523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114229248,"num_objects":1743,"num_object_clones":0,"num_object_copies":5229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1743,"num_write_kb":111552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"76'100","reported_seq":32,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.723730+0000","last_change":"2024-09-14T04:27:14.686147+0000","last_active":"2024-09-14T04:27:32.723730+0000","last_peered":"2024-09-14T04:27:32.723730+0000","last_clean":"2024-09-14T04:27:32.723730+0000","last_became_active":"2024-09-14T04:27:12.689883+0000","last_became_peered":"2024-09-14T04:27:12.689883+0000","last_unstale":"2024-09-14T04:27:32.723730+0000","last_undegraded":"2024-09-14T04:27:32.723730+0000","last_fullsized":"2024-09-14T04:27:32.723730+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T03:39:29.365581+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-14T04:27:44.325192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":451,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682742+0000","last_change":"2024-09-14T04:26:15.760792+0000","last_active":"2024-09-14T04:27:24.682742+0000","last_peered":"2024-09-14T04:27:24.682742+0000","last_clean":"2024-09-14T04:27:24.682742+0000","last_became_active":"2024-09-14T04:26:15.745969+0000","last_became_peered":"2024-09-14T04:26:15.745969+0000","last_unstale":"2024-09-14T04:27:24.682742+0000","last_undegraded":"2024-09-14T04:27:24.682742+0000","last_fullsized":"2024-09-14T04:27:24.682742+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:41:56.094887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":408,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.685599+0000","last_change":"2024-09-14T04:25:42.332231+0000","last_active":"2024-09-14T04:27:24.685599+0000","last_peered":"2024-09-14T04:27:24.685599+0000","last_clean":"2024-09-14T04:27:24.685599+0000","last_became_active":"2024-09-14T04:25:42.331514+0000","last_became_peered":"2024-09-14T04:25:42.331514+0000","last_unstale":"2024-09-14T04:27:24.685599+0000","last_undegraded":"2024-09-14T04:27:24.685599+0000","last_fullsized":"2024-09-14T04:27:24.685599+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:33:37.729778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":75,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682349+0000","last_change":"2024-09-14T04:22:08.465532+0000","last_active":"2024-09-14T04:27:24.682349+0000","last_peered":"2024-09-14T04:27:24.682349+0000","last_clean":"2024-09-14T04:27:24.682349+0000","last_became_active":"2024-09-14T04:22:08.464571+0000","last_became_peered":"2024-09-14T04:22:08.464571+0000","last_unstale":"2024-09-14T04:27:24.682349+0000","last_undegraded":"2024-09-14T04:27:24.682349+0000","last_fullsized":"2024-09-14T04:27:24.682349+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:25:23.109822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":155,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.685038+0000","last_change":"2024-09-14T04:25:53.301718+0000","last_active":"2024-09-14T04:27:24.685038+0000","last_peered":"2024-09-14T04:27:24.685038+0000","last_clean":"2024-09-14T04:27:24.685038+0000","last_became_active":"2024-09-14T04:25:53.301421+0000","last_became_peered":"2024-09-14T04:25:53.301421+0000","last_unstale":"2024-09-14T04:27:24.685038+0000","last_undegraded":"2024-09-14T04:27:24.685038+0000","last_fullsized":"2024-09-14T04:27:24.685038+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:45:47.546722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":112,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.685314+0000","last_change":"2024-09-14T04:25:42.375734+0000","last_active":"2024-09-14T04:27:24.685314+0000","last_peered":"2024-09-14T04:27:24.685314+0000","last_clean":"2024-09-14T04:27:24.685314+0000","last_became_active":"2024-09-14T04:25:42.259500+0000","last_became_peered":"2024-09-14T04:25:42.259500+0000","last_unstale":"2024-09-14T04:27:24.685314+0000","last_undegraded":"2024-09-14T04:27:24.685314+0000","last_fullsized":"2024-09-14T04:27:24.685314+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:27:37.236272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":585,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.684864+0000","last_change":"2024-09-14T04:26:15.728258+0000","last_active":"2024-09-14T04:27:24.684864+0000","last_peered":"2024-09-14T04:27:24.684864+0000","last_clean":"2024-09-14T04:27:24.684864+0000","last_became_active":"2024-09-14T04:26:15.726521+0000","last_became_peered":"2024-09-14T04:26:15.726521+0000","last_unstale":"2024-09-14T04:27:24.684864+0000","last_undegraded":"2024-09-14T04:27:24.684864+0000","last_fullsized":"2024-09-14T04:27:24.684864+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:22:20.663544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":310,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.728530+0000","last_change":"2024-09-14T04:26:15.758946+0000","last_active":"2024-09-14T04:27:32.728530+0000","last_peered":"2024-09-14T04:27:32.728530+0000","last_clean":"2024-09-14T04:27:32.728530+0000","last_became_active":"2024-09-14T04:26:15.725634+0000","last_became_peered":"2024-09-14T04:26:15.725634+0000","last_unstale":"2024-09-14T04:27:32.728530+0000","last_undegraded":"2024-09-14T04:27:32.728530+0000","last_fullsized":"2024-09-14T04:27:32.728530+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:21:59.189929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":105,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682966+0000","last_change":"2024-09-14T04:25:42.256482+0000","last_active":"2024-09-14T04:27:24.682966+0000","last_peered":"2024-09-14T04:27:24.682966+0000","last_clean":"2024-09-14T04:27:24.682966+0000","last_became_active":"2024-09-14T04:25:42.256182+0000","last_became_peered":"2024-09-14T04:25:42.256182+0000","last_unstale":"2024-09-14T04:27:24.682966+0000","last_undegraded":"2024-09-14T04:27:24.682966+0000","last_fullsized":"2024-09-14T04:27:24.682966+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:04:58.123110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":590,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.726540+0000","last_change":"2024-09-14T04:25:03.864958+0000","last_active":"2024-09-14T04:27:32.726540+0000","last_peered":"2024-09-14T04:27:32.726540+0000","last_clean":"2024-09-14T04:27:32.726540+0000","last_became_active":"2024-09-14T04:25:03.864529+0000","last_became_peered":"2024-09-14T04:25:03.864529+0000","last_unstale":"2024-09-14T04:27:32.726540+0000","last_undegraded":"2024-09-14T04:27:32.726540+0000","last_fullsized":"2024-09-14T04:27:32.726540+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:12:25.237701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":635,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.683191+0000","last_change":"2024-09-14T04:26:15.736843+0000","last_active":"2024-09-14T04:27:24.683191+0000","last_peered":"2024-09-14T04:27:24.683191+0000","last_clean":"2024-09-14T04:27:24.683191+0000","last_became_active":"2024-09-14T04:26:15.734655+0000","last_became_peered":"2024-09-14T04:26:15.734655+0000","last_unstale":"2024-09-14T04:27:24.683191+0000","last_undegraded":"2024-09-14T04:27:24.683191+0000","last_fullsized":"2024-09-14T04:27:24.683191+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:47:05.810191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":602,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.723894+0000","last_change":"2024-09-14T04:24:47.613484+0000","last_active":"2024-09-14T04:27:32.723894+0000","last_peered":"2024-09-14T04:27:32.723894+0000","last_clean":"2024-09-14T04:27:32.723894+0000","last_became_active":"2024-09-14T04:24:47.613215+0000","last_became_peered":"2024-09-14T04:24:47.613215+0000","last_unstale":"2024-09-14T04:27:32.723894+0000","last_undegraded":"2024-09-14T04:27:32.723894+0000","last_fullsized":"2024-09-14T04:27:32.723894+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:12:35.605770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":415,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.685548+0000","last_change":"2024-09-14T04:26:15.731440+0000","last_active":"2024-09-14T04:27:24.685548+0000","last_peered":"2024-09-14T04:27:24.685548+0000","last_clean":"2024-09-14T04:27:24.685548+0000","last_became_active":"2024-09-14T04:26:15.726850+0000","last_became_peered":"2024-09-14T04:26:15.726850+0000","last_unstale":"2024-09-14T04:27:24.685548+0000","last_undegraded":"2024-09-14T04:27:24.685548+0000","last_fullsized":"2024-09-14T04:27:24.685548+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:00:48.044324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":436,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.723722+0000","last_change":"2024-09-14T04:26:15.727272+0000","last_active":"2024-09-14T04:27:32.723722+0000","last_peered":"2024-09-14T04:27:32.723722+0000","last_clean":"2024-09-14T04:27:32.723722+0000","last_became_active":"2024-09-14T04:26:15.726200+0000","last_became_peered":"2024-09-14T04:26:15.726200+0000","last_unstale":"2024-09-14T04:27:32.723722+0000","last_undegraded":"2024-09-14T04:27:32.723722+0000","last_fullsized":"2024-09-14T04:27:32.723722+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:38:30.447563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":432,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724827+0000","last_change":"2024-09-14T04:26:15.745872+0000","last_active":"2024-09-14T04:27:32.724827+0000","last_peered":"2024-09-14T04:27:32.724827+0000","last_clean":"2024-09-14T04:27:32.724827+0000","last_became_active":"2024-09-14T04:26:15.745533+0000","last_became_peered":"2024-09-14T04:26:15.745533+0000","last_unstale":"2024-09-14T04:27:32.724827+0000","last_undegraded":"2024-09-14T04:27:32.724827+0000","last_fullsized":"2024-09-14T04:27:32.724827+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:28:40.179766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":451,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.727143+0000","last_change":"2024-09-14T04:24:47.612687+0000","last_active":"2024-09-14T04:27:32.727143+0000","last_peered":"2024-09-14T04:27:32.727143+0000","last_clean":"2024-09-14T04:27:32.727143+0000","last_became_active":"2024-09-14T04:24:47.612392+0000","last_became_peered":"2024-09-14T04:24:47.612392+0000","last_unstale":"2024-09-14T04:27:32.727143+0000","last_undegraded":"2024-09-14T04:27:32.727143+0000","last_fullsized":"2024-09-14T04:27:32.727143+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:52:51.954808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":421,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.728477+0000","last_change":"2024-09-14T04:24:47.613822+0000","last_active":"2024-09-14T04:27:32.728477+0000","last_peered":"2024-09-14T04:27:32.728477+0000","last_clean":"2024-09-14T04:27:32.728477+0000","last_became_active":"2024-09-14T04:24:47.613315+0000","last_became_peered":"2024-09-14T04:24:47.613315+0000","last_unstale":"2024-09-14T04:27:32.728477+0000","last_undegraded":"2024-09-14T04:27:32.728477+0000","last_fullsized":"2024-09-14T04:27:32.728477+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:22.682884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":220,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.723324+0000","last_change":"2024-09-14T04:26:15.727713+0000","last_active":"2024-09-14T04:27:32.723324+0000","last_peered":"2024-09-14T04:27:32.723324+0000","last_clean":"2024-09-14T04:27:32.723324+0000","last_became_active":"2024-09-14T04:26:15.727020+0000","last_became_peered":"2024-09-14T04:26:15.727020+0000","last_unstale":"2024-09-14T04:27:32.723324+0000","last_undegraded":"2024-09-14T04:27:32.723324+0000","last_fullsized":"2024-09-14T04:27:32.723324+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:35:46.915353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":77,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.725164+0000","last_change":"2024-09-14T04:26:15.721332+0000","last_active":"2024-09-14T04:27:32.725164+0000","last_peered":"2024-09-14T04:27:32.725164+0000","last_clean":"2024-09-14T04:27:32.725164+0000","last_became_active":"2024-09-14T04:26:15.721017+0000","last_became_peered":"2024-09-14T04:26:15.721017+0000","last_unstale":"2024-09-14T04:27:32.725164+0000","last_undegraded":"2024-09-14T04:27:32.725164+0000","last_fullsized":"2024-09-14T04:27:32.725164+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:44:50.434373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":96,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.726552+0000","last_change":"2024-09-14T04:26:15.731906+0000","last_active":"2024-09-14T04:27:32.726552+0000","last_peered":"2024-09-14T04:27:32.726552+0000","last_clean":"2024-09-14T04:27:32.726552+0000","last_became_active":"2024-09-14T04:26:15.730836+0000","last_became_peered":"2024-09-14T04:26:15.730836+0000","last_unstale":"2024-09-14T04:27:32.726552+0000","last_undegraded":"2024-09-14T04:27:32.726552+0000","last_fullsized":"2024-09-14T04:27:32.726552+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:03:41.330132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":519,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.726745+0000","last_change":"2024-09-14T04:25:42.236692+0000","last_active":"2024-09-14T04:27:32.726745+0000","last_peered":"2024-09-14T04:27:32.726745+0000","last_clean":"2024-09-14T04:27:32.726745+0000","last_became_active":"2024-09-14T04:25:42.236449+0000","last_became_peered":"2024-09-14T04:25:42.236449+0000","last_unstale":"2024-09-14T04:27:32.726745+0000","last_undegraded":"2024-09-14T04:27:32.726745+0000","last_fullsized":"2024-09-14T04:27:32.726745+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:19:34.323210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":93,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.682833+0000","last_change":"2024-09-14T04:24:47.706532+0000","last_active":"2024-09-14T04:27:24.682833+0000","last_peered":"2024-09-14T04:27:24.682833+0000","last_clean":"2024-09-14T04:27:24.682833+0000","last_became_active":"2024-09-14T04:24:47.628189+0000","last_became_peered":"2024-09-14T04:24:47.628189+0000","last_unstale":"2024-09-14T04:27:24.682833+0000","last_undegraded":"2024-09-14T04:27:24.682833+0000","last_fullsized":"2024-09-14T04:27:24.682833+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:29:46.206803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":505,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.727367+0000","last_change":"2024-09-14T04:26:15.722033+0000","last_active":"2024-09-14T04:27:32.727367+0000","last_peered":"2024-09-14T04:27:32.727367+0000","last_clean":"2024-09-14T04:27:32.727367+0000","last_became_active":"2024-09-14T04:26:15.721545+0000","last_became_peered":"2024-09-14T04:26:15.721545+0000","last_unstale":"2024-09-14T04:27:32.727367+0000","last_undegraded":"2024-09-14T04:27:32.727367+0000","last_fullsized":"2024-09-14T04:27:32.727367+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:48:21.769211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":79,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.683016+0000","last_change":"2024-09-14T04:22:08.472912+0000","last_active":"2024-09-14T04:27:24.683016+0000","last_peered":"2024-09-14T04:27:24.683016+0000","last_clean":"2024-09-14T04:27:24.683016+0000","last_became_active":"2024-09-14T04:22:08.463787+0000","last_became_peered":"2024-09-14T04:22:08.463787+0000","last_unstale":"2024-09-14T04:27:24.683016+0000","last_undegraded":"2024-09-14T04:27:24.683016+0000","last_fullsized":"2024-09-14T04:27:24.683016+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:56:56.756448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":429,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724896+0000","last_change":"2024-09-14T04:26:15.727588+0000","last_active":"2024-09-14T04:27:32.724896+0000","last_peered":"2024-09-14T04:27:32.724896+0000","last_clean":"2024-09-14T04:27:32.724896+0000","last_became_active":"2024-09-14T04:26:15.727153+0000","last_became_peered":"2024-09-14T04:26:15.727153+0000","last_unstale":"2024-09-14T04:27:32.724896+0000","last_undegraded":"2024-09-14T04:27:32.724896+0000","last_fullsized":"2024-09-14T04:27:32.724896+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:15:27.230932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":437,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-14T04:27:32.724916+0000","last_change":"2024-09-14T04:25:42.235102+0000","last_active":"2024-09-14T04:27:32.724916+0000","last_peered":"2024-09-14T04:27:32.724916+0000","last_clean":"2024-09-14T04:27:32.724916+0000","last_became_active":"2024-09-14T04:25:42.234776+0000","last_became_peered":"2024-09-14T04:25:42.234776+0000","last_unstale":"2024-09-14T04:27:32.724916+0000","last_undegraded":"2024-09-14T04:27:32.724916+0000","last_fullsized":"2024-09-14T04:27:32.724916+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:59:52.026651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":204,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.684489+0000","last_change":"2024-09-14T04:25:53.300338+0000","last_active":"2024-09-14T04:27:24.684489+0000","last_peered":"2024-09-14T04:27:24.684489+0000","last_clean":"2024-09-14T04:27:24.684489+0000","last_became_active":"2024-09-14T04:25:53.299919+0000","last_became_peered":"2024-09-14T04:25:53.299919+0000","last_unstale":"2024-09-14T04:27:24.684489+0000","last_undegraded":"2024-09-14T04:27:24.684489+0000","last_fullsized":"2024-09-14T04:27:24.684489+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:17.341402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":664,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-09-14T04:27:24.683102+0000","last_change":"2024-09-14T04:26:15.738521+0000","last_active":"2024-09-14T04:27:24.683102+0000","last_peered":"2024-09-14T04:27:24.683102+0000","last_clean":"2024-09-14T04:27:24.683102+0000","last_became_active":"2024-09-14T04:26:15.736609+0000","last_became_peered":"2024-09-14T04:26:15.736609+0000","last_unstale":"2024-09-14T04:27:24.683102+0000","last_undegraded":"2024-09-14T04:27:24.683102+0000","last_fullsized":"2024-09-14T04:27:24.683102+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:54:47.371271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":6886,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":21458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":14,"num_pg":16,"stat_sum":{"num_bytes":1825112064,"num_objects":27849,"num_object_clones":0,"num_object_copies":83547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27849,"num_write_kb":1782336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5286264832,"data_stored":5286264832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27849,"ondisk_log_size":27849,"up":48,"acting":48,"num_store_stats":5},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":375690,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":187,"num_read_kb":476,"num_write":96,"num_write_kb":1036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":966058,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":0,"seq":206158430767,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626996,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2414688,"kb_used_data":913944,"kb_used_omap":1,"kb_used_meta":1500734,"kb_avail":91330464,"statfs":{"total":95995035648,"available":93522395136,"internally_reserved":0,"allocated":935878656,"data_stored":933977733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1536751960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823224,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2709840,"kb_used_data":1203528,"kb_used_omap":1,"kb_used_meta":1506302,"kb_avail":91035312,"statfs":{"total":95995035648,"available":93220159488,"internally_reserved":0,"allocated":1232412672,"data_stored":1230523516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1542453592},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917306,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137752,"kb_used_data":795280,"kb_used_omap":6,"kb_used_meta":342457,"kb_avail":92607400,"statfs":{"total":95995035648,"available":94829977600,"internally_reserved":0,"allocated":814366720,"data_stored":812473352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":350676991},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552605,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2153560,"kb_used_data":1114128,"kb_used_omap":4,"kb_used_meta":1039419,"kb_avail":91591592,"statfs":{"total":95995035648,"available":93789790208,"internally_reserved":0,"allocated":1140867072,"data_stored":1138969894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":1064365501},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435646,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1943700,"kb_used_data":951820,"kb_used_omap":6,"kb_used_meta":991865,"kb_avail":91801452,"statfs":{"total":95995035648,"available":94004686848,"internally_reserved":0,"allocated":974663680,"data_stored":972774283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":1015670221},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":416611828116,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":343597384105,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1051000832,"data_stored":1051000832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1172504576,"data_stored":1172504576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":858849280,"data_stored":858849280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1230372864,"data_stored":1230372864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":973537280,"data_stored":973537280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:27:47.124 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-14T04:27:47.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_9 pg_num 26 2024-09-14T04:27:47.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:47.170 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:47.211 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:47.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:47.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:47.312 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:47.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:47.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:47.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:47 smithi114 ceph-mon[103029]: 1.8 scrub starts 2024-09-14T04:27:47.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:47 smithi114 ceph-mon[103029]: 1.8 scrub ok 2024-09-14T04:27:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:47 smithi114 ceph-mon[106969]: 1.8 scrub starts 2024-09-14T04:27:47.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:47 smithi114 ceph-mon[106969]: 1.8 scrub ok 2024-09-14T04:27:47.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:47.487 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:47.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:47.542 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 17 16 43586 43570 160.165 125.312 0.00625713 0.00623943 2024-09-14T04:27:47.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:47.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:47.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:47 smithi160 ceph-mon[88407]: 1.8 scrub starts 2024-09-14T04:27:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:47 smithi160 ceph-mon[88407]: 1.8 scrub ok 2024-09-14T04:27:47.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:47.764 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:47.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:47.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:47.902 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:47.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:48.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:48.036 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:48.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:48.137 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:48.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:48.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:48.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:48.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:48.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:48.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:48.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:48.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 18 16 45514 45498 157.961 120.5 0.00902531 0.00632631 2024-09-14T04:27:48.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:48.576 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:48.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:48.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:48 smithi114 ceph-mon[103029]: pgmap v1484: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-09-14T04:27:48.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:48 smithi114 ceph-mon[106969]: pgmap v1484: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-09-14T04:27:48.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:27:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:48 smithi160 ceph-mon[88407]: pgmap v1484: 99 pgs: 99 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-09-14T04:27:48.678 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:48.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:48.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:48.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:48.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:48.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:48.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:48.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:48.994 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:48.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:49.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:49.134 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:49.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:49.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:49.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:49.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:49.373 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:49.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:49.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:49.540 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:49.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:49.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:49.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 19 16 47443 47427 155.989 120.562 0.0146765 0.00640542 2024-09-14T04:27:49.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:49.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:49.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:49.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:49.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:49.819 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:49.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:49.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:49.949 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:49.955 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:49.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:50.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:50.100 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:50.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:50.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:50.245 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:50.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:50.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:50.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:50.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:50.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:50.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:50 smithi114 ceph-mon[103029]: pgmap v1485: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 132 MiB/s wr, 2.10k op/s 2024-09-14T04:27:50.505 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:50 smithi114 ceph-mon[106969]: pgmap v1485: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 132 MiB/s wr, 2.10k op/s 2024-09-14T04:27:50.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:50.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:50.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:27:50.543181+0000 min lat: 0.00118 max lat: 0.139467 avg lat: 0.00649406 2024-09-14T04:27:50.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:27:50.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 20 16 49265 49249 153.882 113.875 0.00507585 0.00649406 2024-09-14T04:27:50.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:27:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:27:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:27:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:50 smithi160 ceph-mon[88407]: pgmap v1485: 99 pgs: 99 active+clean; 2.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 132 MiB/s wr, 2.10k op/s 2024-09-14T04:27:50.667 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:27:50.667 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:27:50.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.668 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:49469 14.8 14:1fde7074:::benchmark_data_smithi114_329146_object49468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-14T04:27:50.668 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.654243+0000", 2024-09-14T04:27:50.668 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0092250809999999996, 2024-09-14T04:27:50.668 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0092552229999999999, 2024-09-14T04:27:50.668 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.668 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.668 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:27:50.668 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:50.668 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:27:50.668 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49469 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.654243+0000", 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.669 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.654243+0000", 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.654245+0000", 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.178e-06 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.654269+0000", 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4646000000000001e-05 2024-09-14T04:27:50.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.654279+0000", 2024-09-14T04:27:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.9899999999999992e-06 2024-09-14T04:27:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.654286+0000", 2024-09-14T04:27:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1509999999999998e-06 2024-09-14T04:27:50.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.654365+0000", 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8831999999999993e-05 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.654501+0000", 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000135993 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.654635+0000", 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000133783 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.659187+0000", 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0045520100000000004 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:27:50.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.659452+0000", 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00026458699999999999 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:49470 14.8 14:1cf72d8e:::benchmark_data_smithi114_329146_object49469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.656414+0000", 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0070547020000000004, 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0071632199999999997, 2024-09-14T04:27:50.674 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49470 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656414+0000", 2024-09-14T04:27:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656414+0000", 2024-09-14T04:27:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656415+0000", 2024-09-14T04:27:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3149999999999999e-06 2024-09-14T04:27:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656437+0000", 2024-09-14T04:27:50.677 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2011e-05 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656447+0000", 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0407999999999999e-05 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656454+0000", 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4679999999999997e-06 2024-09-14T04:27:50.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656900+0000", 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00044641800000000002 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.657077+0000", 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017704400000000001 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.679 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,5", 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.657228+0000", 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000150433 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.659460+0000", 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022320130000000001 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.680 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:27:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662593+0000", 2024-09-14T04:27:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031327989999999999 2024-09-14T04:27:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:49471 14.0 14:0fb3f1d2:::benchmark_data_smithi114_329146_object49470:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-14T04:27:50.681 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.656466+0000", 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0070023960000000001, 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0071806259999999998, 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49471 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656466+0000", 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656466+0000", 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656466+0000", 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4000000000000002e-07 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656485+0000", 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8576999999999998e-05 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.684 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656495+0000", 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0181e-05 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.656501+0000", 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5500000000000002e-06 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.657260+0000", 2024-09-14T04:27:50.685 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00075896299999999998 2024-09-14T04:27:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.657379+0000", 2024-09-14T04:27:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000119017 2024-09-14T04:27:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:27:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.657490+0000", 2024-09-14T04:27:50.686 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011100899999999999 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.659465+0000", 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019747839999999998 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.687 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:49476 14.0 14:02077115:::benchmark_data_smithi114_329146_object49475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.660123+0000", 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0033459309999999999, 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.003592769, 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49476 2024-09-14T04:27:50.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660123+0000", 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660123+0000", 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660124+0000", 2024-09-14T04:27:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1659999999999999e-06 2024-09-14T04:27:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660144+0000", 2024-09-14T04:27:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9885e-05 2024-09-14T04:27:50.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660154+0000", 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.004e-05 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660160+0000", 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8510000000000001e-06 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.661804+0000", 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001643219 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.661943+0000", 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013900699999999999 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 5,6", 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662070+0000", 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00012685800000000001 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:49480 14.e 14:7c9a6984:::benchmark_data_smithi114_329146_object49479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.660179+0000", 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00328963, 2024-09-14T04:27:50.693 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0035956690000000001, 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49480 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.694 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660179+0000", 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660179+0000", 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.695 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660179+0000", 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.3300000000000002e-07 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660200+0000", 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1111000000000002e-05 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660209+0000", 2024-09-14T04:27:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0989999999999999e-06 2024-09-14T04:27:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660214+0000", 2024-09-14T04:27:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7370000000000003e-06 2024-09-14T04:27:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660307+0000", 2024-09-14T04:27:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2978000000000003e-05 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660453+0000", 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014582399999999999 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.660593+0000", 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013968100000000001 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.698 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.47180.0:49478 14.7 e390/385 14:e6dea19a:::benchmark_data_smithi114_329146_object49477:head v 390'3111, mlcod=390'3109)", 2024-09-14T04:27:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.662112+0000", 2024-09-14T04:27:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.001356604, 2024-09-14T04:27:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001725056, 2024-09-14T04:27:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.699 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:27:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662112+0000", 2024-09-14T04:27:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662112+0000", 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662113+0000", 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.1399999999999996e-07 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662149+0000", 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.6201000000000001e-05 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662161+0000", 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2573e-05 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662169+0000", 2024-09-14T04:27:50.702 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4610000000000002e-06 2024-09-14T04:27:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662631+0000", 2024-09-14T04:27:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00046191599999999999 2024-09-14T04:27:50.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662649+0000", 2024-09-14T04:27:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8683000000000001e-05 2024-09-14T04:27:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.704 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.47180.0:49471 14.0 e390/385)", 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.662216+0000", 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0012526320000000001, 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001665794, 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662216+0000", 2024-09-14T04:27:50.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662216+0000", 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662216+0000", 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2000000000000001e-07 2024-09-14T04:27:50.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662218+0000", 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2359999999999999e-06 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662224+0000", 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9390000000000002e-06 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662229+0000", 2024-09-14T04:27:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3440000000000003e-06 2024-09-14T04:27:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.47180.0:49471 14.0 e390/385)", 2024-09-14T04:27:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.662447+0000", 2024-09-14T04:27:50.708 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0010211249999999999, 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001468076, 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662447+0000", 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662447+0000", 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662448+0000", 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0399999999999995e-07 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662455+0000", 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3459999999999996e-06 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662465+0000", 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.7920000000000004e-06 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662472+0000", 2024-09-14T04:27:50.711 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4899999999999997e-06 2024-09-14T04:27:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:49482 14.6 14:6a358d5d:::benchmark_data_smithi114_329146_object49481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-14T04:27:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.662497+0000", 2024-09-14T04:27:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00097131400000000003, 2024-09-14T04:27:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001458254, 2024-09-14T04:27:50.712 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49482 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.713 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662497+0000", 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662497+0000", 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662498+0000", 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.119e-06 2024-09-14T04:27:50.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662548+0000", 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9447999999999998e-05 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662557+0000", 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4550000000000001e-06 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.715 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662563+0000", 2024-09-14T04:27:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9950000000000002e-06 2024-09-14T04:27:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:27:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.663055+0000", 2024-09-14T04:27:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00049148300000000002 2024-09-14T04:27:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.716 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.663189+0000", 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013426599999999999 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 3,6", 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.663321+0000", 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013171400000000001 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:49483 14.0 14.879a2550 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.662854+0000", 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00061428100000000005, 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011557270000000001, 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:27:50.718 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 49483 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662854+0000", 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662854+0000", 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662855+0000", 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1600000000000001e-07 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662880+0000", 2024-09-14T04:27:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4655e-05 2024-09-14T04:27:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.721 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662890+0000", 2024-09-14T04:27:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:50.722 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:50.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:50.722 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0910999999999999e-05 2024-09-14T04:27:50.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.722 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662897+0000", 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9650000000000002e-06 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.47180.0:49469 14.8 e390/385)", 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.662916+0000", 2024-09-14T04:27:50.723 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00055293400000000004, 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0011330699999999999, 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662916+0000", 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662916+0000", 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662916+0000", 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9999999999999998e-07 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662918+0000", 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.413e-06 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662925+0000", 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7959999999999998e-06 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662930+0000", 2024-09-14T04:27:50.726 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8849999999999998e-06 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.47180.0:49470 14.8 e390/385)", 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:27:50.662937+0000", 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00053113300000000004, 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00114466, 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:27:50.727 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662937+0000", 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662937+0000", 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:27:50.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662937+0000", 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5800000000000001e-07 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662939+0000", 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6330000000000001e-06 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.729 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662944+0000", 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2159999999999997e-06 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:27:50.662948+0000", 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.963e-06 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.730 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:27:50.731 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:27:50.731 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 12 2024-09-14T04:27:50.731 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:27:50.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:50.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:50.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:50.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:50.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:50.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:51.098 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:51.133 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:51.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:51.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:51.235 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:51.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:51.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:51.369 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:51.398 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:51.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:51.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:51.529 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:27:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:27:51] "GET /metrics HTTP/1.1" 200 36563 "" "Prometheus/2.51.0" 2024-09-14T04:27:51.536 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:51.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:51.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:51.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 21 16 51078 51062 151.948 113.312 0.0132397 0.0065765 2024-09-14T04:27:51.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:51.673 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:51.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:51.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:51.775 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:51.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:51.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:51.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:51.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:51.957 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:51.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:51.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:52.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-14T04:27:52.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:52.096 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:52.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:52.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:52.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:52.226 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:52.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:52.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:52.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:52.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:52.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:52.485 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:27:52.502 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:52.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:52.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:52 smithi114 ceph-mon[103029]: pgmap v1486: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 434 GiB / 447 GiB avail; 160 MiB/s wr, 2.55k op/s 2024-09-14T04:27:52.529 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:52 smithi114 ceph-mon[106969]: pgmap v1486: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 434 GiB / 447 GiB avail; 160 MiB/s wr, 2.55k op/s 2024-09-14T04:27:52.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 22 16 52790 52774 149.905 107 0.0107306 0.00666662 2024-09-14T04:27:52.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:52.641 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:52.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:52 smithi160 ceph-mon[88407]: pgmap v1486: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 434 GiB / 447 GiB avail; 160 MiB/s wr, 2.55k op/s 2024-09-14T04:27:52.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:52.779 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:52.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:52.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:52.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:52.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:53.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:53.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:53.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:53.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:53.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:53.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:53.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:53.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:53.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:53.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:53.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:53.320 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:27:53.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:53.462 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:53.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:53.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 23 16 54487 54471 147.998 106.062 0.00737572 0.00675245 2024-09-14T04:27:53.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:53.594 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:53.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:53.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:53 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/479858434' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:27:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:53 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:27:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:53 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/479858434' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:27:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:53 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:27:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:27:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:53 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/479858434' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:27:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:53 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:27:53.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:53.736 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:53.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:53.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:53.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:53.874 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:53.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:53.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:54.010 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:54.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:54.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:54.144 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:54.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:54.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:54.277 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:54.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:54.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:54.423 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:54.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:54.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:54.543 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 24 16 56022 56006 145.829 95.9375 0.0026368 0.00684483 2024-09-14T04:27:54.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:54.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:54.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[106969]: pgmap v1487: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-14T04:27:54.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:27:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:27:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:27:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[106969]: osdmap e391: 8 total, 8 up, 5 in 2024-09-14T04:27:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 2]} : dispatch 2024-09-14T04:27:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 2]} : dispatch 2024-09-14T04:27:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]} : dispatch 2024-09-14T04:27:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-14T04:27:54.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 5]} : dispatch 2024-09-14T04:27:54.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[103029]: pgmap v1487: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-14T04:27:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:27:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:27:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:27:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[103029]: osdmap e391: 8 total, 8 up, 5 in 2024-09-14T04:27:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 2]} : dispatch 2024-09-14T04:27:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 2]} : dispatch 2024-09-14T04:27:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]} : dispatch 2024-09-14T04:27:54.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-14T04:27:54.625 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 5]} : dispatch 2024-09-14T04:27:54.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:54 smithi160 ceph-mon[88407]: pgmap v1487: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-14T04:27:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:54 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:27:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:54 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:27:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:54 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:27:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:54 smithi160 ceph-mon[88407]: osdmap e391: 8 total, 8 up, 5 in 2024-09-14T04:27:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 2]} : dispatch 2024-09-14T04:27:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 2]} : dispatch 2024-09-14T04:27:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]} : dispatch 2024-09-14T04:27:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-14T04:27:54.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 5]} : dispatch 2024-09-14T04:27:54.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:54.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:54.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:54.814 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:54.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:54.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:54.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:54.949 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:54.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:55.051 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:55.090 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:55.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:55.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:55.220 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:55.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:55.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:55.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:55.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:55.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:55.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:55.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:55.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 25 16 57536 57520 143.78 94.625 0.00795518 0.00694938 2024-09-14T04:27:55.580 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:55.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:55.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:55.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 2]}]': finished 2024-09-14T04:27:55.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 2]}]': finished 2024-09-14T04:27:55.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]': finished 2024-09-14T04:27:55.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-14T04:27:55.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 5]}]': finished 2024-09-14T04:27:55.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[106969]: osdmap e392: 8 total, 8 up, 5 in 2024-09-14T04:27:55.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 2]}]': finished 2024-09-14T04:27:55.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 2]}]': finished 2024-09-14T04:27:55.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]': finished 2024-09-14T04:27:55.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-14T04:27:55.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 5]}]': finished 2024-09-14T04:27:55.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:55 smithi114 ceph-mon[103029]: osdmap e392: 8 total, 8 up, 5 in 2024-09-14T04:27:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 2]}]': finished 2024-09-14T04:27:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 2]}]': finished 2024-09-14T04:27:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]': finished 2024-09-14T04:27:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-14T04:27:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [3, 5]}]': finished 2024-09-14T04:27:55.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:55 smithi160 ceph-mon[88407]: osdmap e392: 8 total, 8 up, 5 in 2024-09-14T04:27:55.715 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:55.745 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:55.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:55.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:55.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:55.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:55.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:56.010 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:56.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:56.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:56.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:56.144 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:56.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:56.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:56.282 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:56.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:56.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:56.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:56.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:56.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:56.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 26 16 58983 58967 141.728 90.4375 0.00911406 0.00705089 2024-09-14T04:27:56.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:56.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:56.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:56 smithi114 ceph-mon[103029]: pgmap v1490: 99 pgs: 3 peering, 96 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-14T04:27:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:56 smithi114 ceph-mon[103029]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-14T04:27:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:56 smithi114 ceph-mon[103029]: osdmap e393: 8 total, 8 up, 5 in 2024-09-14T04:27:56.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:56 smithi114 ceph-mon[106969]: pgmap v1490: 99 pgs: 3 peering, 96 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-14T04:27:56.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:56 smithi114 ceph-mon[106969]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-14T04:27:56.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:56 smithi114 ceph-mon[106969]: osdmap e393: 8 total, 8 up, 5 in 2024-09-14T04:27:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:56 smithi160 ceph-mon[88407]: pgmap v1490: 99 pgs: 3 peering, 96 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-14T04:27:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:56 smithi160 ceph-mon[88407]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-14T04:27:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:56 smithi160 ceph-mon[88407]: osdmap e393: 8 total, 8 up, 5 in 2024-09-14T04:27:56.658 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:56.689 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:56.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:56.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:56.822 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:56.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:56.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:56.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:56.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:57.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:57.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:57.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:57.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:27:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:57.232 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:57.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:27:57.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:57.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:57.366 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:57.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:57.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:27:57.498 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:57.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:57.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:27:57.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 27 16 60527 60511 140.053 96.5 0.00755033 0.00713559 2024-09-14T04:27:57.599 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:57.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:57.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:57.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:57.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:57.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:57.872 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:57.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:57.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:58.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:58.045 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:58.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:58.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:58.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:58.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:58.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:27:58.316 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:58.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:27:58.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:58.455 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:58.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:58.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 28 16 61866 61850 138.04 83.6875 0.00702479 0.00723913 2024-09-14T04:27:58.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:58.592 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:58.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:58.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:27:58 smithi114 ceph-mon[106969]: pgmap v1492: 99 pgs: 3 peering, 96 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-14T04:27:58.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:27:58 smithi114 ceph-mon[103029]: pgmap v1492: 99 pgs: 3 peering, 96 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-14T04:27:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:27:58 smithi160 ceph-mon[88407]: pgmap v1492: 99 pgs: 3 peering, 96 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-14T04:27:58.694 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:58.729 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:58.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:58.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:58.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:58.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:58.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:58.973 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:59.008 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:59.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:59.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:59.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:27:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:59.153 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:59.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:27:59.254 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:27:59.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:59.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:27:59.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:59.433 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:59.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:59.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:59.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 29 16 63448 63432 136.689 98.875 0.00910837 0.00731101 2024-09-14T04:27:59.572 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:59.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:27:59.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:27:59.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:59.704 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:27:59.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:27:59.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:27:59.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:59.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:27:59.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:27:59.983 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:27:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:27:59.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:00.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:00.120 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:00.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:00.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:00.252 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:00.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:00.322 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:00.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:00.362 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 14 pg_num to 26 2024-09-14T04:28:00.387 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:00.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:00.489 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:00.510 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:00.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:00.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 30 16 64884 64868 135.124 89.75 0.00860976 0.00739522 2024-09-14T04:28:00.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:00 smithi114 ceph-mon[106969]: pgmap v1493: 99 pgs: 3 peering, 96 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T04:28:00.617 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:00 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/88377290' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:28:00.617 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:00 smithi114 ceph-mon[103029]: pgmap v1493: 99 pgs: 3 peering, 96 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T04:28:00.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:00 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/88377290' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:28:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:00.645 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:00.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:00 smithi160 ceph-mon[88407]: pgmap v1493: 99 pgs: 3 peering, 96 active+clean; 3.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-14T04:28:00.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:00 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/88377290' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"} : dispatch 2024-09-14T04:28:00.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:28:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:28:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:28:00.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:00.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:00.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:00.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:00.913 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:00.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:00.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:01.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:01.045 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:01.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:01.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:01.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:01.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:01.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:01.349 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:01.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:01.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:01.490 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:01.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:01.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 31 16 65988 65972 132.991 69 0.164308 0.00748121 2024-09-14T04:28:01.592 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:01.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:01.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:01.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:01 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/88377290' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]': finished 2024-09-14T04:28:01.660 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:01 smithi160 ceph-mon[88407]: osdmap e394: 8 total, 8 up, 5 in 2024-09-14T04:28:01.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:28:01.661 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-14T04:28:01.677 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:28:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:28:01] "GET /metrics HTTP/1.1" 200 36553 "" "Prometheus/2.51.0" 2024-09-14T04:28:01.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:01 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/88377290' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]': finished 2024-09-14T04:28:01.677 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:01 smithi114 ceph-mon[103029]: osdmap e394: 8 total, 8 up, 5 in 2024-09-14T04:28:01.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:28:01.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-14T04:28:01.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:01 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/88377290' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "26"}]': finished 2024-09-14T04:28:01.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:01 smithi114 ceph-mon[106969]: osdmap e394: 8 total, 8 up, 5 in 2024-09-14T04:28:01.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:28:01.678 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-14T04:28:01.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:01.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:01.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:01.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:01.875 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:01.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:01.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:01.997 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:01.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:02.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:02.120 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:02.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:02.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:02.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:02.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:02.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:02.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:02.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:28:02.360 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:28:02.360 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:28:02.360 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:28:02.360 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:28:02.360 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:28:02.360 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:28:02.361 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:28:02.361 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:28:02.361 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:28:02.361 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:28:02.361 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:28:02.361 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:28:02.361 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:28:02.362 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:28:02.362 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:28:02.362 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:28:02.368 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:02.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:02.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:02.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:02.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:02.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 32 16 66137 66121 129.126 9.3125 0.00660145 0.0077398 2024-09-14T04:28:02.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:02.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[103029]: pgmap v1495: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 116 MiB/s wr, 1.85k op/s; 0 B/s, 0 objects/s recovering 2024-09-14T04:28:02.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[103029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-14T04:28:02.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:28:02.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-14T04:28:02.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[103029]: osdmap e395: 8 total, 8 up, 5 in 2024-09-14T04:28:02.618 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2833002669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:02.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[106969]: pgmap v1495: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 116 MiB/s wr, 1.85k op/s; 0 B/s, 0 objects/s recovering 2024-09-14T04:28:02.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[106969]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-14T04:28:02.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:28:02.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-14T04:28:02.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[106969]: osdmap e395: 8 total, 8 up, 5 in 2024-09-14T04:28:02.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:02 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2833002669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:02.647 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:02.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:02 smithi160 ceph-mon[88407]: pgmap v1495: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 116 MiB/s wr, 1.85k op/s; 0 B/s, 0 objects/s recovering 2024-09-14T04:28:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:02 smithi160 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-14T04:28:02.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:02 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:28:02.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:02 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-14T04:28:02.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:02 smithi160 ceph-mon[88407]: osdmap e395: 8 total, 8 up, 5 in 2024-09-14T04:28:02.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:02 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2833002669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:02.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:02.785 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:02.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:02.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:02.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:02.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:03.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:03.092 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:03.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:03.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:03.222 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:03.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:03.324 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:03.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:03.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:03.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:03.466 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:03.503 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:03.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:03.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 33 16 67792 67776 128.348 103.438 0.00440765 0.00778667 2024-09-14T04:28:03.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:03.637 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:03.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:03 smithi160 ceph-mon[88407]: osdmap e396: 8 total, 8 up, 5 in 2024-09-14T04:28:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:03 smithi160 ceph-mon[88407]: 14.2 scrub starts 2024-09-14T04:28:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:03 smithi160 ceph-mon[88407]: 14.1 scrub starts 2024-09-14T04:28:03.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:03 smithi160 ceph-mon[88407]: 14.6 scrub starts 2024-09-14T04:28:03.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:03 smithi160 ceph-mon[88407]: 14.1 scrub ok 2024-09-14T04:28:03.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:03 smithi160 ceph-mon[88407]: 14.6 scrub ok 2024-09-14T04:28:03.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:03 smithi160 ceph-mon[88407]: 14.2 scrub ok 2024-09-14T04:28:03.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:03.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:03.822 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[106969]: osdmap e396: 8 total, 8 up, 5 in 2024-09-14T04:28:03.823 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[106969]: 14.2 scrub starts 2024-09-14T04:28:03.823 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[106969]: 14.1 scrub starts 2024-09-14T04:28:03.823 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[106969]: 14.6 scrub starts 2024-09-14T04:28:03.823 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[106969]: 14.1 scrub ok 2024-09-14T04:28:03.823 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[106969]: 14.6 scrub ok 2024-09-14T04:28:03.823 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[106969]: 14.2 scrub ok 2024-09-14T04:28:03.827 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[103029]: osdmap e396: 8 total, 8 up, 5 in 2024-09-14T04:28:03.827 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[103029]: 14.2 scrub starts 2024-09-14T04:28:03.827 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[103029]: 14.1 scrub starts 2024-09-14T04:28:03.827 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[103029]: 14.6 scrub starts 2024-09-14T04:28:03.827 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[103029]: 14.1 scrub ok 2024-09-14T04:28:03.827 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[103029]: 14.6 scrub ok 2024-09-14T04:28:03.827 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:03 smithi114 ceph-mon[103029]: 14.2 scrub ok 2024-09-14T04:28:03.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:03.909 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:03.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:03.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:04.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:04.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:04.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:04.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:04.172 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:04.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:04.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:04.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:04.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:04.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:04.436 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:04.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:04.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:04.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 34 16 69337 69321 127.413 96.5625 0.0162925 0.00783708 2024-09-14T04:28:04.559 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:04.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:04 smithi160 ceph-mon[88407]: pgmap v1498: 109 pgs: 10 unknown, 99 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 0 B/s, 0 objects/s recovering 2024-09-14T04:28:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:04 smithi160 ceph-mon[88407]: 14.4 deep-scrub starts 2024-09-14T04:28:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:04 smithi160 ceph-mon[88407]: 14.11 scrub starts 2024-09-14T04:28:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:04 smithi160 ceph-mon[88407]: 14.16 scrub starts 2024-09-14T04:28:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:04 smithi160 ceph-mon[88407]: 14.11 scrub ok 2024-09-14T04:28:04.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:04 smithi160 ceph-mon[88407]: 14.16 scrub ok 2024-09-14T04:28:04.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:04.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:04.696 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:04.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:04.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:04.835 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:04.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:04.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[106969]: pgmap v1498: 109 pgs: 10 unknown, 99 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 0 B/s, 0 objects/s recovering 2024-09-14T04:28:04.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[106969]: 14.4 deep-scrub starts 2024-09-14T04:28:04.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[106969]: 14.11 scrub starts 2024-09-14T04:28:04.853 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[106969]: 14.16 scrub starts 2024-09-14T04:28:04.854 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[106969]: 14.11 scrub ok 2024-09-14T04:28:04.854 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[106969]: 14.16 scrub ok 2024-09-14T04:28:04.854 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[103029]: pgmap v1498: 109 pgs: 10 unknown, 99 active+clean; 3.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 77 MiB/s wr, 1.23k op/s; 0 B/s, 0 objects/s recovering 2024-09-14T04:28:04.854 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[103029]: 14.4 deep-scrub starts 2024-09-14T04:28:04.854 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[103029]: 14.11 scrub starts 2024-09-14T04:28:04.854 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[103029]: 14.16 scrub starts 2024-09-14T04:28:04.854 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[103029]: 14.11 scrub ok 2024-09-14T04:28:04.854 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:04 smithi114 ceph-mon[103029]: 14.16 scrub ok 2024-09-14T04:28:04.923 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:04.937 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:04.966 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:04.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:04.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:05.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:05.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:05.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:05.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:05.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:05.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:05.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:05.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:05.377 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:05.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:05.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[106969]: 14.9 scrub starts 2024-09-14T04:28:05.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[106969]: 14.5 scrub starts 2024-09-14T04:28:05.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[106969]: 14.8 scrub starts 2024-09-14T04:28:05.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[106969]: 14.5 scrub ok 2024-09-14T04:28:05.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[106969]: 14.8 scrub ok 2024-09-14T04:28:05.445 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-14T04:28:05.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[103029]: 14.9 scrub starts 2024-09-14T04:28:05.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[103029]: 14.5 scrub starts 2024-09-14T04:28:05.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[103029]: 14.8 scrub starts 2024-09-14T04:28:05.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[103029]: 14.5 scrub ok 2024-09-14T04:28:05.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[103029]: 14.8 scrub ok 2024-09-14T04:28:05.446 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:05 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-14T04:28:05.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:05.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:05.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:05.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 35 16 70581 70565 125.994 77.75 0.0138851 0.00792806 2024-09-14T04:28:05.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:05.639 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:05.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:05 smithi160 ceph-mon[88407]: 14.9 scrub starts 2024-09-14T04:28:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:05 smithi160 ceph-mon[88407]: 14.5 scrub starts 2024-09-14T04:28:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:05 smithi160 ceph-mon[88407]: 14.8 scrub starts 2024-09-14T04:28:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:05 smithi160 ceph-mon[88407]: 14.5 scrub ok 2024-09-14T04:28:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:05 smithi160 ceph-mon[88407]: 14.8 scrub ok 2024-09-14T04:28:05.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:05 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-14T04:28:05.686 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:28:05.686 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:28:05.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:05.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:05.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:05.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:05.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:05.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:06.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:06.031 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:06.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:06.132 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:06.157 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:06.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:06.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:06.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:06.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:06.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:06.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:06.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:06.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:06.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:06.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:06.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:06.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 36 16 70866 70850 122.989 17.8125 0.0102341 0.0079276 2024-09-14T04:28:06.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[103029]: 14.9 scrub ok 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[103029]: 14.7 deep-scrub starts 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[103029]: pgmap v1499: 109 pgs: 3 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 105 active+clean; 4.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 14 MiB/s wr, 229 op/s; 75 KiB/s, 1 objects/s recovering 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[103029]: osdmap e397: 8 total, 8 up, 5 in 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[103029]: 14.3 scrub starts 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[103029]: 14.18 scrub starts 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[103029]: 14.3 scrub ok 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[103029]: 14.18 scrub ok 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[106969]: 14.9 scrub ok 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[106969]: 14.7 deep-scrub starts 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[106969]: pgmap v1499: 109 pgs: 3 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 105 active+clean; 4.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 14 MiB/s wr, 229 op/s; 75 KiB/s, 1 objects/s recovering 2024-09-14T04:28:06.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-14T04:28:06.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[106969]: osdmap e397: 8 total, 8 up, 5 in 2024-09-14T04:28:06.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[106969]: 14.3 scrub starts 2024-09-14T04:28:06.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[106969]: 14.18 scrub starts 2024-09-14T04:28:06.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[106969]: 14.3 scrub ok 2024-09-14T04:28:06.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:06 smithi114 ceph-mon[106969]: 14.18 scrub ok 2024-09-14T04:28:06.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:06.671 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:06.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:06.773 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:06.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:06.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:06.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:06.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:28:06.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:06 smithi160 ceph-mon[88407]: 14.9 scrub ok 2024-09-14T04:28:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:06 smithi160 ceph-mon[88407]: 14.7 deep-scrub starts 2024-09-14T04:28:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:06 smithi160 ceph-mon[88407]: pgmap v1499: 109 pgs: 3 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 105 active+clean; 4.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 14 MiB/s wr, 229 op/s; 75 KiB/s, 1 objects/s recovering 2024-09-14T04:28:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:06 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-14T04:28:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:06 smithi160 ceph-mon[88407]: osdmap e397: 8 total, 8 up, 5 in 2024-09-14T04:28:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:06 smithi160 ceph-mon[88407]: 14.3 scrub starts 2024-09-14T04:28:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:06 smithi160 ceph-mon[88407]: 14.18 scrub starts 2024-09-14T04:28:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:06 smithi160 ceph-mon[88407]: 14.3 scrub ok 2024-09-14T04:28:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:06 smithi160 ceph-mon[88407]: 14.18 scrub ok 2024-09-14T04:28:06.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:06.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:07.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:07.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:07.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:07.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:07.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:07.176 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:07.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:07.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:07.299 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:07.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:07.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:07.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:07.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:07.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:07.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:07.544 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 37 16 70908 70892 119.736 2.625 0.00715292 0.00834606 2024-09-14T04:28:07.554 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:07.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:07.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[103029]: 14.19 scrub starts 2024-09-14T04:28:07.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[103029]: 14.19 scrub ok 2024-09-14T04:28:07.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[103029]: osdmap e398: 8 total, 8 up, 5 in 2024-09-14T04:28:07.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[103029]: 14.15 scrub starts 2024-09-14T04:28:07.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[103029]: 14.0 scrub starts 2024-09-14T04:28:07.680 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[103029]: 14.15 scrub ok 2024-09-14T04:28:07.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[103029]: 14.0 scrub ok 2024-09-14T04:28:07.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[103029]: 14.4 deep-scrub ok 2024-09-14T04:28:07.681 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-14T04:28:07.681 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[106969]: 14.19 scrub starts 2024-09-14T04:28:07.681 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[106969]: 14.19 scrub ok 2024-09-14T04:28:07.681 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[106969]: osdmap e398: 8 total, 8 up, 5 in 2024-09-14T04:28:07.681 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[106969]: 14.15 scrub starts 2024-09-14T04:28:07.681 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[106969]: 14.0 scrub starts 2024-09-14T04:28:07.681 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[106969]: 14.15 scrub ok 2024-09-14T04:28:07.682 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[106969]: 14.0 scrub ok 2024-09-14T04:28:07.682 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[106969]: 14.4 deep-scrub ok 2024-09-14T04:28:07.682 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-14T04:28:07.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:07.683 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:07 smithi160 ceph-mon[88407]: 14.19 scrub starts 2024-09-14T04:28:07.683 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:07 smithi160 ceph-mon[88407]: 14.19 scrub ok 2024-09-14T04:28:07.683 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:07 smithi160 ceph-mon[88407]: osdmap e398: 8 total, 8 up, 5 in 2024-09-14T04:28:07.683 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:07 smithi160 ceph-mon[88407]: 14.15 scrub starts 2024-09-14T04:28:07.683 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:07 smithi160 ceph-mon[88407]: 14.0 scrub starts 2024-09-14T04:28:07.683 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:07 smithi160 ceph-mon[88407]: 14.15 scrub ok 2024-09-14T04:28:07.683 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:07 smithi160 ceph-mon[88407]: 14.0 scrub ok 2024-09-14T04:28:07.683 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:07 smithi160 ceph-mon[88407]: 14.4 deep-scrub ok 2024-09-14T04:28:07.683 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-14T04:28:07.716 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:07.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:07.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:07.840 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:07.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:07.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:07.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:07.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:08.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:08.089 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:08.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:08.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:08.213 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:08.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:08.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:08.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:08.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:08.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:08.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-14T04:28:08.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:08.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:08.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 38 16 71255 71239 117.156 21.6875 0.00204248 0.00833203 2024-09-14T04:28:08.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:08.586 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:08.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: pgmap v1502: 109 pgs: 3 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 105 active+clean; 4.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 77 KiB/s, 1 objects/s recovering 2024-09-14T04:28:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: 14.7 deep-scrub ok 2024-09-14T04:28:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-14T04:28:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: osdmap e399: 8 total, 8 up, 5 in 2024-09-14T04:28:08.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: 14.12 scrub starts 2024-09-14T04:28:08.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: 14.13 scrub starts 2024-09-14T04:28:08.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:08.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:08.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: 14.12 scrub ok 2024-09-14T04:28:08.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: 14.13 scrub ok 2024-09-14T04:28:08.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:08.687 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:08.717 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:08.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:08.771 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:08.776 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:08.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:08.841 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:08.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:08.864 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 4, 5, 2] out_osds: [1, 7, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:28:08.864 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:28:08.864 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-14T04:28:08.864 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:28:08.864 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:28:08.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-14T04:28:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: pgmap v1502: 109 pgs: 3 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 105 active+clean; 4.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 77 KiB/s, 1 objects/s recovering 2024-09-14T04:28:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: 14.7 deep-scrub ok 2024-09-14T04:28:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-14T04:28:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: osdmap e399: 8 total, 8 up, 5 in 2024-09-14T04:28:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: 14.12 scrub starts 2024-09-14T04:28:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: 14.13 scrub starts 2024-09-14T04:28:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: 14.12 scrub ok 2024-09-14T04:28:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: 14.13 scrub ok 2024-09-14T04:28:08.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: pgmap v1502: 109 pgs: 3 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 105 active+clean; 4.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 77 KiB/s, 1 objects/s recovering 2024-09-14T04:28:08.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: 14.7 deep-scrub ok 2024-09-14T04:28:08.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-14T04:28:08.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: osdmap e399: 8 total, 8 up, 5 in 2024-09-14T04:28:08.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: 14.12 scrub starts 2024-09-14T04:28:08.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: 14.13 scrub starts 2024-09-14T04:28:08.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:08.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:08.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: 14.12 scrub ok 2024-09-14T04:28:08.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: 14.13 scrub ok 2024-09-14T04:28:08.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:08.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:08.966 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:08.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:09.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:09.093 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:09.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:09.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:09.217 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:09.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:09.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:09.340 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:09.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:09.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:09.431 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:28:09.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:09.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:09.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:09.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 39 16 71308 71292 114.236 3.3125 0.00686907 0.00874959 2024-09-14T04:28:09.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:09.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:09.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:09.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[103029]: 14.17 scrub starts 2024-09-14T04:28:09.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[103029]: osdmap e400: 8 total, 8 up, 5 in 2024-09-14T04:28:09.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:09.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:09.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/976450416' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:28:09.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:28:09.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[106969]: 14.17 scrub starts 2024-09-14T04:28:09.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[106969]: osdmap e400: 8 total, 8 up, 5 in 2024-09-14T04:28:09.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:09.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:09.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/976450416' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:28:09.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:28:09.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:09.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:09.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:09.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:09.881 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:09.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:09 smithi160 ceph-mon[88407]: 14.17 scrub starts 2024-09-14T04:28:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:09 smithi160 ceph-mon[88407]: osdmap e400: 8 total, 8 up, 5 in 2024-09-14T04:28:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:09 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:09 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/976450416' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:28:09.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:28:09.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:10.012 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:10.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:10.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:10.145 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:10.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:10.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:10.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:10.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:10.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:10.419 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:10.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:10.520 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:10.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:28:10.545112+0000 min lat: 0.00118 max lat: 2.09573 avg lat: 0.00876864 2024-09-14T04:28:10.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:28:10.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 40 16 72974 72958 113.983 104.125 0.0175489 0.00876864 2024-09-14T04:28:10.558 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:10.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:10.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:10 smithi114 ceph-mon[106969]: 14.17 scrub starts 2024-09-14T04:28:10.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:10 smithi114 ceph-mon[106969]: pgmap v1505: 109 pgs: 1 remapped+peering, 3 active+clean+scrubbing, 105 active+clean; 4.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail 2024-09-14T04:28:10.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:10 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:28:10.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:10 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:28:10.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:10 smithi114 ceph-mon[106969]: osdmap e401: 8 total, 8 up, 5 in 2024-09-14T04:28:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:10 smithi114 ceph-mon[103029]: 14.17 scrub starts 2024-09-14T04:28:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:10 smithi114 ceph-mon[103029]: pgmap v1505: 109 pgs: 1 remapped+peering, 3 active+clean+scrubbing, 105 active+clean; 4.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail 2024-09-14T04:28:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:10 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:28:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:10 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:28:10.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:10 smithi114 ceph-mon[103029]: osdmap e401: 8 total, 8 up, 5 in 2024-09-14T04:28:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:28:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:28:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:28:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:10 smithi160 ceph-mon[88407]: 14.17 scrub starts 2024-09-14T04:28:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:10 smithi160 ceph-mon[88407]: pgmap v1505: 109 pgs: 1 remapped+peering, 3 active+clean+scrubbing, 105 active+clean; 4.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail 2024-09-14T04:28:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:10 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:28:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:10 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:28:10.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:10 smithi160 ceph-mon[88407]: osdmap e401: 8 total, 8 up, 5 in 2024-09-14T04:28:10.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:10.707 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:10.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:10.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:10.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:10.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:10.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:10.947 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:10.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:10.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:11.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:11.111 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:11.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:11.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:11.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:11.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:11.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:11.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:11.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:11.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:11.522 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:11.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:11.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 41 16 74636 74620 113.737 103.875 0.0245925 0.00878785 2024-09-14T04:28:11.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:11.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:11.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:11.713 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:28:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:28:11] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-09-14T04:28:11.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:11.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:11.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:11.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:11.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:12.032 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:12.068 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:12.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:12.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:12.207 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:12.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:12.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:12.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:12.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:12.440 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:12.473 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:12.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:12.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 42 15 76303 76288 113.511 104.25 0.00689962 0.00880518 2024-09-14T04:28:12.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:12.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:12.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:12.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:12.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:12.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:12.848 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:12.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:12 smithi114 ceph-mon[106969]: pgmap v1507: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 107 active+clean; 4.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1786/214479 objects misplaced (0.833%); 4.1 MiB/s, 65 objects/s recovering 2024-09-14T04:28:12.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:12 smithi114 ceph-mon[103029]: pgmap v1507: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 107 active+clean; 4.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1786/214479 objects misplaced (0.833%); 4.1 MiB/s, 65 objects/s recovering 2024-09-14T04:28:12.882 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:12.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:12.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:12 smithi160 ceph-mon[88407]: pgmap v1507: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 107 active+clean; 4.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1786/214479 objects misplaced (0.833%); 4.1 MiB/s, 65 objects/s recovering 2024-09-14T04:28:12.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:13.015 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:13.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:13.079 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:13.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:13.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:13.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:13.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:13.308 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:13.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:13.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:13.418 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:28:13.418 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:28:13.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:13.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:13.545 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 43 16 77940 77924 113.249 102.25 0.0104232 0.00882542 2024-09-14T04:28:13.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:13.586 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:13.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:13.687 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:13.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:13.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:13.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:13.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:13.862 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:13.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:13.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:28:13.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:14.001 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:14.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:14.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:14.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:14.143 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:14.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:14.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:14.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:14.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:14.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:14.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:14.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:14.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:14.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:14.539 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:14 smithi114 ceph-mon[103029]: pgmap v1508: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 107 active+clean; 4.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1786/214479 objects misplaced (0.833%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:28:14.540 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:14 smithi114 ceph-mon[106969]: pgmap v1508: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 107 active+clean; 4.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1786/214479 objects misplaced (0.833%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:28:14.546 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 44 16 79553 79537 112.963 100.812 0.0104715 0.00884769 2024-09-14T04:28:14.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:14.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:14.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:14.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:14.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:14.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:14.831 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:14.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:14 smithi160 ceph-mon[88407]: pgmap v1508: 109 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 107 active+clean; 4.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1786/214479 objects misplaced (0.833%); 3.3 MiB/s, 52 objects/s recovering 2024-09-14T04:28:14.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:14.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:14.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:15.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:15.113 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:15.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:15.171 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:28:15.171 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:28:15.210 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:28:15.210 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:28:15.215 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:15.241 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:28:15.241 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:28:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:15.249 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:15.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:15.275 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:28:15.275 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:28:15.302 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:28:15.302 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:28:15.330 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:28:15.331 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:28:15.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:15.357 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:28:15.357 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:28:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:28:15.383 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:28:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:15.389 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:15.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:15.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:15.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:15.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:15.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:15.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 45 16 81155 81139 112.678 100.125 0.0112329 0.00887089 2024-09-14T04:28:15.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3708144796' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:15.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3708144796' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:15.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:15.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:15.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:15.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:15.761 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:15.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:15.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:15.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:15.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3708144796' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:15.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:15.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:16.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:16.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:16.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:16.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:16.215 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:16.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:16.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-14T04:28:16.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:16.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:16.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:16.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:16.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:16.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:16 smithi114 ceph-mon[106969]: pgmap v1509: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 4.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 2225/238479 objects degraded (0.933%); 1384/238479 objects misplaced (0.580%); 5.8 MiB/s, 92 objects/s recovering 2024-09-14T04:28:16.516 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:16 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 2225/238479 objects degraded (0.933%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:16.517 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:16 smithi114 ceph-mon[103029]: pgmap v1509: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 4.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 2225/238479 objects degraded (0.933%); 1384/238479 objects misplaced (0.580%); 5.8 MiB/s, 92 objects/s recovering 2024-09-14T04:28:16.517 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:16 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 2225/238479 objects degraded (0.933%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:16.524 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:16.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:16.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 46 15 82763 82748 112.414 100.562 0.00871847 0.00889121 2024-09-14T04:28:16.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:16.652 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:16.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:16.753 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:16.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:16.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:16.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:16 smithi160 ceph-mon[88407]: pgmap v1509: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 4.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 2225/238479 objects degraded (0.933%); 1384/238479 objects misplaced (0.580%); 5.8 MiB/s, 92 objects/s recovering 2024-09-14T04:28:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:16 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 2225/238479 objects degraded (0.933%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:16.928 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:16.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:17.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:17.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:17.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:17.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:17.201 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:17.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:17.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:17.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:17.335 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:17.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:17.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:17.478 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:17.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:17.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 47 16 84289 84273 112.05 95.3125 0.00520902 0.00892051 2024-09-14T04:28:17.579 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:17.611 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:17.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:17.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:17.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:17.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:17.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:17.885 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:17.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:17.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:18.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:18.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:18.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:18.131 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:18.165 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:18.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:18.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:18.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:18.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:18.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:18.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:18.440 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:18.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:18.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:18.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 48 16 85847 85831 111.744 97.375 0.00433412 0.00894442 2024-09-14T04:28:18.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:18.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:18.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:18.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:18.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:18.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:18.854 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:18.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:18.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:18.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:18 smithi114 ceph-mon[103029]: pgmap v1510: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 4.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 2225/238479 objects degraded (0.933%); 1384/238479 objects misplaced (0.580%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T04:28:18.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:18 smithi114 ceph-mon[106969]: pgmap v1510: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 4.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 2225/238479 objects degraded (0.933%); 1384/238479 objects misplaced (0.580%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T04:28:18.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:18 smithi160 ceph-mon[88407]: pgmap v1510: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 4.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 2225/238479 objects degraded (0.933%); 1384/238479 objects misplaced (0.580%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T04:28:18.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:18.998 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:18.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:19.100 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:19.135 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:19.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:19.236 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:19.274 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:19.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:19.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:19.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:19.417 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:19.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:19.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:19.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 49 16 87361 87345 111.394 94.625 0.0193745 0.0089722 2024-09-14T04:28:19.562 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:19.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:19.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:19.705 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:19.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:19.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:19.833 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:19.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:19.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:19.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:19.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:20.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:28:20.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:20.103 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:20.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:20.205 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:20.242 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:20.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:20.344 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:20.384 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:20.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:20.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:20.465 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:28:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:28:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:28:20.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:20.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:20.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:20.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:20.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 50 16 88826 88810 110.998 91.5625 0.00878464 0.00900479 2024-09-14T04:28:20.629 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:20.665 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:20.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:20.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:20.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:20.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:20.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:20 smithi114 ceph-mon[106969]: pgmap v1511: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 5.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 2225/243621 objects degraded (0.913%); 1384/243621 objects misplaced (0.568%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:28:20.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:20 smithi114 ceph-mon[103029]: pgmap v1511: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 5.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 2225/243621 objects degraded (0.913%); 1384/243621 objects misplaced (0.568%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:28:20.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:20.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:20 smithi160 ceph-mon[88407]: pgmap v1511: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 5.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 2225/243621 objects degraded (0.913%); 1384/243621 objects misplaced (0.568%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:28:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:20.941 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:20.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:21.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:21.083 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:21.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:21.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:21.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:21.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:21.323 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:21.360 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:21.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:21.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:21.495 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:21.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:21.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 51 16 90327 90311 110.661 93.8125 0.0100311 0.00903191 2024-09-14T04:28:21.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:21.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:21.726 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:21.765 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:21.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:21.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:21.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:21.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:21 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2225/262068 objects degraded (0.849%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:21.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:21 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2225/262068 objects degraded (0.849%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:21.872 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:28:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:28:21] "GET /metrics HTTP/1.1" 200 36564 "" "Prometheus/2.51.0" 2024-09-14T04:28:21.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:21.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:21.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:21 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2225/262068 objects degraded (0.849%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:22.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:22.044 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:22.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:22.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:22.172 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:22.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:22.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:22.315 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:22.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:22.417 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:22.443 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:22.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:22.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:22.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:22.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 52 16 91583 91567 110.042 78.5 0.0162349 0.00908268 2024-09-14T04:28:22.584 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:22.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:22.685 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:22.724 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:22.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:22.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:22.847 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:22.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:22.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:22 smithi114 ceph-mon[106969]: pgmap v1512: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 9.0 MiB/s wr, 144 op/s; 2225/262068 objects degraded (0.849%); 922/262068 objects misplaced (0.352%); 6.3 MiB/s, 100 objects/s recovering 2024-09-14T04:28:22.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:22 smithi114 ceph-mon[103029]: pgmap v1512: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 9.0 MiB/s wr, 144 op/s; 2225/262068 objects degraded (0.849%); 922/262068 objects misplaced (0.352%); 6.3 MiB/s, 100 objects/s recovering 2024-09-14T04:28:22.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:22 smithi160 ceph-mon[88407]: pgmap v1512: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 9.0 MiB/s wr, 144 op/s; 2225/262068 objects degraded (0.849%); 922/262068 objects misplaced (0.352%); 6.3 MiB/s, 100 objects/s recovering 2024-09-14T04:28:22.948 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:22.987 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:22.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:22.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:23.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:23.117 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:23.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:23.218 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:23.247 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:23.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:23.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:23.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:23.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:23.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:23.528 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:23.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:23.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 53 16 92919 92903 109.541 83.5 0.00693586 0.00912462 2024-09-14T04:28:23.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:23.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:23.665 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:23.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:23.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:23.792 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:23.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:23.894 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:23.928 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:23.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:24.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:24.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:24.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:24.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:24.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:24.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:24.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:24.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:24.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:24.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:24.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:24.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:24.322 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:24.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:24.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:24.458 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:24.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:24.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 54 16 94124 94108 108.907 75.3125 0.00403068 0.00917765 2024-09-14T04:28:24.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:24.586 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:24.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:24.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:24.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:24.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:24.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:24.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:24.859 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:24.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:24.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:24.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:24.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:25.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:25.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:25.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:25.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:24 smithi114 ceph-mon[103029]: pgmap v1513: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 83 MiB/s wr, 1.32k op/s; 2225/262068 objects degraded (0.849%); 922/262068 objects misplaced (0.352%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:28:25.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:24 smithi114 ceph-mon[106969]: pgmap v1513: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 83 MiB/s wr, 1.32k op/s; 2225/262068 objects degraded (0.849%); 922/262068 objects misplaced (0.352%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:28:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:24 smithi160 ceph-mon[88407]: pgmap v1513: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 83 MiB/s wr, 1.32k op/s; 2225/262068 objects degraded (0.849%); 922/262068 objects misplaced (0.352%); 4.5 MiB/s, 71 objects/s recovering 2024-09-14T04:28:25.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:25.263 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:25.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:25.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:25.365 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:25.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:25.398 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:25.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:25.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:25.529 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:25.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:25.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:25.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 55 16 95396 95380 108.372 79.5 0.00781604 0.00922313 2024-09-14T04:28:25.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:25.660 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:25.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:25.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:25.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:25.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:25.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:25.895 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:25.943 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:25.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:26.044 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:26.077 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:26.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:26.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:26.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:26.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:26.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:26.354 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:26.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:26.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:26.488 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:26.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:26.547 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 56 16 96783 96767 107.985 86.6875 0.00653084 0.00925631 2024-09-14T04:28:26.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:26.620 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:26.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:26.721 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:26.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:26.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:26.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:26.903 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:26.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:27.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:27.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:27.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:27.043 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:26 smithi114 ceph-mon[106969]: pgmap v1514: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 5.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 118 MiB/s wr, 1.89k op/s; 2225/282414 objects degraded (0.788%); 425/282414 objects misplaced (0.150%); 7.1 MiB/s, 113 objects/s recovering 2024-09-14T04:28:27.043 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:26 smithi114 ceph-mon[103029]: pgmap v1514: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 5.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 118 MiB/s wr, 1.89k op/s; 2225/282414 objects degraded (0.788%); 425/282414 objects misplaced (0.150%); 7.1 MiB/s, 113 objects/s recovering 2024-09-14T04:28:27.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:26 smithi160 ceph-mon[88407]: pgmap v1514: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 5.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 118 MiB/s wr, 1.89k op/s; 2225/282414 objects degraded (0.788%); 425/282414 objects misplaced (0.150%); 7.1 MiB/s, 113 objects/s recovering 2024-09-14T04:28:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:27.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:27.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:27.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:27.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:27.292 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:27.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:27.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:27.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:27.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:27.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:27.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 57 16 98111 98095 107.547 83 0.00464679 0.00929407 2024-09-14T04:28:27.554 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:27.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:27.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:27.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:27.701 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:27.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:27.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:27.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:27.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:27.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:27.963 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:27.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:28.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:28.097 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:28.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:28.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:28.221 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:28.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:28.323 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:28.362 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:28.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:28.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:28.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:28.491 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:28.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:28.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:28.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 58 16 99535 99519 107.227 89 0.0080972 0.00932211 2024-09-14T04:28:28.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:28.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:28.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:28.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:28.753 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:28.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:28.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:28.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:28.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:28.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:28.994 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:29.022 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:29.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:29.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:28 smithi114 ceph-mon[103029]: pgmap v1515: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 5.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 76 MiB/s wr, 1.22k op/s; 2225/282414 objects degraded (0.788%); 425/282414 objects misplaced (0.150%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T04:28:29.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:28 smithi114 ceph-mon[106969]: pgmap v1515: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 5.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 76 MiB/s wr, 1.22k op/s; 2225/282414 objects degraded (0.788%); 425/282414 objects misplaced (0.150%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T04:28:29.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:29.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:29.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:29.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:28 smithi160 ceph-mon[88407]: pgmap v1515: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 5.7 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 76 MiB/s wr, 1.22k op/s; 2225/282414 objects degraded (0.788%); 425/282414 objects misplaced (0.150%); 5.0 MiB/s, 79 objects/s recovering 2024-09-14T04:28:29.248 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:29.271 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:29.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:29.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:29.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:29.395 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:29.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:29.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:29.519 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:29.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:29.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 59 16 100348 100332 106.27 50.8125 0.00290644 0.00931236 2024-09-14T04:28:29.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:29.643 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:29.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:29.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:29.770 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:29.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:29.871 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:29.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:29.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:30.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:30.034 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:30.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:30.053 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:29 smithi114 ceph-mon[106969]: osdmap e402: 8 total, 8 up, 5 in 2024-09-14T04:28:30.053 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:29 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-14T04:28:30.053 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 2225/282414 objects degraded (0.788%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:30.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:29 smithi114 ceph-mon[103029]: osdmap e402: 8 total, 8 up, 5 in 2024-09-14T04:28:30.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:29 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-14T04:28:30.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 2225/282414 objects degraded (0.788%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:29 smithi160 ceph-mon[88407]: osdmap e402: 8 total, 8 up, 5 in 2024-09-14T04:28:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:29 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-14T04:28:30.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 2225/282414 objects degraded (0.788%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:30.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:30.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:30.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:30.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:30.290 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:30.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:30.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:30.408 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:28:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:28:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:28:30.414 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:30.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:30.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:30.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:30.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:30.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:30.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:28:30.548010+0000 min lat: 0.00118 max lat: 2.09573 avg lat: 0.00943162 2024-09-14T04:28:30.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:28:30.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 60 16 100916 100900 105.091 35.5 0.00290854 0.00943162 2024-09-14T04:28:30.639 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:30.661 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:30.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:30.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:30.786 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:30.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:30.888 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:30.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:30 smithi160 ceph-mon[88407]: pgmap v1517: 109 pgs: 2 active+recovering+undersized+remapped, 107 active+clean; 5.8 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 90 MiB/s wr, 1.44k op/s; 2604/286776 objects misplaced (0.908%); 6.3 MiB/s, 100 objects/s recovering 2024-09-14T04:28:30.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:30 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2225/282414 objects degraded (0.788%), 1 pg degraded) 2024-09-14T04:28:30.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:30 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-14T04:28:30.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:30 smithi160 ceph-mon[88407]: osdmap e403: 8 total, 8 up, 5 in 2024-09-14T04:28:30.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:30 smithi160 ceph-mon[88407]: 14.10 scrub starts 2024-09-14T04:28:30.909 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:30.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:31.012 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:30 smithi114 ceph-mon[103029]: pgmap v1517: 109 pgs: 2 active+recovering+undersized+remapped, 107 active+clean; 5.8 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 90 MiB/s wr, 1.44k op/s; 2604/286776 objects misplaced (0.908%); 6.3 MiB/s, 100 objects/s recovering 2024-09-14T04:28:31.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:30 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2225/282414 objects degraded (0.788%), 1 pg degraded) 2024-09-14T04:28:31.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:30 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-14T04:28:31.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:30 smithi114 ceph-mon[103029]: osdmap e403: 8 total, 8 up, 5 in 2024-09-14T04:28:31.013 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:30 smithi114 ceph-mon[103029]: 14.10 scrub starts 2024-09-14T04:28:31.014 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:30 smithi114 ceph-mon[106969]: pgmap v1517: 109 pgs: 2 active+recovering+undersized+remapped, 107 active+clean; 5.8 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 90 MiB/s wr, 1.44k op/s; 2604/286776 objects misplaced (0.908%); 6.3 MiB/s, 100 objects/s recovering 2024-09-14T04:28:31.014 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:30 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2225/282414 objects degraded (0.788%), 1 pg degraded) 2024-09-14T04:28:31.014 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:30 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-14T04:28:31.014 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:30 smithi114 ceph-mon[106969]: osdmap e403: 8 total, 8 up, 5 in 2024-09-14T04:28:31.014 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:30 smithi114 ceph-mon[106969]: 14.10 scrub starts 2024-09-14T04:28:31.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:31.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:31.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:31.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:31.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:31.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:31.283 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:31.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:31.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:31.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:31.408 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:31.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:31.510 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:31.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:31.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:31.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 61 16 100916 100900 103.368 0 - 0.00943162 2024-09-14T04:28:31.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:28:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:28:31] "GET /metrics HTTP/1.1" 200 36575 "" "Prometheus/2.51.0" 2024-09-14T04:28:31.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:31.658 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:31.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:31.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:31.781 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:31.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:31.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:31.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:31.907 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:31.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:32.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:32.043 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:32.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:32.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:31 smithi114 ceph-mon[103029]: osdmap e404: 8 total, 8 up, 5 in 2024-09-14T04:28:32.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:31 smithi114 ceph-mon[106969]: osdmap e404: 8 total, 8 up, 5 in 2024-09-14T04:28:32.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:32.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:31 smithi160 ceph-mon[88407]: osdmap e404: 8 total, 8 up, 5 in 2024-09-14T04:28:32.181 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:32.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:32.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:32.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:32.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:32.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:32.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:32.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:32.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:32.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 62 16 101749 101733 102.54 26.0312 0.0156537 0.00974744 2024-09-14T04:28:32.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:32.568 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:32.607 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:32.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:32.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:32.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:32.743 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:32.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:32.813 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:28:32.813 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "" 2024-09-14T04:28:32.813 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:28:32.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:32.883 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:32.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:32.912 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:32 smithi114 ceph-mon[103029]: pgmap v1520: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 107 active+clean; 6.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 65 MiB/s wr, 1.04k op/s; 2179/301209 objects misplaced (0.723%); 4.9 MiB/s, 78 objects/s recovering 2024-09-14T04:28:32.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:32 smithi114 ceph-mon[103029]: osdmap e405: 8 total, 8 up, 5 in 2024-09-14T04:28:32.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:32.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:32.913 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:32 smithi114 ceph-mon[106969]: pgmap v1520: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 107 active+clean; 6.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 65 MiB/s wr, 1.04k op/s; 2179/301209 objects misplaced (0.723%); 4.9 MiB/s, 78 objects/s recovering 2024-09-14T04:28:32.913 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:32 smithi114 ceph-mon[106969]: osdmap e405: 8 total, 8 up, 5 in 2024-09-14T04:28:32.913 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:32.913 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:32.985 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:33.030 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:33.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:33.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:33.133 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:32 smithi160 ceph-mon[88407]: pgmap v1520: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 107 active+clean; 6.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 65 MiB/s wr, 1.04k op/s; 2179/301209 objects misplaced (0.723%); 4.9 MiB/s, 78 objects/s recovering 2024-09-14T04:28:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:32 smithi160 ceph-mon[88407]: osdmap e405: 8 total, 8 up, 5 in 2024-09-14T04:28:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:33.178 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:33.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:33.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:33.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:33.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:33.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:33.439 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:33.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:33.540 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:33.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 63 16 103093 103077 102.246 84 0.0041957 0.00977568 2024-09-14T04:28:33.579 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:33.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:33.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:33.710 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:33.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:33.812 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:33.841 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:33.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:33.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:33.974 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:33.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:34.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:34.118 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:34.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:34.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:34.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:28:34.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:28:34.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:34.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:33 smithi114 ceph-mon[106969]: pgmap v1522: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 107 active+clean; 6.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s; 2179/301209 objects misplaced (0.723%); 6.5 MiB/s, 104 objects/s recovering 2024-09-14T04:28:34.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:28:34.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:28:34.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:34.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:33 smithi114 ceph-mon[103029]: pgmap v1522: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 107 active+clean; 6.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s; 2179/301209 objects misplaced (0.723%); 6.5 MiB/s, 104 objects/s recovering 2024-09-14T04:28:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:28:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:28:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:28:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:33 smithi160 ceph-mon[88407]: pgmap v1522: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 107 active+clean; 6.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 87 MiB/s wr, 1.39k op/s; 2179/301209 objects misplaced (0.723%); 6.5 MiB/s, 104 objects/s recovering 2024-09-14T04:28:34.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:34.260 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:34.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:34.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:34.397 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:34.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:34.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:34.538 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:34.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:34.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:34.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 64 16 104509 104493 102.031 88.5 0.00743515 0.00979671 2024-09-14T04:28:34.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:34.668 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:34.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:34.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:34.812 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:34.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:34.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:34.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:34.947 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:34.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:35.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:35.083 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:35.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:35.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:35.218 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:35.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:35.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:35.364 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:35.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:35.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:35.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:35.500 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:35.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:35.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 65 16 105860 105844 101.76 84.4375 0.0213195 0.00982235 2024-09-14T04:28:35.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:35.643 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:35.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:35.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:35.778 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:35.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:35.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:35.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:35 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 3157/313557 objects degraded (1.007%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:35.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:35 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 3157/313557 objects degraded (1.007%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:35.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:35.922 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:35.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:36.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:36.066 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:36.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:36.118 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:36.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:35 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 3157/313557 objects degraded (1.007%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:36.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:36.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:36.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:36.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:36.334 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:36.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:36.411 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:28:36.411 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:28:36.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:107043 14.13 14:cbf7e22b:::benchmark_data_smithi114_329146_object107042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:28:36.401747+0000", 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0076201389999999997, 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0076558800000000003, 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 107043 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.412 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.401747+0000", 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.401747+0000", 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.401748+0000", 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.5499999999999996e-07 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.401771+0000", 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.2631000000000001e-05 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.401784+0000", 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2473999999999999e-05 2024-09-14T04:28:36.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.401793+0000", 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4790000000000002e-06 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.401854+0000", 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0844000000000002e-05 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:28:36.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.401985+0000", 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000130891 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,5", 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.402305+0000", 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000320064 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.404516+0000", 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022113720000000001 2024-09-14T04:28:36.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.404807+0000", 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00029060700000000002 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.47180.0:107046 14.4 e405/395 14:22297ec7:::benchmark_data_smithi114_329146_object107045:head v 405'5810, mlcod=405'5808)", 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:28:36.406349+0000", 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0030184550000000002, 2024-09-14T04:28:36.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0031410940000000001, 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.406349+0000", 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.406349+0000", 2024-09-14T04:28:36.418 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.406350+0000", 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.46e-07 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.406389+0000", 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9576999999999997e-05 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.419 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.406397+0000", 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.6769999999999995e-06 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.406402+0000", 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2610000000000001e-06 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.406529+0000", 2024-09-14T04:28:36.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000127429 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.406551+0000", 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1552e-05 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.421 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:107051 14.12 14:4a0b199f:::benchmark_data_smithi114_329146_object107050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:28:36.407205+0000", 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00216278, 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023441149999999999, 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "reached pg", 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 107051 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:28:36.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407205+0000", 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407205+0000", 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407206+0000", 2024-09-14T04:28:36.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.31e-06 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407231+0000", 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.516e-05 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407247+0000", 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6180000000000001e-05 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407255+0000", 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9410000000000007e-06 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.409438+0000", 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021824769999999999 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.425 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:107054 14.f 14:f786473e:::benchmark_data_smithi114_329146_object107053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-14T04:28:36.426 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:28:36.407271+0000", 2024-09-14T04:28:36.426 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.002096409, 2024-09-14T04:28:36.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002338101, 2024-09-14T04:28:36.426 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:28:36.426 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:28:36.426 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:28:36.426 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 107054 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407271+0000", 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407271+0000", 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407272+0000", 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9900000000000001e-07 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:28:36.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407292+0000", 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0664000000000001e-05 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407300+0000", 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8050000000000007e-06 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407306+0000", 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.879e-06 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407358+0000", 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2268999999999997e-05 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407507+0000", 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014876200000000001 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.407663+0000", 2024-09-14T04:28:36.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000155771 2024-09-14T04:28:36.431 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.431 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:28:36.431 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.431 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.47180.0:107053 14.2 e405/395 14:442881b8:::benchmark_data_smithi114_329146_object107052:head v 405'5412, mlcod=405'5410)", 2024-09-14T04:28:36.431 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:28:36.408670+0000", 2024-09-14T04:28:36.431 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00069720299999999997, 2024-09-14T04:28:36.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001005429, 2024-09-14T04:28:36.431 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:28:36.432 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:28:36.432 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:28:36.432 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:28:36.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:28:36.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408670+0000", 2024-09-14T04:28:36.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408670+0000", 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408671+0000", 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.6700000000000002e-07 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408729+0000", 2024-09-14T04:28:36.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7605999999999997e-05 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408741+0000", 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2313000000000001e-05 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408748+0000", 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7490000000000001e-06 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.409186+0000", 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00043766200000000001 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.409204+0000", 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8000999999999999e-05 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.435 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop(client.47180.0:107050 14.4 e405/395 14:2021f605:::benchmark_data_smithi114_329146_object107049:head v 405'5811, mlcod=405'5808)", 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:28:36.408762+0000", 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.00060559100000000005, 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00096625699999999999, 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "started", 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408762+0000", 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408762+0000", 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408762+0000", 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.4200000000000002e-07 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408789+0000", 2024-09-14T04:28:36.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.654e-05 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408800+0000", 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0652000000000001e-05 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408804+0000", 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7450000000000003e-06 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408981+0000", 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017627800000000001 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.408998+0000", 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7393000000000001e-05 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_repop_reply(client.47180.0:107043 14.13 e405/395)", 2024-09-14T04:28:36.439 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:28:36.409532+0000", 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4294967295.999836, 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024623000000000001, 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "queued for pg", 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.409532+0000", 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:28:36.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.409532+0000", 2024-09-14T04:28:36.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:28:36.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.409533+0000", 2024-09-14T04:28:36.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.3300000000000001e-07 2024-09-14T04:28:36.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.409540+0000", 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0820000000000001e-06 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.409548+0000", 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.619e-06 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.409554+0000", 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0209999999999996e-06 2024-09-14T04:28:36.442 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:107038 14.f 14:f470ced7:::benchmark_data_smithi114_329146_object107037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:28:36.396825+0000", 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.012542653000000001, 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013013540000000001, 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:28:36.443 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 107038 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.396825+0000", 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:28:36.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.396825+0000", 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.396827+0000", 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8369999999999999e-06 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.396851+0000", 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.4273e-05 2024-09-14T04:28:36.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.396878+0000", 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.7338000000000002e-05 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.396888+0000", 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4879999999999996e-06 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.403630+0000", 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0067416669999999998 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.403763+0000", 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000133227 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 2,5", 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:28:36.403934+0000", 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00017143099999999999 2024-09-14T04:28:36.447 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.448 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:28:36.448 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.448 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:28:36.448 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-09-14T04:28:36.448 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 8 2024-09-14T04:28:36.448 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:28:36.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:36.485 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:36.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:36.548 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 66 16 107243 107227 101.528 86.4375 0.00790627 0.00984513 2024-09-14T04:28:36.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:36.621 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:36.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:28:36.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:36.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:36.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:36.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:36.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:36 smithi114 ceph-mon[106969]: pgmap v1523: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 93 MiB/s wr, 1.49k op/s; 3157/313557 objects degraded (1.007%); 4741/313557 objects misplaced (1.512%); 11 MiB/s, 169 objects/s recovering 2024-09-14T04:28:36.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:36 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2410300730' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:28:36.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:36 smithi114 ceph-mon[103029]: pgmap v1523: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 93 MiB/s wr, 1.49k op/s; 3157/313557 objects degraded (1.007%); 4741/313557 objects misplaced (1.512%); 11 MiB/s, 169 objects/s recovering 2024-09-14T04:28:36.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:36 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2410300730' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:28:36.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:36.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:36.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:36.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:37.027 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:37.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:37.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:37.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:36 smithi160 ceph-mon[88407]: pgmap v1523: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 93 MiB/s wr, 1.49k op/s; 3157/313557 objects degraded (1.007%); 4741/313557 objects misplaced (1.512%); 11 MiB/s, 169 objects/s recovering 2024-09-14T04:28:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:36 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2410300730' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:28:37.167 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:37.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:37.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:37.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:37.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:37.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:37.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:37.430 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:37.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:37.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:37.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 67 16 108641 108625 101.316 87.375 0.00920683 0.0098659 2024-09-14T04:28:37.565 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:37.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:37.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:37.667 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:37.714 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:37.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:37.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:37.853 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:37.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:37.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:37 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:28:37.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:37 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2410300730' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:28:37.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:37 smithi114 ceph-mon[106969]: osdmap e406: 8 total, 8 up, 5 in 2024-09-14T04:28:37.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:37.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:37 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:28:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:37 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2410300730' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:28:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:37 smithi114 ceph-mon[103029]: osdmap e406: 8 total, 8 up, 5 in 2024-09-14T04:28:37.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:37.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:37.983 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:37.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:38.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:38.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:38.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:38.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:37 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:28:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:37 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2410300730' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:28:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:37 smithi160 ceph-mon[88407]: osdmap e406: 8 total, 8 up, 5 in 2024-09-14T04:28:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:38.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:38.262 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:38.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:38.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:38.402 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:38.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:38.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:38.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:38.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:38.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 68 16 109969 109953 101.047 83 0.00757991 0.00989208 2024-09-14T04:28:38.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:38.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:38.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:38.779 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:38.813 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:38.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:38.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:38.943 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:38.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:38.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:39.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:39.074 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:39.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:39.097 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:38 smithi114 ceph-mon[103029]: pgmap v1525: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 40 MiB/s wr, 633 op/s; 3157/313557 objects degraded (1.007%); 4741/313557 objects misplaced (1.512%); 5.7 MiB/s, 91 objects/s recovering 2024-09-14T04:28:39.098 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:38 smithi114 ceph-mon[106969]: pgmap v1525: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 40 MiB/s wr, 633 op/s; 3157/313557 objects degraded (1.007%); 4741/313557 objects misplaced (1.512%); 5.7 MiB/s, 91 objects/s recovering 2024-09-14T04:28:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:38 smithi160 ceph-mon[88407]: pgmap v1525: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 40 MiB/s wr, 633 op/s; 3157/313557 objects degraded (1.007%); 4741/313557 objects misplaced (1.512%); 5.7 MiB/s, 91 objects/s recovering 2024-09-14T04:28:39.177 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:39.212 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:39.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:39.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:39.353 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:39.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:39.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:39.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:39.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:39.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 69 16 111356 111340 100.839 86.6875 0.00653619 0.00991299 2024-09-14T04:28:39.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:39.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:39.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:39.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:39.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:39.757 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:39.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:39.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:39.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:39.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:40.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:40.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:40.050 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:40.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:40.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:40.193 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:40.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:40.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:40.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-14T04:28:40.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:40.331 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:40.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:40.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:40.388 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 4, 5, 2] out_osds: [1, 7, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:28:40.388 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:28:40.389 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_9 2024-09-14T04:28:40.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:28:40.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:40.480 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:40.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:40.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 70 16 112714 112698 100.611 84.875 0.00877799 0.0099346 2024-09-14T04:28:40.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:40.615 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:40.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:28:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:28:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:28:40.717 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:40.733 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:40.767 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:40.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:40.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:40.913 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:40.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:40.970 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:40 smithi114 ceph-mon[106969]: pgmap v1526: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.5 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 42 MiB/s wr, 678 op/s; 3157/317490 objects degraded (0.994%); 4435/317490 objects misplaced (1.397%); 7.0 MiB/s, 112 objects/s recovering 2024-09-14T04:28:40.970 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:40 smithi114 ceph-mon[103029]: pgmap v1526: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.5 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 42 MiB/s wr, 678 op/s; 3157/317490 objects degraded (0.994%); 4435/317490 objects misplaced (1.397%); 7.0 MiB/s, 112 objects/s recovering 2024-09-14T04:28:41.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:41.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:41.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:41.127 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:40 smithi160 ceph-mon[88407]: pgmap v1526: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 6.5 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 42 MiB/s wr, 678 op/s; 3157/317490 objects degraded (0.994%); 4435/317490 objects misplaced (1.397%); 7.0 MiB/s, 112 objects/s recovering 2024-09-14T04:28:41.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:41.195 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:41.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:41.297 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:41.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:41.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:41.435 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:41.475 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:41.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:41.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 71 16 114047 114031 100.367 83.3125 0.00612457 0.00995931 2024-09-14T04:28:41.562 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:28:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:28:41] "GET /metrics HTTP/1.1" 200 36576 "" "Prometheus/2.51.0" 2024-09-14T04:28:41.577 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:41.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:41.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:41.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:28:41.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:41.759 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:41.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:41.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:41.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:41 smithi114 ceph-mon[106969]: pgmap v1527: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 72 MiB/s wr, 1.16k op/s; 3157/334098 objects degraded (0.945%); 4435/334098 objects misplaced (1.327%); 5.9 MiB/s, 94 objects/s recovering 2024-09-14T04:28:41.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:41 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3157/334098 objects degraded (0.945%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:41.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:41 smithi114 ceph-mon[103029]: pgmap v1527: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 72 MiB/s wr, 1.16k op/s; 3157/334098 objects degraded (0.945%); 4435/334098 objects misplaced (1.327%); 5.9 MiB/s, 94 objects/s recovering 2024-09-14T04:28:41.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:41 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3157/334098 objects degraded (0.945%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:41.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:41.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:41.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:42.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:42.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:42.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:41 smithi160 ceph-mon[88407]: pgmap v1527: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 72 MiB/s wr, 1.16k op/s; 3157/334098 objects degraded (0.945%); 4435/334098 objects misplaced (1.327%); 5.9 MiB/s, 94 objects/s recovering 2024-09-14T04:28:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:41 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3157/334098 objects degraded (0.945%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:42.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:42.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:42.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:42.318 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:42.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:42.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:42.464 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:42.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:42.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 72 16 115379 115363 100.129 83.25 0.00994104 0.00998309 2024-09-14T04:28:42.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:42.603 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:42.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:42.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:42.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:42.729 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:28:42.729 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:28:42.739 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:42.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:42.759 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:28:42.759 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:28:42.788 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:28:42.788 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:28:42.817 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:28:42.817 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:28:42.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:42.848 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:28:42.849 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:28:42.874 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:42.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:42.881 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:28:42.882 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:28:42.906 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:28:42.906 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:28:42.937 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:28:42.937 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:28:42.937 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3280841405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:42.938 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3280841405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:42.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:43.018 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:43.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:43.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:43.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:43.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3280841405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:43.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:43.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:43.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:43.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:43.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:43.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:43.441 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:43.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:43.454 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:28:43.454 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:28:43.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:43.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 73 16 116700 116684 99.8883 82.5625 0.0131912 0.0100068 2024-09-14T04:28:43.580 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:43.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:43.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:43.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:43.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:43.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:43.826 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:43.856 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:43.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:43.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:43 smithi114 ceph-mon[103029]: pgmap v1528: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 68 MiB/s wr, 1.09k op/s; 3157/334098 objects degraded (0.945%); 4435/334098 objects misplaced (1.327%); 5.6 MiB/s, 90 objects/s recovering 2024-09-14T04:28:43.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:43 smithi114 ceph-mon[106969]: pgmap v1528: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 68 MiB/s wr, 1.09k op/s; 3157/334098 objects degraded (0.945%); 4435/334098 objects misplaced (1.327%); 5.6 MiB/s, 90 objects/s recovering 2024-09-14T04:28:43.958 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:43.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:43.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:44.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:44.132 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:44.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:44.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:43 smithi160 ceph-mon[88407]: pgmap v1528: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 6.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 68 MiB/s wr, 1.09k op/s; 3157/334098 objects degraded (0.945%); 4435/334098 objects misplaced (1.327%); 5.6 MiB/s, 90 objects/s recovering 2024-09-14T04:28:44.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:44.267 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:44.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:44.369 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:44.403 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:44.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:44.505 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:44.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:44.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:44.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 74 16 118039 118023 99.6693 83.6875 0.0175945 0.0100287 2024-09-14T04:28:44.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:44.655 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:44.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:44.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:44.760 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:44.763 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:44.789 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:44.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:44.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:44.917 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:44.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:45.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:45.049 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:45.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:45.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:45.182 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:45.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:45.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:45.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:45.320 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:45.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:45.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:45.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:45.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:45.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 75 16 119280 119264 99.3745 77.5625 0.0192492 0.0100589 2024-09-14T04:28:45.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:45.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:45.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:45.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:45.688 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:45.718 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:45.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:45.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:45.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:45.854 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:45.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:45.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:45.988 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:45.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:46.016 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-14T04:28:46.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:46.135 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:46.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:46.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:46.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:46.273 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:46.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:46.374 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:46.415 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:46.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:46.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:46.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:46.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:46.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:46.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 76 16 120619 120603 99.168 83.6875 0.00957512 0.0100797 2024-09-14T04:28:46.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:46 smithi114 ceph-mon[106969]: pgmap v1529: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.2 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 84 MiB/s wr, 1.35k op/s; 3157/354042 objects degraded (0.892%); 4128/354042 objects misplaced (1.166%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T04:28:46.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:46 smithi114 ceph-mon[103029]: pgmap v1529: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.2 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 84 MiB/s wr, 1.35k op/s; 3157/354042 objects degraded (0.892%); 4128/354042 objects misplaced (1.166%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T04:28:46.649 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:46.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:46 smithi160 ceph-mon[88407]: pgmap v1529: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.2 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 84 MiB/s wr, 1.35k op/s; 3157/354042 objects degraded (0.892%); 4128/354042 objects misplaced (1.166%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T04:28:46.678 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:46.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:46.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:46.780 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:46.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:28:46.808 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:46.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:46.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:46.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:46.948 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:46.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:47.050 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:47.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:47.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:47.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:47.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:47.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:47.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:47.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:47.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:47.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:47.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:47.502 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:47.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:47.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:47.549 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 77 16 121997 121981 98.9985 86.125 0.0119515 0.0100969 2024-09-14T04:28:47.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:47.640 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:47.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:47.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:47.782 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:47.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:47.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:47.925 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:47.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:48.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:48.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:48.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:48.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:48.193 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:48.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:48.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:48.295 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:48.329 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:48.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:48.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:48.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:48 smithi114 ceph-mon[103029]: pgmap v1530: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.2 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 80 MiB/s wr, 1.28k op/s; 3157/354042 objects degraded (0.892%); 4128/354042 objects misplaced (1.166%); 3.6 MiB/s, 58 objects/s recovering 2024-09-14T04:28:48.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:48 smithi114 ceph-mon[106969]: pgmap v1530: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.2 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 80 MiB/s wr, 1.28k op/s; 3157/354042 objects degraded (0.892%); 4128/354042 objects misplaced (1.166%); 3.6 MiB/s, 58 objects/s recovering 2024-09-14T04:28:48.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:48.470 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:48.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:48.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 78 16 123352 123336 98.815 84.6875 0.0192717 0.0101154 2024-09-14T04:28:48.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:48.606 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:48.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:48 smithi160 ceph-mon[88407]: pgmap v1530: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.2 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 80 MiB/s wr, 1.28k op/s; 3157/354042 objects degraded (0.892%); 4128/354042 objects misplaced (1.166%); 3.6 MiB/s, 58 objects/s recovering 2024-09-14T04:28:48.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:48.747 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:48.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:48.848 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:48.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:48.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:48.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:49.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:49.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:49.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:49.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:49.270 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:49.304 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:49.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:49.406 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:49.442 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:49.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:49.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:49.548 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:49 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3157/357645 objects degraded (0.883%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:49.548 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:49 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3157/357645 objects degraded (0.883%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:49.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 79 16 124688 124672 98.6209 83.5 0.00630458 0.0101353 2024-09-14T04:28:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:49.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:49 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3157/357645 objects degraded (0.883%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:49.689 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:49.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:49.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:49.746 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:28:49.750 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:28:49.761 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:49.789 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:28:49.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:49.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:49.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:49.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:49.993 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:49.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:50.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:50.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:50.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:50.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:50.261 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:50.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:50.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:50.339 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:28:50.358 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:50 smithi114 ceph-mon[103029]: pgmap v1531: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 7.3 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 77 MiB/s wr, 1.22k op/s; 3157/357645 objects degraded (0.883%); 3738/357645 objects misplaced (1.045%); 5.2 MiB/s, 83 objects/s recovering 2024-09-14T04:28:50.358 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:50 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2225595946' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:28:50.358 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:50 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:28:50.358 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:50 smithi114 ceph-mon[106969]: pgmap v1531: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 7.3 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 77 MiB/s wr, 1.22k op/s; 3157/357645 objects degraded (0.883%); 3738/357645 objects misplaced (1.045%); 5.2 MiB/s, 83 objects/s recovering 2024-09-14T04:28:50.358 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:50 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2225595946' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:28:50.358 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:50 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:28:50.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:50.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:50.401 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:50.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:50.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:50.531 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:50.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:50.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:2024-09-14T04:28:50.550094+0000 min lat: 0.00118 max lat: 2.12324 avg lat: 0.0101581 2024-09-14T04:28:50.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-14T04:28:50.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 80 16 125981 125965 98.3982 80.8125 0.018319 0.0101581 2024-09-14T04:28:50.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:50 smithi160 ceph-mon[88407]: pgmap v1531: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 7.3 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 77 MiB/s wr, 1.22k op/s; 3157/357645 objects degraded (0.883%); 3738/357645 objects misplaced (1.045%); 5.2 MiB/s, 83 objects/s recovering 2024-09-14T04:28:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:50 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2225595946' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:28:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:50 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:28:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:28:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:28:50.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:28:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:50.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:50.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:50.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:50.810 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:50.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:50.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:50.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:50.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:50.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:51.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:51.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:51.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:51.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:51.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:51.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:51.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:51.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:51.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:51.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:51.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:51.499 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:51.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:51.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 81 16 127342 127326 98.2335 85.0625 0.00935115 0.0101756 2024-09-14T04:28:51.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:51.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:51 smithi114 ceph-mon[103029]: from='client.47234 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:28:51.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:51 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:28:51.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:51 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:28:51.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:51 smithi114 ceph-mon[103029]: osdmap e407: 8 total, 8 up, 5 in 2024-09-14T04:28:51.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:28:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:28:51] "GET /metrics HTTP/1.1" 200 36576 "" "Prometheus/2.51.0" 2024-09-14T04:28:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:51 smithi114 ceph-mon[106969]: from='client.47234 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:28:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:51 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:28:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:51 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:28:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:51 smithi114 ceph-mon[106969]: osdmap e407: 8 total, 8 up, 5 in 2024-09-14T04:28:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:51.633 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:51.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:51 smithi160 ceph-mon[88407]: from='client.47234 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:28:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:51 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:28:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:51 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:28:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:51 smithi160 ceph-mon[88407]: osdmap e407: 8 total, 8 up, 5 in 2024-09-14T04:28:51.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:51.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:51.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:51.879 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:51.907 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:51.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:52.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:52.038 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:52.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:52.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:52.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:52.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:52.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:52.301 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:52.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:52.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:52.423 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1531,"stamp":"2024-09-14T04:28:49.209139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7811733453,"num_objects":119215,"num_object_clones":0,"num_object_copies":357645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3157,"num_objects_misplaced":3738,"num_objects_unfound":0,"num_objects_dirty":119215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82222,"num_write_kb":5258175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3808,"num_bytes_recovered":245898015,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146763,"ondisk_log_size":146763,"up":327,"acting":325,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":331,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":29398772,"kb_used_data":23213196,"kb_used_omap":20,"kb_used_meta":6185515,"kb_avail":439326988,"statfs":{"total":479975178240,"available":449870835712,"internally_reserved":0,"allocated":23770312704,"data_stored":23760535792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20935,"internal_metadata":6333967929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":963117056,"num_objects":14696,"num_object_clones":0,"num_object_copies":44088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1003,"num_objects_unfound":0,"num_objects_dirty":14696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14696,"num_write_kb":940544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":65732608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003489"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":648,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678122+0000","last_change":"2024-09-14T04:25:42.350690+0000","last_active":"2024-09-14T04:28:36.678122+0000","last_peered":"2024-09-14T04:28:36.678122+0000","last_clean":"2024-09-14T04:28:36.678122+0000","last_became_active":"2024-09-14T04:25:42.350363+0000","last_became_peered":"2024-09-14T04:25:42.350363+0000","last_unstale":"2024-09-14T04:28:36.678122+0000","last_undegraded":"2024-09-14T04:28:36.678122+0000","last_fullsized":"2024-09-14T04:28:36.678122+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:11:38.039845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":109,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.675512+0000","last_change":"2024-09-14T04:25:42.287293+0000","last_active":"2024-09-14T04:28:36.675512+0000","last_peered":"2024-09-14T04:28:36.675512+0000","last_clean":"2024-09-14T04:28:36.675512+0000","last_became_active":"2024-09-14T04:25:42.237213+0000","last_became_peered":"2024-09-14T04:25:42.237213+0000","last_unstale":"2024-09-14T04:28:36.675512+0000","last_undegraded":"2024-09-14T04:28:36.675512+0000","last_fullsized":"2024-09-14T04:28:36.675512+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:49:14.895808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":111,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723851+0000","last_change":"2024-09-14T04:27:55.395641+0000","last_active":"2024-09-14T04:28:31.723851+0000","last_peered":"2024-09-14T04:28:31.723851+0000","last_clean":"2024-09-14T04:28:31.723851+0000","last_became_active":"2024-09-14T04:27:55.395369+0000","last_became_peered":"2024-09-14T04:27:55.395369+0000","last_unstale":"2024-09-14T04:28:31.723851+0000","last_undegraded":"2024-09-14T04:28:31.723851+0000","last_fullsized":"2024-09-14T04:28:31.723851+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:09:44.628733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":473,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678639+0000","last_change":"2024-09-14T04:26:18.735795+0000","last_active":"2024-09-14T04:28:36.678639+0000","last_peered":"2024-09-14T04:28:36.678639+0000","last_clean":"2024-09-14T04:28:36.678639+0000","last_became_active":"2024-09-14T04:26:18.735453+0000","last_became_peered":"2024-09-14T04:26:18.735453+0000","last_unstale":"2024-09-14T04:28:36.678639+0000","last_undegraded":"2024-09-14T04:28:36.678639+0000","last_fullsized":"2024-09-14T04:28:36.678639+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:11:19.345201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.722948+0000","last_change":"2024-09-14T04:27:55.402319+0000","last_active":"2024-09-14T04:28:31.722948+0000","last_peered":"2024-09-14T04:28:31.722948+0000","last_clean":"2024-09-14T04:28:31.722948+0000","last_became_active":"2024-09-14T04:27:55.401998+0000","last_became_peered":"2024-09-14T04:27:55.401998+0000","last_unstale":"2024-09-14T04:28:31.722948+0000","last_undegraded":"2024-09-14T04:28:31.722948+0000","last_fullsized":"2024-09-14T04:28:31.722948+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:37.324383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":379,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.727578+0000","last_change":"2024-09-14T04:26:15.737623+0000","last_active":"2024-09-14T04:28:31.727578+0000","last_peered":"2024-09-14T04:28:31.727578+0000","last_clean":"2024-09-14T04:28:31.727578+0000","last_became_active":"2024-09-14T04:26:15.731253+0000","last_became_peered":"2024-09-14T04:26:15.731253+0000","last_unstale":"2024-09-14T04:28:31.727578+0000","last_undegraded":"2024-09-14T04:28:31.727578+0000","last_fullsized":"2024-09-14T04:28:31.727578+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:25:47.213793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":461,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677787+0000","last_change":"2024-09-14T04:25:42.334456+0000","last_active":"2024-09-14T04:28:36.677787+0000","last_peered":"2024-09-14T04:28:36.677787+0000","last_clean":"2024-09-14T04:28:36.677787+0000","last_became_active":"2024-09-14T04:25:42.334177+0000","last_became_peered":"2024-09-14T04:25:42.334177+0000","last_unstale":"2024-09-14T04:28:36.677787+0000","last_undegraded":"2024-09-14T04:28:36.677787+0000","last_fullsized":"2024-09-14T04:28:36.677787+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:12:07.154369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":440,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.726434+0000","last_change":"2024-09-14T04:26:15.744615+0000","last_active":"2024-09-14T04:28:31.726434+0000","last_peered":"2024-09-14T04:28:31.726434+0000","last_clean":"2024-09-14T04:28:31.726434+0000","last_became_active":"2024-09-14T04:26:15.726296+0000","last_became_peered":"2024-09-14T04:26:15.726296+0000","last_unstale":"2024-09-14T04:28:31.726434+0000","last_undegraded":"2024-09-14T04:28:31.726434+0000","last_fullsized":"2024-09-14T04:28:31.726434+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:46:10.656996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.2","version":"406'5972","reported_seq":6102,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:48.928784+0000","last_change":"2024-09-14T04:28:03.205932+0000","last_active":"2024-09-14T04:28:48.928784+0000","last_peered":"2024-09-14T04:28:48.928784+0000","last_clean":"2024-09-14T04:28:48.928784+0000","last_became_active":"2024-09-14T04:28:02.430518+0000","last_became_peered":"2024-09-14T04:28:02.430518+0000","last_unstale":"2024-09-14T04:28:48.928784+0000","last_undegraded":"2024-09-14T04:28:48.928784+0000","last_fullsized":"2024-09-14T04:28:48.928784+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4166","last_scrub_stamp":"2024-09-14T04:28:03.205488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:03.205488+0000","objects_scrubbed":2056,"log_size":5972,"log_dups_size":0,"ondisk_log_size":5972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:28.934195+0000","scrub_duration":0.69549441300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254279680,"num_objects":3880,"num_object_clones":0,"num_object_copies":11640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1824,"num_write_kb":116736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":405,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.724673+0000","last_change":"2024-09-14T04:26:15.750280+0000","last_active":"2024-09-14T04:28:31.724673+0000","last_peered":"2024-09-14T04:28:31.724673+0000","last_clean":"2024-09-14T04:28:31.724673+0000","last_became_active":"2024-09-14T04:26:15.736153+0000","last_became_peered":"2024-09-14T04:26:15.736153+0000","last_unstale":"2024-09-14T04:28:31.724673+0000","last_undegraded":"2024-09-14T04:28:31.724673+0000","last_fullsized":"2024-09-14T04:28:31.724673+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:00:25.087633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":101,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.722687+0000","last_change":"2024-09-14T04:23:09.282698+0000","last_active":"2024-09-14T04:28:31.722687+0000","last_peered":"2024-09-14T04:28:31.722687+0000","last_clean":"2024-09-14T04:28:31.722687+0000","last_became_active":"2024-09-14T04:23:09.282181+0000","last_became_peered":"2024-09-14T04:23:09.282181+0000","last_unstale":"2024-09-14T04:28:31.722687+0000","last_undegraded":"2024-09-14T04:28:31.722687+0000","last_fullsized":"2024-09-14T04:28:31.722687+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:18:12.880523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"406'5734","reported_seq":5832,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.736226+0000","last_change":"2024-09-14T04:28:05.894600+0000","last_active":"2024-09-14T04:28:44.736226+0000","last_peered":"2024-09-14T04:28:44.736226+0000","last_clean":"2024-09-14T04:28:44.736226+0000","last_became_active":"2024-09-14T04:28:02.411993+0000","last_became_peered":"2024-09-14T04:28:02.411993+0000","last_unstale":"2024-09-14T04:28:44.736226+0000","last_undegraded":"2024-09-14T04:28:44.736226+0000","last_fullsized":"2024-09-14T04:28:44.736226+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'4263","last_scrub_stamp":"2024-09-14T04:28:05.894214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:05.894214+0000","objects_scrubbed":2271,"log_size":5734,"log_dups_size":0,"ondisk_log_size":5734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:29:12.246655+0000","scrub_duration":373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245366784,"num_objects":3744,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":94272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":82,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723299+0000","last_change":"2024-09-14T04:26:15.755751+0000","last_active":"2024-09-14T04:28:31.723299+0000","last_peered":"2024-09-14T04:28:31.723299+0000","last_clean":"2024-09-14T04:28:31.723299+0000","last_became_active":"2024-09-14T04:26:15.745572+0000","last_became_peered":"2024-09-14T04:26:15.745572+0000","last_unstale":"2024-09-14T04:28:31.723299+0000","last_undegraded":"2024-09-14T04:28:31.723299+0000","last_fullsized":"2024-09-14T04:28:31.723299+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:54:48.838252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":430,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678302+0000","last_change":"2024-09-14T04:24:47.613225+0000","last_active":"2024-09-14T04:28:36.678302+0000","last_peered":"2024-09-14T04:28:36.678302+0000","last_clean":"2024-09-14T04:28:36.678302+0000","last_became_active":"2024-09-14T04:24:47.612805+0000","last_became_peered":"2024-09-14T04:24:47.612805+0000","last_unstale":"2024-09-14T04:28:36.678302+0000","last_undegraded":"2024-09-14T04:28:36.678302+0000","last_fullsized":"2024-09-14T04:28:36.678302+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:35.992716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"406'5727","reported_seq":5826,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.583926+0000","last_change":"2024-09-14T04:28:06.826238+0000","last_active":"2024-09-14T04:28:44.583926+0000","last_peered":"2024-09-14T04:28:44.583926+0000","last_clean":"2024-09-14T04:28:44.583926+0000","last_became_active":"2024-09-14T04:28:02.429720+0000","last_became_peered":"2024-09-14T04:28:02.429720+0000","last_unstale":"2024-09-14T04:28:44.583926+0000","last_undegraded":"2024-09-14T04:28:44.583926+0000","last_fullsized":"2024-09-14T04:28:44.583926+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'4245","last_scrub_stamp":"2024-09-14T04:28:06.826118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:06.826118+0000","objects_scrubbed":2276,"log_size":5727,"log_dups_size":0,"ondisk_log_size":5727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:07:37.806780+0000","scrub_duration":273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246284288,"num_objects":3758,"num_object_clones":0,"num_object_copies":11274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1482,"num_write_kb":94848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":109,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723678+0000","last_change":"2024-09-14T04:25:03.878325+0000","last_active":"2024-09-14T04:28:31.723678+0000","last_peered":"2024-09-14T04:28:31.723678+0000","last_clean":"2024-09-14T04:28:31.723678+0000","last_became_active":"2024-09-14T04:25:03.861286+0000","last_became_peered":"2024-09-14T04:25:03.861286+0000","last_unstale":"2024-09-14T04:28:31.723678+0000","last_undegraded":"2024-09-14T04:28:31.723678+0000","last_fullsized":"2024-09-14T04:28:31.723678+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:57:29.738051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":421,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.726352+0000","last_change":"2024-09-14T04:26:15.726841+0000","last_active":"2024-09-14T04:28:31.726352+0000","last_peered":"2024-09-14T04:28:31.726352+0000","last_clean":"2024-09-14T04:28:31.726352+0000","last_became_active":"2024-09-14T04:26:15.725967+0000","last_became_peered":"2024-09-14T04:26:15.725967+0000","last_unstale":"2024-09-14T04:28:31.726352+0000","last_undegraded":"2024-09-14T04:28:31.726352+0000","last_fullsized":"2024-09-14T04:28:31.726352+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:30:20.743135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1","version":"406'5668","reported_seq":5761,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.728845+0000","last_change":"2024-09-14T04:28:02.952869+0000","last_active":"2024-09-14T04:28:44.728845+0000","last_peered":"2024-09-14T04:28:44.728845+0000","last_clean":"2024-09-14T04:28:44.728845+0000","last_became_active":"2024-09-14T04:28:02.390261+0000","last_became_peered":"2024-09-14T04:28:02.390261+0000","last_unstale":"2024-09-14T04:28:44.728845+0000","last_undegraded":"2024-09-14T04:28:44.728845+0000","last_fullsized":"2024-09-14T04:28:44.728845+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4086","last_scrub_stamp":"2024-09-14T04:28:02.952661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:02.952661+0000","objects_scrubbed":2017,"log_size":5668,"log_dups_size":0,"ondisk_log_size":5668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:54:25.165165+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236322816,"num_objects":3606,"num_object_clones":0,"num_object_copies":10818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1589,"num_write_kb":101696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":91,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723047+0000","last_change":"2024-09-14T04:22:08.468750+0000","last_active":"2024-09-14T04:28:31.723047+0000","last_peered":"2024-09-14T04:28:31.723047+0000","last_clean":"2024-09-14T04:28:31.723047+0000","last_became_active":"2024-09-14T04:22:08.468217+0000","last_became_peered":"2024-09-14T04:22:08.468217+0000","last_unstale":"2024-09-14T04:28:31.723047+0000","last_undegraded":"2024-09-14T04:28:31.723047+0000","last_fullsized":"2024-09-14T04:28:31.723047+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:06.709182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":136,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678358+0000","last_change":"2024-09-14T04:26:15.732663+0000","last_active":"2024-09-14T04:28:36.678358+0000","last_peered":"2024-09-14T04:28:36.678358+0000","last_clean":"2024-09-14T04:28:36.678358+0000","last_became_active":"2024-09-14T04:26:15.731910+0000","last_became_peered":"2024-09-14T04:26:15.731910+0000","last_unstale":"2024-09-14T04:28:36.678358+0000","last_undegraded":"2024-09-14T04:28:36.678358+0000","last_fullsized":"2024-09-14T04:28:36.678358+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:44.768223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.6","version":"406'5772","reported_seq":5864,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.566353+0000","last_change":"2024-09-14T04:28:03.114846+0000","last_active":"2024-09-14T04:28:44.566353+0000","last_peered":"2024-09-14T04:28:44.566353+0000","last_clean":"2024-09-14T04:28:44.566353+0000","last_became_active":"2024-09-14T04:28:02.388695+0000","last_became_peered":"2024-09-14T04:28:02.388695+0000","last_unstale":"2024-09-14T04:28:44.566353+0000","last_undegraded":"2024-09-14T04:28:44.566353+0000","last_fullsized":"2024-09-14T04:28:44.566353+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4203","last_scrub_stamp":"2024-09-14T04:28:03.114713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:03.114713+0000","objects_scrubbed":2118,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:15:47.136816+0000","scrub_duration":513,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242548736,"num_objects":3701,"num_object_clones":0,"num_object_copies":11103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1583,"num_write_kb":101312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'100","reported_seq":67,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678155+0000","last_change":"2024-09-14T04:27:43.391069+0000","last_active":"2024-09-14T04:28:36.678155+0000","last_peered":"2024-09-14T04:28:36.678155+0000","last_clean":"2024-09-14T04:28:36.678155+0000","last_became_active":"2024-09-14T04:27:12.689443+0000","last_became_peered":"2024-09-14T04:27:12.689443+0000","last_unstale":"2024-09-14T04:28:36.678155+0000","last_undegraded":"2024-09-14T04:28:36.678155+0000","last_fullsized":"2024-09-14T04:28:36.678155+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:43.390999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:43.390999+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:55:51.689081+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":689,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678542+0000","last_change":"2024-09-14T04:26:15.751894+0000","last_active":"2024-09-14T04:28:36.678542+0000","last_peered":"2024-09-14T04:28:36.678542+0000","last_clean":"2024-09-14T04:28:36.678542+0000","last_became_active":"2024-09-14T04:26:15.731434+0000","last_became_peered":"2024-09-14T04:26:15.731434+0000","last_unstale":"2024-09-14T04:28:36.678542+0000","last_undegraded":"2024-09-14T04:28:36.678542+0000","last_fullsized":"2024-09-14T04:28:36.678542+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:17.854732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":514,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678236+0000","last_change":"2024-09-14T04:17:48.083765+0000","last_active":"2024-09-14T04:28:36.678236+0000","last_peered":"2024-09-14T04:28:36.678236+0000","last_clean":"2024-09-14T04:28:36.678236+0000","last_became_active":"2024-09-14T04:17:48.083385+0000","last_became_peered":"2024-09-14T04:17:48.083385+0000","last_unstale":"2024-09-14T04:28:36.678236+0000","last_undegraded":"2024-09-14T04:28:36.678236+0000","last_fullsized":"2024-09-14T04:28:36.678236+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:13:49.268399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.7","version":"406'6380","reported_seq":6518,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.225036+0000","last_change":"2024-09-14T04:28:07.388643+0000","last_active":"2024-09-14T04:28:44.225036+0000","last_peered":"2024-09-14T04:28:44.225036+0000","last_clean":"2024-09-14T04:28:44.225036+0000","last_became_active":"2024-09-14T04:28:02.428617+0000","last_became_peered":"2024-09-14T04:28:02.428617+0000","last_unstale":"2024-09-14T04:28:44.225036+0000","last_undegraded":"2024-09-14T04:28:44.225036+0000","last_fullsized":"2024-09-14T04:28:44.225036+0000","mapping_epoch":385,"log_start":"389'139","ondisk_log_start":"389'139","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'4926","last_scrub_stamp":"2024-09-14T04:28:07.388588+0000","last_deep_scrub":"398'4926","last_deep_scrub_stamp":"2024-09-14T04:28:07.388588+0000","last_clean_scrub_stamp":"2024-09-14T04:28:07.388588+0000","objects_scrubbed":2217,"log_size":6241,"log_dups_size":0,"ondisk_log_size":6241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:39:05.724015+0000","scrub_duration":2.2206136920000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240713728,"num_objects":3673,"num_object_clones":0,"num_object_copies":11019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1456,"num_write_kb":93184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"76'100","reported_seq":47,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.722217+0000","last_change":"2024-09-14T04:27:46.144096+0000","last_active":"2024-09-14T04:28:31.722217+0000","last_peered":"2024-09-14T04:28:31.722217+0000","last_clean":"2024-09-14T04:28:31.722217+0000","last_became_active":"2024-09-14T04:27:12.688140+0000","last_became_peered":"2024-09-14T04:27:12.688140+0000","last_unstale":"2024-09-14T04:28:31.722217+0000","last_undegraded":"2024-09-14T04:28:31.722217+0000","last_fullsized":"2024-09-14T04:28:31.722217+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:46.144003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:46.144003+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:00:07.652536+0000","scrub_duration":0.0071911659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":402,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.726439+0000","last_change":"2024-09-14T04:24:47.615423+0000","last_active":"2024-09-14T04:28:31.726439+0000","last_peered":"2024-09-14T04:28:31.726439+0000","last_clean":"2024-09-14T04:28:31.726439+0000","last_became_active":"2024-09-14T04:24:47.609944+0000","last_became_peered":"2024-09-14T04:24:47.609944+0000","last_unstale":"2024-09-14T04:28:31.726439+0000","last_undegraded":"2024-09-14T04:28:31.726439+0000","last_fullsized":"2024-09-14T04:28:31.726439+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:30:03.475490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":103,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.674590+0000","last_change":"2024-09-14T04:25:42.247602+0000","last_active":"2024-09-14T04:28:36.674590+0000","last_peered":"2024-09-14T04:28:36.674590+0000","last_clean":"2024-09-14T04:28:36.674590+0000","last_became_active":"2024-09-14T04:25:42.247304+0000","last_became_peered":"2024-09-14T04:25:42.247304+0000","last_unstale":"2024-09-14T04:28:36.674590+0000","last_undegraded":"2024-09-14T04:28:36.674590+0000","last_fullsized":"2024-09-14T04:28:36.674590+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:18:17.260700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"406'6348","reported_seq":6484,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:48.995667+0000","last_change":"2024-09-14T04:28:06.867085+0000","last_active":"2024-09-14T04:28:48.995667+0000","last_peered":"2024-09-14T04:28:48.995667+0000","last_clean":"2024-09-14T04:28:48.995667+0000","last_became_active":"2024-09-14T04:28:02.414511+0000","last_became_peered":"2024-09-14T04:28:02.414511+0000","last_unstale":"2024-09-14T04:28:48.995667+0000","last_undegraded":"2024-09-14T04:28:48.995667+0000","last_fullsized":"2024-09-14T04:28:48.995667+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'4718","last_scrub_stamp":"2024-09-14T04:28:06.866991+0000","last_deep_scrub":"398'4718","last_deep_scrub_stamp":"2024-09-14T04:28:06.866991+0000","last_clean_scrub_stamp":"2024-09-14T04:28:06.866991+0000","objects_scrubbed":2158,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:10:32.942361+0000","scrub_duration":3.3310999840000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249298944,"num_objects":3804,"num_object_clones":0,"num_object_copies":11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1646,"num_write_kb":105344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":246,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.676067+0000","last_change":"2024-09-14T04:25:53.296317+0000","last_active":"2024-09-14T04:28:36.676067+0000","last_peered":"2024-09-14T04:28:36.676067+0000","last_clean":"2024-09-14T04:28:36.676067+0000","last_became_active":"2024-09-14T04:25:53.296039+0000","last_became_peered":"2024-09-14T04:25:53.296039+0000","last_unstale":"2024-09-14T04:28:36.676067+0000","last_undegraded":"2024-09-14T04:28:36.676067+0000","last_fullsized":"2024-09-14T04:28:36.676067+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:26:31.287496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":284,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.727217+0000","last_change":"2024-09-14T04:24:47.616564+0000","last_active":"2024-09-14T04:28:31.727217+0000","last_peered":"2024-09-14T04:28:31.727217+0000","last_clean":"2024-09-14T04:28:31.727217+0000","last_became_active":"2024-09-14T04:24:47.616196+0000","last_became_peered":"2024-09-14T04:24:47.616196+0000","last_unstale":"2024-09-14T04:28:31.727217+0000","last_undegraded":"2024-09-14T04:28:31.727217+0000","last_fullsized":"2024-09-14T04:28:31.727217+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:38:56.531650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"406'5805","reported_seq":5901,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.755842+0000","last_change":"2024-09-14T04:28:05.093393+0000","last_active":"2024-09-14T04:28:44.755842+0000","last_peered":"2024-09-14T04:28:44.755842+0000","last_clean":"2024-09-14T04:28:44.755842+0000","last_became_active":"2024-09-14T04:28:02.411968+0000","last_became_peered":"2024-09-14T04:28:02.411968+0000","last_unstale":"2024-09-14T04:28:44.755842+0000","last_undegraded":"2024-09-14T04:28:44.755842+0000","last_fullsized":"2024-09-14T04:28:44.755842+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4326","last_scrub_stamp":"2024-09-14T04:28:05.093275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:05.093275+0000","objects_scrubbed":2220,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:40.929397+0000","scrub_duration":584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243138560,"num_objects":3710,"num_object_clones":0,"num_object_copies":11130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1490,"num_write_kb":95360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'100","reported_seq":42,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.722771+0000","last_change":"2024-09-14T04:27:35.272869+0000","last_active":"2024-09-14T04:28:31.722771+0000","last_peered":"2024-09-14T04:28:31.722771+0000","last_clean":"2024-09-14T04:28:31.722771+0000","last_became_active":"2024-09-14T04:27:12.688026+0000","last_became_peered":"2024-09-14T04:27:12.688026+0000","last_unstale":"2024-09-14T04:28:31.722771+0000","last_undegraded":"2024-09-14T04:28:31.722771+0000","last_fullsized":"2024-09-14T04:28:31.722771+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:35.272785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:35.272785+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:00:40.802976+0000","scrub_duration":0.0029595770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":455,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677283+0000","last_change":"2024-09-14T04:25:03.848027+0000","last_active":"2024-09-14T04:28:36.677283+0000","last_peered":"2024-09-14T04:28:36.677283+0000","last_clean":"2024-09-14T04:28:36.677283+0000","last_became_active":"2024-09-14T04:25:03.847621+0000","last_became_peered":"2024-09-14T04:25:03.847621+0000","last_unstale":"2024-09-14T04:28:36.677283+0000","last_undegraded":"2024-09-14T04:28:36.677283+0000","last_fullsized":"2024-09-14T04:28:36.677283+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:52.585026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":490,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677896+0000","last_change":"2024-09-14T04:25:03.856805+0000","last_active":"2024-09-14T04:28:36.677896+0000","last_peered":"2024-09-14T04:28:36.677896+0000","last_clean":"2024-09-14T04:28:36.677896+0000","last_became_active":"2024-09-14T04:25:03.856459+0000","last_became_peered":"2024-09-14T04:25:03.856459+0000","last_unstale":"2024-09-14T04:28:36.677896+0000","last_undegraded":"2024-09-14T04:28:36.677896+0000","last_fullsized":"2024-09-14T04:28:36.677896+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:17:41.568556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.f","version":"406'7373","reported_seq":7431,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.758769+0000","last_change":"2024-09-14T04:27:15.713172+0000","last_active":"2024-09-14T04:28:44.758769+0000","last_peered":"2024-09-14T04:28:44.758769+0000","last_clean":"2024-09-14T04:28:44.758769+0000","last_became_active":"2024-09-14T04:27:15.712862+0000","last_became_peered":"2024-09-14T04:27:15.712862+0000","last_unstale":"2024-09-14T04:28:44.758769+0000","last_undegraded":"2024-09-14T04:28:44.758769+0000","last_fullsized":"2024-09-14T04:28:44.758769+0000","mapping_epoch":385,"log_start":"390'3200","ondisk_log_start":"390'3200","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:15:57.744217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483196928,"num_objects":7373,"num_object_clones":0,"num_object_copies":22119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":471872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":279,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.726409+0000","last_change":"2024-09-14T04:27:34.456942+0000","last_active":"2024-09-14T04:28:31.726409+0000","last_peered":"2024-09-14T04:28:31.726409+0000","last_clean":"2024-09-14T04:28:31.726409+0000","last_became_active":"2024-09-14T04:26:15.726724+0000","last_became_peered":"2024-09-14T04:26:15.726724+0000","last_unstale":"2024-09-14T04:28:31.726409+0000","last_undegraded":"2024-09-14T04:28:31.726409+0000","last_fullsized":"2024-09-14T04:28:31.726409+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:34.456083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:34.456083+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:01:39.830454+0000","scrub_duration":0.0045693399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":131,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723264+0000","last_change":"2024-09-14T04:25:52.311202+0000","last_active":"2024-09-14T04:28:31.723264+0000","last_peered":"2024-09-14T04:28:31.723264+0000","last_clean":"2024-09-14T04:28:31.723264+0000","last_became_active":"2024-09-14T04:25:42.253580+0000","last_became_peered":"2024-09-14T04:25:42.253580+0000","last_unstale":"2024-09-14T04:28:31.723264+0000","last_undegraded":"2024-09-14T04:28:31.723264+0000","last_fullsized":"2024-09-14T04:28:31.723264+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:59:26.500812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":456,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678498+0000","last_change":"2024-09-14T04:22:08.472386+0000","last_active":"2024-09-14T04:28:36.678498+0000","last_peered":"2024-09-14T04:28:36.678498+0000","last_clean":"2024-09-14T04:28:36.678498+0000","last_became_active":"2024-09-14T04:22:08.472100+0000","last_became_peered":"2024-09-14T04:22:08.472100+0000","last_unstale":"2024-09-14T04:28:36.678498+0000","last_undegraded":"2024-09-14T04:28:36.678498+0000","last_fullsized":"2024-09-14T04:28:36.678498+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:36:03.114984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":394,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.727074+0000","last_change":"2024-09-14T04:26:15.727747+0000","last_active":"2024-09-14T04:28:31.727074+0000","last_peered":"2024-09-14T04:28:31.727074+0000","last_clean":"2024-09-14T04:28:31.727074+0000","last_became_active":"2024-09-14T04:26:15.726210+0000","last_became_peered":"2024-09-14T04:26:15.726210+0000","last_unstale":"2024-09-14T04:28:31.727074+0000","last_undegraded":"2024-09-14T04:28:31.727074+0000","last_fullsized":"2024-09-14T04:28:31.727074+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:01.720621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.c","version":"406'7492","reported_seq":7550,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.618641+0000","last_change":"2024-09-14T04:27:15.710653+0000","last_active":"2024-09-14T04:28:44.618641+0000","last_peered":"2024-09-14T04:28:44.618641+0000","last_clean":"2024-09-14T04:28:44.618641+0000","last_became_active":"2024-09-14T04:27:15.710386+0000","last_became_peered":"2024-09-14T04:27:15.710386+0000","last_unstale":"2024-09-14T04:28:44.618641+0000","last_undegraded":"2024-09-14T04:28:44.618641+0000","last_fullsized":"2024-09-14T04:28:44.618641+0000","mapping_epoch":385,"log_start":"390'2147","ondisk_log_start":"390'2147","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:47:23.570399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490995712,"num_objects":7492,"num_object_clones":0,"num_object_copies":22476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7492,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"76'100","reported_seq":288,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.726464+0000","last_change":"2024-09-14T04:27:36.436313+0000","last_active":"2024-09-14T04:28:31.726464+0000","last_peered":"2024-09-14T04:28:31.726464+0000","last_clean":"2024-09-14T04:28:31.726464+0000","last_became_active":"2024-09-14T04:27:12.685740+0000","last_became_peered":"2024-09-14T04:27:12.685740+0000","last_unstale":"2024-09-14T04:28:31.726464+0000","last_undegraded":"2024-09-14T04:28:31.726464+0000","last_fullsized":"2024-09-14T04:28:31.726464+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:36.436230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:36.436230+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:26:41.485866+0000","scrub_duration":0.0026608669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":143,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677160+0000","last_change":"2024-09-14T04:26:15.727773+0000","last_active":"2024-09-14T04:28:36.677160+0000","last_peered":"2024-09-14T04:28:36.677160+0000","last_clean":"2024-09-14T04:28:36.677160+0000","last_became_active":"2024-09-14T04:26:15.727477+0000","last_became_peered":"2024-09-14T04:26:15.727477+0000","last_unstale":"2024-09-14T04:28:36.677160+0000","last_undegraded":"2024-09-14T04:28:36.677160+0000","last_fullsized":"2024-09-14T04:28:36.677160+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:09:08.222067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":510,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677676+0000","last_change":"2024-09-14T04:26:15.724485+0000","last_active":"2024-09-14T04:28:36.677676+0000","last_peered":"2024-09-14T04:28:36.677676+0000","last_clean":"2024-09-14T04:28:36.677676+0000","last_became_active":"2024-09-14T04:26:15.723989+0000","last_became_peered":"2024-09-14T04:26:15.723989+0000","last_unstale":"2024-09-14T04:28:36.677676+0000","last_undegraded":"2024-09-14T04:28:36.677676+0000","last_fullsized":"2024-09-14T04:28:36.677676+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:27:04.257761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":110,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.675656+0000","last_change":"2024-09-14T04:26:15.753288+0000","last_active":"2024-09-14T04:28:36.675656+0000","last_peered":"2024-09-14T04:28:36.675656+0000","last_clean":"2024-09-14T04:28:36.675656+0000","last_became_active":"2024-09-14T04:26:15.753003+0000","last_became_peered":"2024-09-14T04:26:15.753003+0000","last_unstale":"2024-09-14T04:28:36.675656+0000","last_undegraded":"2024-09-14T04:28:36.675656+0000","last_fullsized":"2024-09-14T04:28:36.675656+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:05:37.332422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.d","version":"406'7686","reported_seq":7715,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:49.001994+0000","last_change":"2024-09-14T04:27:15.716650+0000","last_active":"2024-09-14T04:28:49.001994+0000","last_peered":"2024-09-14T04:28:49.001994+0000","last_clean":"2024-09-14T04:28:49.001994+0000","last_became_active":"2024-09-14T04:27:15.716380+0000","last_became_peered":"2024-09-14T04:27:15.716380+0000","last_unstale":"2024-09-14T04:28:49.001994+0000","last_undegraded":"2024-09-14T04:28:49.001994+0000","last_fullsized":"2024-09-14T04:28:49.001994+0000","mapping_epoch":385,"log_start":"390'3100","ondisk_log_start":"390'3100","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:39:59.769243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503709696,"num_objects":7686,"num_object_clones":0,"num_object_copies":23058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7686,"num_write_kb":491904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"76'100","reported_seq":272,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.674201+0000","last_change":"2024-09-14T04:27:44.819744+0000","last_active":"2024-09-14T04:28:36.674201+0000","last_peered":"2024-09-14T04:28:36.674201+0000","last_clean":"2024-09-14T04:28:36.674201+0000","last_became_active":"2024-09-14T04:27:12.689439+0000","last_became_peered":"2024-09-14T04:27:12.689439+0000","last_unstale":"2024-09-14T04:28:36.674201+0000","last_undegraded":"2024-09-14T04:28:36.674201+0000","last_fullsized":"2024-09-14T04:28:36.674201+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:44.819663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:44.819663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:25:22.854494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":110,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.676342+0000","last_change":"2024-09-14T04:26:15.723403+0000","last_active":"2024-09-14T04:28:36.676342+0000","last_peered":"2024-09-14T04:28:36.676342+0000","last_clean":"2024-09-14T04:28:36.676342+0000","last_became_active":"2024-09-14T04:26:15.723058+0000","last_became_peered":"2024-09-14T04:26:15.723058+0000","last_unstale":"2024-09-14T04:28:36.676342+0000","last_undegraded":"2024-09-14T04:28:36.676342+0000","last_fullsized":"2024-09-14T04:28:36.676342+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:53:41.162301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":109,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.675179+0000","last_change":"2024-09-14T04:26:15.727903+0000","last_active":"2024-09-14T04:28:36.675179+0000","last_peered":"2024-09-14T04:28:36.675179+0000","last_clean":"2024-09-14T04:28:36.675179+0000","last_became_active":"2024-09-14T04:26:15.727050+0000","last_became_peered":"2024-09-14T04:26:15.727050+0000","last_unstale":"2024-09-14T04:28:36.675179+0000","last_undegraded":"2024-09-14T04:28:36.675179+0000","last_fullsized":"2024-09-14T04:28:36.675179+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:27:59.930610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":365,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.724837+0000","last_change":"2024-09-14T04:26:15.737341+0000","last_active":"2024-09-14T04:28:31.724837+0000","last_peered":"2024-09-14T04:28:31.724837+0000","last_clean":"2024-09-14T04:28:31.724837+0000","last_became_active":"2024-09-14T04:26:15.736045+0000","last_became_peered":"2024-09-14T04:26:15.736045+0000","last_unstale":"2024-09-14T04:28:31.724837+0000","last_undegraded":"2024-09-14T04:28:31.724837+0000","last_fullsized":"2024-09-14T04:28:31.724837+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:08:08.143667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.e","version":"406'7354","reported_seq":7412,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.607525+0000","last_change":"2024-09-14T04:27:15.712178+0000","last_active":"2024-09-14T04:28:44.607525+0000","last_peered":"2024-09-14T04:28:44.607525+0000","last_clean":"2024-09-14T04:28:44.607525+0000","last_became_active":"2024-09-14T04:27:15.711715+0000","last_became_peered":"2024-09-14T04:27:15.711715+0000","last_unstale":"2024-09-14T04:28:44.607525+0000","last_undegraded":"2024-09-14T04:28:44.607525+0000","last_fullsized":"2024-09-14T04:28:44.607525+0000","mapping_epoch":385,"log_start":"390'2028","ondisk_log_start":"390'2028","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":5326,"log_dups_size":0,"ondisk_log_size":5326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:29:41.525368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481951744,"num_objects":7354,"num_object_clones":0,"num_object_copies":22062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7354,"num_write_kb":470656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"76'100","reported_seq":42,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.722295+0000","last_change":"2024-09-14T04:27:43.130461+0000","last_active":"2024-09-14T04:28:31.722295+0000","last_peered":"2024-09-14T04:28:31.722295+0000","last_clean":"2024-09-14T04:28:31.722295+0000","last_became_active":"2024-09-14T04:27:12.687762+0000","last_became_peered":"2024-09-14T04:27:12.687762+0000","last_unstale":"2024-09-14T04:28:31.722295+0000","last_undegraded":"2024-09-14T04:28:31.722295+0000","last_fullsized":"2024-09-14T04:28:31.722295+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:43.130258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:43.130258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:26:17.918367+0000","scrub_duration":0.0054891430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":123,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723291+0000","last_change":"2024-09-14T04:27:06.910535+0000","last_active":"2024-09-14T04:28:31.723291+0000","last_peered":"2024-09-14T04:28:31.723291+0000","last_clean":"2024-09-14T04:28:31.723291+0000","last_became_active":"2024-09-14T04:26:15.729031+0000","last_became_peered":"2024-09-14T04:26:15.729031+0000","last_unstale":"2024-09-14T04:28:31.723291+0000","last_undegraded":"2024-09-14T04:28:31.723291+0000","last_fullsized":"2024-09-14T04:28:31.723291+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:12.734036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":38,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.675074+0000","last_change":"2024-09-14T04:27:55.393446+0000","last_active":"2024-09-14T04:28:36.675074+0000","last_peered":"2024-09-14T04:28:36.675074+0000","last_clean":"2024-09-14T04:28:36.675074+0000","last_became_active":"2024-09-14T04:27:55.393202+0000","last_became_peered":"2024-09-14T04:27:55.393202+0000","last_unstale":"2024-09-14T04:28:36.675074+0000","last_undegraded":"2024-09-14T04:28:36.675074+0000","last_fullsized":"2024-09-14T04:28:36.675074+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:00:14.696662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":103,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.676539+0000","last_change":"2024-09-14T04:25:42.246310+0000","last_active":"2024-09-14T04:28:36.676539+0000","last_peered":"2024-09-14T04:28:36.676539+0000","last_clean":"2024-09-14T04:28:36.676539+0000","last_became_active":"2024-09-14T04:25:42.245997+0000","last_became_peered":"2024-09-14T04:25:42.245997+0000","last_unstale":"2024-09-14T04:28:36.676539+0000","last_undegraded":"2024-09-14T04:28:36.676539+0000","last_fullsized":"2024-09-14T04:28:36.676539+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:01:43.109388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.8","version":"406'5696","reported_seq":5791,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.583665+0000","last_change":"2024-09-14T04:28:05.139226+0000","last_active":"2024-09-14T04:28:44.583665+0000","last_peered":"2024-09-14T04:28:44.583665+0000","last_clean":"2024-09-14T04:28:44.583665+0000","last_became_active":"2024-09-14T04:28:02.407635+0000","last_became_peered":"2024-09-14T04:28:02.407635+0000","last_unstale":"2024-09-14T04:28:44.583665+0000","last_undegraded":"2024-09-14T04:28:44.583665+0000","last_fullsized":"2024-09-14T04:28:44.583665+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4184","last_scrub_stamp":"2024-09-14T04:28:05.139073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:05.139073+0000","objects_scrubbed":2132,"log_size":5696,"log_dups_size":0,"ondisk_log_size":5696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:17:35.385374+0000","scrub_duration":552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239403008,"num_objects":3653,"num_object_clones":0,"num_object_copies":10959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1521,"num_write_kb":97344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"76'100","reported_seq":288,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.727003+0000","last_change":"2024-09-14T04:27:38.451495+0000","last_active":"2024-09-14T04:28:31.727003+0000","last_peered":"2024-09-14T04:28:31.727003+0000","last_clean":"2024-09-14T04:28:31.727003+0000","last_became_active":"2024-09-14T04:27:12.687176+0000","last_became_peered":"2024-09-14T04:27:12.687176+0000","last_unstale":"2024-09-14T04:28:31.727003+0000","last_undegraded":"2024-09-14T04:28:31.727003+0000","last_fullsized":"2024-09-14T04:28:31.727003+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:38.451408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:38.451408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:08.295464+0000","scrub_duration":0.0034959740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":111,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723767+0000","last_change":"2024-09-14T04:25:42.253453+0000","last_active":"2024-09-14T04:28:31.723767+0000","last_peered":"2024-09-14T04:28:31.723767+0000","last_clean":"2024-09-14T04:28:31.723767+0000","last_became_active":"2024-09-14T04:25:42.252529+0000","last_became_peered":"2024-09-14T04:25:42.252529+0000","last_unstale":"2024-09-14T04:28:31.723767+0000","last_undegraded":"2024-09-14T04:28:31.723767+0000","last_fullsized":"2024-09-14T04:28:31.723767+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:19.539802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":110,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.676636+0000","last_change":"2024-09-14T04:26:15.723378+0000","last_active":"2024-09-14T04:28:36.676636+0000","last_peered":"2024-09-14T04:28:36.676636+0000","last_clean":"2024-09-14T04:28:36.676636+0000","last_became_active":"2024-09-14T04:26:15.723070+0000","last_became_peered":"2024-09-14T04:26:15.723070+0000","last_unstale":"2024-09-14T04:28:36.676636+0000","last_undegraded":"2024-09-14T04:28:36.676636+0000","last_fullsized":"2024-09-14T04:28:36.676636+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:38:14.395886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":588,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723115+0000","last_change":"2024-09-14T04:26:15.723597+0000","last_active":"2024-09-14T04:28:31.723115+0000","last_peered":"2024-09-14T04:28:31.723115+0000","last_clean":"2024-09-14T04:28:31.723115+0000","last_became_active":"2024-09-14T04:26:15.723289+0000","last_became_peered":"2024-09-14T04:26:15.723289+0000","last_unstale":"2024-09-14T04:28:31.723115+0000","last_undegraded":"2024-09-14T04:28:31.723115+0000","last_fullsized":"2024-09-14T04:28:31.723115+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:55:41.253830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"406'5817","reported_seq":5956,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.298007+0000","last_change":"2024-09-14T04:28:04.855497+0000","last_active":"2024-09-14T04:28:44.298007+0000","last_peered":"2024-09-14T04:28:44.298007+0000","last_clean":"2024-09-14T04:28:44.298007+0000","last_became_active":"2024-09-14T04:28:02.428275+0000","last_became_peered":"2024-09-14T04:28:02.428275+0000","last_unstale":"2024-09-14T04:28:44.298007+0000","last_undegraded":"2024-09-14T04:28:44.298007+0000","last_fullsized":"2024-09-14T04:28:44.298007+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4320","last_scrub_stamp":"2024-09-14T04:28:04.855410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:04.855410+0000","objects_scrubbed":2256,"log_size":5817,"log_dups_size":0,"ondisk_log_size":5817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:52:26.414450+0000","scrub_duration":0.70843562500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247267328,"num_objects":3773,"num_object_clones":0,"num_object_copies":11319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1517,"num_write_kb":97088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"76'100","reported_seq":288,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.727042+0000","last_change":"2024-09-14T04:27:43.482881+0000","last_active":"2024-09-14T04:28:31.727042+0000","last_peered":"2024-09-14T04:28:31.727042+0000","last_clean":"2024-09-14T04:28:31.727042+0000","last_became_active":"2024-09-14T04:27:12.686451+0000","last_became_peered":"2024-09-14T04:27:12.686451+0000","last_unstale":"2024-09-14T04:28:31.727042+0000","last_undegraded":"2024-09-14T04:28:31.727042+0000","last_fullsized":"2024-09-14T04:28:31.727042+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:43.482790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:43.482790+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:48:43.699493+0000","scrub_duration":0.013209995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":110,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.676732+0000","last_change":"2024-09-14T04:26:15.734625+0000","last_active":"2024-09-14T04:28:36.676732+0000","last_peered":"2024-09-14T04:28:36.676732+0000","last_clean":"2024-09-14T04:28:36.676732+0000","last_became_active":"2024-09-14T04:26:15.734309+0000","last_became_peered":"2024-09-14T04:26:15.734309+0000","last_unstale":"2024-09-14T04:28:36.676732+0000","last_undegraded":"2024-09-14T04:28:36.676732+0000","last_fullsized":"2024-09-14T04:28:36.676732+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:58:57.797342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.726775+0000","last_change":"2024-09-14T04:26:15.727947+0000","last_active":"2024-09-14T04:28:31.726775+0000","last_peered":"2024-09-14T04:28:31.726775+0000","last_clean":"2024-09-14T04:28:31.726775+0000","last_became_active":"2024-09-14T04:26:15.726608+0000","last_became_peered":"2024-09-14T04:26:15.726608+0000","last_unstale":"2024-09-14T04:28:31.726775+0000","last_undegraded":"2024-09-14T04:28:31.726775+0000","last_fullsized":"2024-09-14T04:28:31.726775+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:33:03.772215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":648,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723174+0000","last_change":"2024-09-14T04:26:15.732599+0000","last_active":"2024-09-14T04:28:31.723174+0000","last_peered":"2024-09-14T04:28:31.723174+0000","last_clean":"2024-09-14T04:28:31.723174+0000","last_became_active":"2024-09-14T04:26:15.725658+0000","last_became_peered":"2024-09-14T04:26:15.725658+0000","last_unstale":"2024-09-14T04:28:31.723174+0000","last_undegraded":"2024-09-14T04:28:31.723174+0000","last_fullsized":"2024-09-14T04:28:31.723174+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:48:20.963473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.a","version":"406'7448","reported_seq":7477,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.290065+0000","last_change":"2024-09-14T04:27:15.713659+0000","last_active":"2024-09-14T04:28:44.290065+0000","last_peered":"2024-09-14T04:28:44.290065+0000","last_clean":"2024-09-14T04:28:44.290065+0000","last_became_active":"2024-09-14T04:27:15.713394+0000","last_became_peered":"2024-09-14T04:27:15.713394+0000","last_unstale":"2024-09-14T04:28:44.290065+0000","last_undegraded":"2024-09-14T04:28:44.290065+0000","last_fullsized":"2024-09-14T04:28:44.290065+0000","mapping_epoch":385,"log_start":"390'3200","ondisk_log_start":"390'3200","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:14:07.900514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488112128,"num_objects":7448,"num_object_clones":0,"num_object_copies":22344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7448,"num_write_kb":476672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'100","reported_seq":272,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.674507+0000","last_change":"2024-09-14T04:27:43.802100+0000","last_active":"2024-09-14T04:28:36.674507+0000","last_peered":"2024-09-14T04:28:36.674507+0000","last_clean":"2024-09-14T04:28:36.674507+0000","last_became_active":"2024-09-14T04:27:12.687977+0000","last_became_peered":"2024-09-14T04:27:12.687977+0000","last_unstale":"2024-09-14T04:28:36.674507+0000","last_undegraded":"2024-09-14T04:28:36.674507+0000","last_fullsized":"2024-09-14T04:28:36.674507+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:43.801906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:43.801906+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:33:01.220893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":678,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677990+0000","last_change":"2024-09-14T04:26:15.748334+0000","last_active":"2024-09-14T04:28:36.677990+0000","last_peered":"2024-09-14T04:28:36.677990+0000","last_clean":"2024-09-14T04:28:36.677990+0000","last_became_active":"2024-09-14T04:26:15.726174+0000","last_became_peered":"2024-09-14T04:26:15.726174+0000","last_unstale":"2024-09-14T04:28:36.677990+0000","last_undegraded":"2024-09-14T04:28:36.677990+0000","last_fullsized":"2024-09-14T04:28:36.677990+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:25:47.468961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":424,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.727256+0000","last_change":"2024-09-14T04:24:47.618083+0000","last_active":"2024-09-14T04:28:31.727256+0000","last_peered":"2024-09-14T04:28:31.727256+0000","last_clean":"2024-09-14T04:28:31.727256+0000","last_became_active":"2024-09-14T04:24:47.617759+0000","last_became_peered":"2024-09-14T04:24:47.617759+0000","last_unstale":"2024-09-14T04:28:31.727256+0000","last_undegraded":"2024-09-14T04:28:31.727256+0000","last_fullsized":"2024-09-14T04:28:31.727256+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:08:35.614869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":208,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677185+0000","last_change":"2024-09-14T04:26:15.726983+0000","last_active":"2024-09-14T04:28:36.677185+0000","last_peered":"2024-09-14T04:28:36.677185+0000","last_clean":"2024-09-14T04:28:36.677185+0000","last_became_active":"2024-09-14T04:26:15.726003+0000","last_became_peered":"2024-09-14T04:26:15.726003+0000","last_unstale":"2024-09-14T04:28:36.677185+0000","last_undegraded":"2024-09-14T04:28:36.677185+0000","last_fullsized":"2024-09-14T04:28:36.677185+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:18:15.328679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.b","version":"406'7404","reported_seq":7462,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.594858+0000","last_change":"2024-09-14T04:27:15.710534+0000","last_active":"2024-09-14T04:28:44.594858+0000","last_peered":"2024-09-14T04:28:44.594858+0000","last_clean":"2024-09-14T04:28:44.594858+0000","last_became_active":"2024-09-14T04:27:15.710259+0000","last_became_peered":"2024-09-14T04:27:15.710259+0000","last_unstale":"2024-09-14T04:28:44.594858+0000","last_undegraded":"2024-09-14T04:28:44.594858+0000","last_fullsized":"2024-09-14T04:28:44.594858+0000","mapping_epoch":385,"log_start":"390'2055","ondisk_log_start":"390'2055","created":385,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":5349,"log_dups_size":0,"ondisk_log_size":5349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:47:11.653676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485228544,"num_objects":7404,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7404,"num_write_kb":473856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"76'100","reported_seq":72,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677766+0000","last_change":"2024-09-14T04:27:44.420656+0000","last_active":"2024-09-14T04:28:36.677766+0000","last_peered":"2024-09-14T04:28:36.677766+0000","last_clean":"2024-09-14T04:28:36.677766+0000","last_became_active":"2024-09-14T04:27:12.689883+0000","last_became_peered":"2024-09-14T04:27:12.689883+0000","last_unstale":"2024-09-14T04:28:36.677766+0000","last_undegraded":"2024-09-14T04:28:36.677766+0000","last_fullsized":"2024-09-14T04:28:36.677766+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:44.420585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:44.420585+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:51:18.088643+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":467,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723078+0000","last_change":"2024-09-14T04:26:15.760792+0000","last_active":"2024-09-14T04:28:31.723078+0000","last_peered":"2024-09-14T04:28:31.723078+0000","last_clean":"2024-09-14T04:28:31.723078+0000","last_became_active":"2024-09-14T04:26:15.745969+0000","last_became_peered":"2024-09-14T04:26:15.745969+0000","last_unstale":"2024-09-14T04:28:31.723078+0000","last_undegraded":"2024-09-14T04:28:31.723078+0000","last_fullsized":"2024-09-14T04:28:31.723078+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:41:56.094887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":424,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.727149+0000","last_change":"2024-09-14T04:25:42.332231+0000","last_active":"2024-09-14T04:28:31.727149+0000","last_peered":"2024-09-14T04:28:31.727149+0000","last_clean":"2024-09-14T04:28:31.727149+0000","last_became_active":"2024-09-14T04:25:42.331514+0000","last_became_peered":"2024-09-14T04:25:42.331514+0000","last_unstale":"2024-09-14T04:28:31.727149+0000","last_undegraded":"2024-09-14T04:28:31.727149+0000","last_fullsized":"2024-09-14T04:28:31.727149+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:33:37.729778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":91,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.722606+0000","last_change":"2024-09-14T04:22:08.465532+0000","last_active":"2024-09-14T04:28:31.722606+0000","last_peered":"2024-09-14T04:28:31.722606+0000","last_clean":"2024-09-14T04:28:31.722606+0000","last_became_active":"2024-09-14T04:22:08.464571+0000","last_became_peered":"2024-09-14T04:22:08.464571+0000","last_unstale":"2024-09-14T04:28:31.722606+0000","last_undegraded":"2024-09-14T04:28:31.722606+0000","last_fullsized":"2024-09-14T04:28:31.722606+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:25:23.109822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.19","version":"406'5829","reported_seq":5968,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.262026+0000","last_change":"2024-09-14T04:28:06.394086+0000","last_active":"2024-09-14T04:28:44.262026+0000","last_peered":"2024-09-14T04:28:44.262026+0000","last_clean":"2024-09-14T04:28:44.262026+0000","last_became_active":"2024-09-14T04:28:02.440766+0000","last_became_peered":"2024-09-14T04:28:02.440766+0000","last_unstale":"2024-09-14T04:28:44.262026+0000","last_undegraded":"2024-09-14T04:28:44.262026+0000","last_fullsized":"2024-09-14T04:28:44.262026+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"397'4347","last_scrub_stamp":"2024-09-14T04:28:06.394033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:06.394033+0000","objects_scrubbed":2196,"log_size":5829,"log_dups_size":0,"ondisk_log_size":5829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:20:41.162067+0000","scrub_duration":0.23278294199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241041408,"num_objects":3678,"num_object_clones":0,"num_object_copies":11034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1482,"num_write_kb":94848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":171,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.726595+0000","last_change":"2024-09-14T04:25:53.301718+0000","last_active":"2024-09-14T04:28:31.726595+0000","last_peered":"2024-09-14T04:28:31.726595+0000","last_clean":"2024-09-14T04:28:31.726595+0000","last_became_active":"2024-09-14T04:25:53.301421+0000","last_became_peered":"2024-09-14T04:25:53.301421+0000","last_unstale":"2024-09-14T04:28:31.726595+0000","last_undegraded":"2024-09-14T04:28:31.726595+0000","last_fullsized":"2024-09-14T04:28:31.726595+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:45:47.546722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":128,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.726895+0000","last_change":"2024-09-14T04:25:42.375734+0000","last_active":"2024-09-14T04:28:31.726895+0000","last_peered":"2024-09-14T04:28:31.726895+0000","last_clean":"2024-09-14T04:28:31.726895+0000","last_became_active":"2024-09-14T04:25:42.259500+0000","last_became_peered":"2024-09-14T04:25:42.259500+0000","last_unstale":"2024-09-14T04:28:31.726895+0000","last_undegraded":"2024-09-14T04:28:31.726895+0000","last_fullsized":"2024-09-14T04:28:31.726895+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:27:37.236272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"406'5705","reported_seq":5800,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.506403+0000","last_change":"2024-09-14T04:28:05.927770+0000","last_active":"2024-09-14T04:28:44.506403+0000","last_peered":"2024-09-14T04:28:44.506403+0000","last_clean":"2024-09-14T04:28:44.506403+0000","last_became_active":"2024-09-14T04:28:02.447084+0000","last_became_peered":"2024-09-14T04:28:02.447084+0000","last_unstale":"2024-09-14T04:28:44.506403+0000","last_undegraded":"2024-09-14T04:28:44.506403+0000","last_fullsized":"2024-09-14T04:28:44.506403+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"397'4212","last_scrub_stamp":"2024-09-14T04:28:05.927656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:05.927656+0000","objects_scrubbed":2194,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:47:48.348394+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241827840,"num_objects":3690,"num_object_clones":0,"num_object_copies":11070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1496,"num_write_kb":95744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":601,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.726411+0000","last_change":"2024-09-14T04:26:15.728258+0000","last_active":"2024-09-14T04:28:31.726411+0000","last_peered":"2024-09-14T04:28:31.726411+0000","last_clean":"2024-09-14T04:28:31.726411+0000","last_became_active":"2024-09-14T04:26:15.726521+0000","last_became_peered":"2024-09-14T04:26:15.726521+0000","last_unstale":"2024-09-14T04:28:31.726411+0000","last_undegraded":"2024-09-14T04:28:31.726411+0000","last_fullsized":"2024-09-14T04:28:31.726411+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:22:20.663544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":342,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678195+0000","last_change":"2024-09-14T04:26:15.758946+0000","last_active":"2024-09-14T04:28:36.678195+0000","last_peered":"2024-09-14T04:28:36.678195+0000","last_clean":"2024-09-14T04:28:36.678195+0000","last_became_active":"2024-09-14T04:26:15.725634+0000","last_became_peered":"2024-09-14T04:26:15.725634+0000","last_unstale":"2024-09-14T04:28:36.678195+0000","last_undegraded":"2024-09-14T04:28:36.678195+0000","last_fullsized":"2024-09-14T04:28:36.678195+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:21:59.189929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":121,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723629+0000","last_change":"2024-09-14T04:25:42.256482+0000","last_active":"2024-09-14T04:28:31.723629+0000","last_peered":"2024-09-14T04:28:31.723629+0000","last_clean":"2024-09-14T04:28:31.723629+0000","last_became_active":"2024-09-14T04:25:42.256182+0000","last_became_peered":"2024-09-14T04:25:42.256182+0000","last_unstale":"2024-09-14T04:28:31.723629+0000","last_undegraded":"2024-09-14T04:28:31.723629+0000","last_fullsized":"2024-09-14T04:28:31.723629+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:04:58.123110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":622,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678252+0000","last_change":"2024-09-14T04:25:03.864958+0000","last_active":"2024-09-14T04:28:36.678252+0000","last_peered":"2024-09-14T04:28:36.678252+0000","last_clean":"2024-09-14T04:28:36.678252+0000","last_became_active":"2024-09-14T04:25:03.864529+0000","last_became_peered":"2024-09-14T04:25:03.864529+0000","last_unstale":"2024-09-14T04:28:36.678252+0000","last_undegraded":"2024-09-14T04:28:36.678252+0000","last_fullsized":"2024-09-14T04:28:36.678252+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:12:25.237701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":651,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.724749+0000","last_change":"2024-09-14T04:26:15.736843+0000","last_active":"2024-09-14T04:28:31.724749+0000","last_peered":"2024-09-14T04:28:31.724749+0000","last_clean":"2024-09-14T04:28:31.724749+0000","last_became_active":"2024-09-14T04:26:15.734655+0000","last_became_peered":"2024-09-14T04:26:15.734655+0000","last_unstale":"2024-09-14T04:28:31.724749+0000","last_undegraded":"2024-09-14T04:28:31.724749+0000","last_fullsized":"2024-09-14T04:28:31.724749+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:47:05.810191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":38,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.674760+0000","last_change":"2024-09-14T04:27:55.394565+0000","last_active":"2024-09-14T04:28:36.674760+0000","last_peered":"2024-09-14T04:28:36.674760+0000","last_clean":"2024-09-14T04:28:36.674760+0000","last_became_active":"2024-09-14T04:27:55.394243+0000","last_became_peered":"2024-09-14T04:27:55.394243+0000","last_unstale":"2024-09-14T04:28:36.674760+0000","last_undegraded":"2024-09-14T04:28:36.674760+0000","last_fullsized":"2024-09-14T04:28:36.674760+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:36:52.365624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"406'5866","reported_seq":5961,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.708969+0000","last_change":"2024-09-14T04:28:06.796053+0000","last_active":"2024-09-14T04:28:44.708969+0000","last_peered":"2024-09-14T04:28:44.708969+0000","last_clean":"2024-09-14T04:28:44.708969+0000","last_became_active":"2024-09-14T04:28:02.446713+0000","last_became_peered":"2024-09-14T04:28:02.446713+0000","last_unstale":"2024-09-14T04:28:44.708969+0000","last_undegraded":"2024-09-14T04:28:44.708969+0000","last_fullsized":"2024-09-14T04:28:44.708969+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"397'4350","last_scrub_stamp":"2024-09-14T04:28:06.795987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:06.795987+0000","objects_scrubbed":2241,"log_size":5866,"log_dups_size":0,"ondisk_log_size":5866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:12:25.820429+0000","scrub_duration":265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246218752,"num_objects":3757,"num_object_clones":0,"num_object_copies":11271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1516,"num_write_kb":97024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":431,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.727107+0000","last_change":"2024-09-14T04:26:15.731440+0000","last_active":"2024-09-14T04:28:31.727107+0000","last_peered":"2024-09-14T04:28:31.727107+0000","last_clean":"2024-09-14T04:28:31.727107+0000","last_became_active":"2024-09-14T04:26:15.726850+0000","last_became_peered":"2024-09-14T04:26:15.726850+0000","last_unstale":"2024-09-14T04:28:31.727107+0000","last_undegraded":"2024-09-14T04:28:31.727107+0000","last_fullsized":"2024-09-14T04:28:31.727107+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:00:48.044324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":468,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677278+0000","last_change":"2024-09-14T04:26:15.727272+0000","last_active":"2024-09-14T04:28:36.677278+0000","last_peered":"2024-09-14T04:28:36.677278+0000","last_clean":"2024-09-14T04:28:36.677278+0000","last_became_active":"2024-09-14T04:26:15.726200+0000","last_became_peered":"2024-09-14T04:26:15.726200+0000","last_unstale":"2024-09-14T04:28:36.677278+0000","last_undegraded":"2024-09-14T04:28:36.677278+0000","last_fullsized":"2024-09-14T04:28:36.677278+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:38:30.447563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"406'5767","reported_seq":5806,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:49.009436+0000","last_change":"2024-09-14T04:28:02.438516+0000","last_active":"2024-09-14T04:28:49.009436+0000","last_peered":"2024-09-14T04:28:49.009436+0000","last_clean":"2024-09-14T04:28:49.009436+0000","last_became_active":"2024-09-14T04:28:02.436585+0000","last_became_peered":"2024-09-14T04:28:02.436585+0000","last_unstale":"2024-09-14T04:28:49.009436+0000","last_undegraded":"2024-09-14T04:28:49.009436+0000","last_fullsized":"2024-09-14T04:28:49.009436+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":5767,"log_dups_size":0,"ondisk_log_size":5767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247070720,"num_objects":3770,"num_object_clones":0,"num_object_copies":11310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3770,"num_write_kb":241280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":465,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677784+0000","last_change":"2024-09-14T04:26:15.745872+0000","last_active":"2024-09-14T04:28:36.677784+0000","last_peered":"2024-09-14T04:28:36.677784+0000","last_clean":"2024-09-14T04:28:36.677784+0000","last_became_active":"2024-09-14T04:26:15.745533+0000","last_became_peered":"2024-09-14T04:26:15.745533+0000","last_unstale":"2024-09-14T04:28:36.677784+0000","last_undegraded":"2024-09-14T04:28:36.677784+0000","last_fullsized":"2024-09-14T04:28:36.677784+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:28:40.179766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":483,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678383+0000","last_change":"2024-09-14T04:24:47.612687+0000","last_active":"2024-09-14T04:28:36.678383+0000","last_peered":"2024-09-14T04:28:36.678383+0000","last_clean":"2024-09-14T04:28:36.678383+0000","last_became_active":"2024-09-14T04:24:47.612392+0000","last_became_peered":"2024-09-14T04:24:47.612392+0000","last_unstale":"2024-09-14T04:28:36.678383+0000","last_undegraded":"2024-09-14T04:28:36.678383+0000","last_fullsized":"2024-09-14T04:28:36.678383+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:52:51.954808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.17","version":"406'5889","reported_seq":5937,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.250019+0000","last_change":"2024-09-14T04:28:09.171746+0000","last_active":"2024-09-14T04:28:44.250019+0000","last_peered":"2024-09-14T04:28:44.250019+0000","last_clean":"2024-09-14T04:28:44.250019+0000","last_became_active":"2024-09-14T04:28:02.446942+0000","last_became_peered":"2024-09-14T04:28:02.446942+0000","last_unstale":"2024-09-14T04:28:44.250019+0000","last_undegraded":"2024-09-14T04:28:44.250019+0000","last_fullsized":"2024-09-14T04:28:44.250019+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":5889,"log_dups_size":0,"ondisk_log_size":5889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249692160,"num_objects":3810,"num_object_clones":0,"num_object_copies":11430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3810,"num_write_kb":243840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":454,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678093+0000","last_change":"2024-09-14T04:24:47.613822+0000","last_active":"2024-09-14T04:28:36.678093+0000","last_peered":"2024-09-14T04:28:36.678093+0000","last_clean":"2024-09-14T04:28:36.678093+0000","last_became_active":"2024-09-14T04:24:47.613315+0000","last_became_peered":"2024-09-14T04:24:47.613315+0000","last_unstale":"2024-09-14T04:28:36.678093+0000","last_undegraded":"2024-09-14T04:28:36.678093+0000","last_fullsized":"2024-09-14T04:28:36.678093+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:22.682884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":253,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.674300+0000","last_change":"2024-09-14T04:26:15.727713+0000","last_active":"2024-09-14T04:28:36.674300+0000","last_peered":"2024-09-14T04:28:36.674300+0000","last_clean":"2024-09-14T04:28:36.674300+0000","last_became_active":"2024-09-14T04:26:15.727020+0000","last_became_peered":"2024-09-14T04:26:15.727020+0000","last_unstale":"2024-09-14T04:28:36.674300+0000","last_undegraded":"2024-09-14T04:28:36.674300+0000","last_fullsized":"2024-09-14T04:28:36.674300+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:35:46.915353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.16","version":"406'5726","reported_seq":5819,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.501401+0000","last_change":"2024-09-14T04:28:04.126036+0000","last_active":"2024-09-14T04:28:44.501401+0000","last_peered":"2024-09-14T04:28:44.501401+0000","last_clean":"2024-09-14T04:28:44.501401+0000","last_became_active":"2024-09-14T04:28:02.403141+0000","last_became_peered":"2024-09-14T04:28:02.403141+0000","last_unstale":"2024-09-14T04:28:44.501401+0000","last_undegraded":"2024-09-14T04:28:44.501401+0000","last_fullsized":"2024-09-14T04:28:44.501401+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"396'4255","last_scrub_stamp":"2024-09-14T04:28:04.125956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:04.125956+0000","objects_scrubbed":2150,"log_size":5726,"log_dups_size":0,"ondisk_log_size":5726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:47:09.636896+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238354432,"num_objects":3637,"num_object_clones":0,"num_object_copies":10911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1487,"num_write_kb":95168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":110,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.676426+0000","last_change":"2024-09-14T04:26:15.721332+0000","last_active":"2024-09-14T04:28:36.676426+0000","last_peered":"2024-09-14T04:28:36.676426+0000","last_clean":"2024-09-14T04:28:36.676426+0000","last_became_active":"2024-09-14T04:26:15.721017+0000","last_became_peered":"2024-09-14T04:26:15.721017+0000","last_unstale":"2024-09-14T04:28:36.676426+0000","last_undegraded":"2024-09-14T04:28:36.676426+0000","last_fullsized":"2024-09-14T04:28:36.676426+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:44:50.434373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":128,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678349+0000","last_change":"2024-09-14T04:26:15.731906+0000","last_active":"2024-09-14T04:28:36.678349+0000","last_peered":"2024-09-14T04:28:36.678349+0000","last_clean":"2024-09-14T04:28:36.678349+0000","last_became_active":"2024-09-14T04:26:15.730836+0000","last_became_peered":"2024-09-14T04:26:15.730836+0000","last_unstale":"2024-09-14T04:28:36.678349+0000","last_undegraded":"2024-09-14T04:28:36.678349+0000","last_fullsized":"2024-09-14T04:28:36.678349+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:03:41.330132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.11","version":"406'5851","reported_seq":9231,"reported_epoch":406,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-14T04:28:49.006958+0000","last_change":"2024-09-14T04:28:28.152207+0000","last_active":"2024-09-14T04:28:49.006958+0000","last_peered":"2024-09-14T04:28:49.006958+0000","last_clean":"2024-09-14T04:28:05.973594+0000","last_became_active":"2024-09-14T04:28:09.489460+0000","last_became_peered":"2024-09-14T04:28:09.489460+0000","last_unstale":"2024-09-14T04:28:49.006958+0000","last_undegraded":"2024-09-14T04:28:49.006958+0000","last_fullsized":"2024-09-14T04:28:09.429450+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"396'4149","last_scrub_stamp":"2024-09-14T04:28:04.000577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:04.000577+0000","objects_scrubbed":2134,"log_size":5851,"log_dups_size":0,"ondisk_log_size":5851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:25:53.024093+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252248064,"num_objects":3849,"num_object_clones":0,"num_object_copies":11547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":581,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1715,"num_write_kb":109760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1644,"num_bytes_recovered":107741184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,2],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":581},{"shards":"2,6","objects":3268}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":552,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678458+0000","last_change":"2024-09-14T04:25:42.236692+0000","last_active":"2024-09-14T04:28:36.678458+0000","last_peered":"2024-09-14T04:28:36.678458+0000","last_clean":"2024-09-14T04:28:36.678458+0000","last_became_active":"2024-09-14T04:25:42.236449+0000","last_became_peered":"2024-09-14T04:25:42.236449+0000","last_unstale":"2024-09-14T04:28:36.678458+0000","last_undegraded":"2024-09-14T04:28:36.678458+0000","last_fullsized":"2024-09-14T04:28:36.678458+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:19:34.323210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":109,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.723158+0000","last_change":"2024-09-14T04:24:47.706532+0000","last_active":"2024-09-14T04:28:31.723158+0000","last_peered":"2024-09-14T04:28:31.723158+0000","last_clean":"2024-09-14T04:28:31.723158+0000","last_became_active":"2024-09-14T04:24:47.628189+0000","last_became_peered":"2024-09-14T04:24:47.628189+0000","last_unstale":"2024-09-14T04:28:31.723158+0000","last_undegraded":"2024-09-14T04:28:31.723158+0000","last_fullsized":"2024-09-14T04:28:31.723158+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:29:46.206803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"406'5672","reported_seq":9960,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.538329+0000","last_change":"2024-09-14T04:28:29.771511+0000","last_active":"2024-09-14T04:28:44.538329+0000","last_peered":"2024-09-14T04:28:44.538329+0000","last_clean":"2024-09-14T04:28:44.538329+0000","last_became_active":"2024-09-14T04:28:29.712822+0000","last_became_peered":"2024-09-14T04:28:29.712822+0000","last_unstale":"2024-09-14T04:28:44.538329+0000","last_undegraded":"2024-09-14T04:28:44.538329+0000","last_fullsized":"2024-09-14T04:28:44.538329+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":395,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-14T04:28:59.749983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237469696,"num_objects":3624,"num_object_clones":0,"num_object_copies":10872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3624,"num_write_kb":231904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2107,"num_bytes_recovered":138084352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":538,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.678721+0000","last_change":"2024-09-14T04:26:15.722033+0000","last_active":"2024-09-14T04:28:36.678721+0000","last_peered":"2024-09-14T04:28:36.678721+0000","last_clean":"2024-09-14T04:28:36.678721+0000","last_became_active":"2024-09-14T04:26:15.721545+0000","last_became_peered":"2024-09-14T04:26:15.721545+0000","last_unstale":"2024-09-14T04:28:36.678721+0000","last_undegraded":"2024-09-14T04:28:36.678721+0000","last_fullsized":"2024-09-14T04:28:36.678721+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:48:21.769211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":42,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.674367+0000","last_change":"2024-09-14T04:27:55.405108+0000","last_active":"2024-09-14T04:28:36.674367+0000","last_peered":"2024-09-14T04:28:36.674367+0000","last_clean":"2024-09-14T04:28:36.674367+0000","last_became_active":"2024-09-14T04:27:55.394475+0000","last_became_peered":"2024-09-14T04:27:55.394475+0000","last_unstale":"2024-09-14T04:28:36.674367+0000","last_undegraded":"2024-09-14T04:28:36.674367+0000","last_fullsized":"2024-09-14T04:28:36.674367+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:44:06.833702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.13","version":"406'5758","reported_seq":5851,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:44.736343+0000","last_change":"2024-09-14T04:28:07.869864+0000","last_active":"2024-09-14T04:28:44.736343+0000","last_peered":"2024-09-14T04:28:44.736343+0000","last_clean":"2024-09-14T04:28:44.736343+0000","last_became_active":"2024-09-14T04:28:02.436137+0000","last_became_peered":"2024-09-14T04:28:02.436137+0000","last_unstale":"2024-09-14T04:28:44.736343+0000","last_undegraded":"2024-09-14T04:28:44.736343+0000","last_fullsized":"2024-09-14T04:28:44.736343+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'4303","last_scrub_stamp":"2024-09-14T04:28:07.869714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:07.869714+0000","objects_scrubbed":2150,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:36:35.124049+0000","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236781568,"num_objects":3613,"num_object_clones":0,"num_object_copies":10839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1463,"num_write_kb":93632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":461,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677836+0000","last_change":"2024-09-14T04:26:15.727588+0000","last_active":"2024-09-14T04:28:36.677836+0000","last_peered":"2024-09-14T04:28:36.677836+0000","last_clean":"2024-09-14T04:28:36.677836+0000","last_became_active":"2024-09-14T04:26:15.727153+0000","last_became_peered":"2024-09-14T04:26:15.727153+0000","last_unstale":"2024-09-14T04:28:36.677836+0000","last_undegraded":"2024-09-14T04:28:36.677836+0000","last_fullsized":"2024-09-14T04:28:36.677836+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:15:27.230932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":470,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-09-14T04:28:36.677883+0000","last_change":"2024-09-14T04:25:42.235102+0000","last_active":"2024-09-14T04:28:36.677883+0000","last_peered":"2024-09-14T04:28:36.677883+0000","last_clean":"2024-09-14T04:28:36.677883+0000","last_became_active":"2024-09-14T04:25:42.234776+0000","last_became_peered":"2024-09-14T04:25:42.234776+0000","last_unstale":"2024-09-14T04:28:36.677883+0000","last_undegraded":"2024-09-14T04:28:36.677883+0000","last_fullsized":"2024-09-14T04:28:36.677883+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:59:52.026651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.12","version":"406'5735","reported_seq":5885,"reported_epoch":406,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-14T04:28:44.697423+0000","last_change":"2024-09-14T04:28:31.832042+0000","last_active":"2024-09-14T04:28:44.697423+0000","last_peered":"2024-09-14T04:28:44.697423+0000","last_clean":"2024-09-14T04:28:29.091793+0000","last_became_active":"2024-09-14T04:28:31.825323+0000","last_became_peered":"2024-09-14T04:28:31.825323+0000","last_unstale":"2024-09-14T04:28:44.697423+0000","last_undegraded":"2024-09-14T04:28:31.799830+0000","last_fullsized":"2024-09-14T04:28:31.724550+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'4281","last_scrub_stamp":"2024-09-14T04:28:07.784588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:07.784588+0000","objects_scrubbed":2244,"log_size":5735,"log_dups_size":0,"ondisk_log_size":5735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.338810958,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242614272,"num_objects":3702,"num_object_clones":0,"num_object_copies":11106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3157,"num_objects_misplaced":3157,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":93312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[3,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5","objects":545},{"shards":"3,5,6","objects":3157}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":220,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.726160+0000","last_change":"2024-09-14T04:25:53.300338+0000","last_active":"2024-09-14T04:28:31.726160+0000","last_peered":"2024-09-14T04:28:31.726160+0000","last_clean":"2024-09-14T04:28:31.726160+0000","last_became_active":"2024-09-14T04:25:53.299919+0000","last_became_peered":"2024-09-14T04:25:53.299919+0000","last_unstale":"2024-09-14T04:28:31.726160+0000","last_undegraded":"2024-09-14T04:28:31.726160+0000","last_fullsized":"2024-09-14T04:28:31.726160+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:17.341402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":680,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-14T04:28:31.724694+0000","last_change":"2024-09-14T04:26:15.738521+0000","last_active":"2024-09-14T04:28:31.724694+0000","last_peered":"2024-09-14T04:28:31.724694+0000","last_clean":"2024-09-14T04:28:31.724694+0000","last_became_active":"2024-09-14T04:26:15.736609+0000","last_became_peered":"2024-09-14T04:26:15.736609+0000","last_unstale":"2024-09-14T04:28:31.724694+0000","last_undegraded":"2024-09-14T04:28:31.724694+0000","last_fullsized":"2024-09-14T04:28:31.724694+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:54:47.371271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":6886,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":21458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":14,"num_pg":26,"stat_sum":{"num_bytes":7811137536,"num_objects":119189,"num_object_clones":0,"num_object_copies":357567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3157,"num_objects_misplaced":3738,"num_objects_unfound":0,"num_objects_dirty":119189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82159,"num_write_kb":5258144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3752,"num_bytes_recovered":245891072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23829872640,"data_stored":23829872640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145605,"ondisk_log_size":145605,"up":78,"acting":76,"num_store_stats":5},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":0,"seq":206158430781,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388627010,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5872416,"kb_used_data":4375832,"kb_used_omap":1,"kb_used_meta":1496574,"kb_avail":87872736,"statfs":{"total":95995035648,"available":89981681664,"internally_reserved":0,"allocated":4480851968,"data_stored":4478889667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1532492120},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823237,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6555152,"kb_used_data":5053000,"kb_used_omap":1,"kb_used_meta":1502142,"kb_avail":87190000,"statfs":{"total":95995035648,"available":89282560000,"internally_reserved":0,"allocated":5174272000,"data_stored":5172321466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1538193752},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917320,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5808536,"kb_used_data":4873936,"kb_used_omap":6,"kb_used_meta":934585,"kb_avail":87936616,"statfs":{"total":95995035648,"available":90047094784,"internally_reserved":0,"allocated":4990910464,"data_stored":4988955718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":957016063},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552618,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5773656,"kb_used_data":4620368,"kb_used_omap":4,"kb_used_meta":1153275,"kb_avail":87971496,"statfs":{"total":95995035648,"available":90082811904,"internally_reserved":0,"allocated":4731256832,"data_stored":4729298276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":1180954045},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435659,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5389012,"kb_used_data":4290060,"kb_used_omap":6,"kb_used_meta":1098937,"kb_avail":88356140,"statfs":{"total":95995035648,"available":90476687360,"internally_reserved":0,"allocated":4393021440,"data_stored":4391070665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":1125311949},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":416611828130,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":343597384118,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4387635200,"data_stored":4387635200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4739432448,"data_stored":4739432448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5018615808,"data_stored":5018615808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5182914560,"data_stored":5182914560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4501274624,"data_stored":4501274624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:28:52.428 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-14T04:28:52.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_9 pg_num 36 2024-09-14T04:28:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:52.442 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:52.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:52.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:52.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 82 16 128666 128650 98.0446 82.75 0.00978139 0.0101952 2024-09-14T04:28:52.581 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:52.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:52 smithi114 ceph-mon[106969]: 14.17 scrub starts 2024-09-14T04:28:52.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:52 smithi114 ceph-mon[106969]: pgmap v1533: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 84 MiB/s wr, 1.34k op/s; 3157/374253 objects degraded (0.844%); 3738/374253 objects misplaced (0.999%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T04:28:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:52 smithi114 ceph-mon[103029]: 14.17 scrub starts 2024-09-14T04:28:52.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:52 smithi114 ceph-mon[103029]: pgmap v1533: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 84 MiB/s wr, 1.34k op/s; 3157/374253 objects degraded (0.844%); 3738/374253 objects misplaced (0.999%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T04:28:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:52 smithi160 ceph-mon[88407]: 14.17 scrub starts 2024-09-14T04:28:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:52 smithi160 ceph-mon[88407]: pgmap v1533: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 84 MiB/s wr, 1.34k op/s; 3157/374253 objects degraded (0.844%); 3738/374253 objects misplaced (0.999%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T04:28:52.683 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:52.732 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:52.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:52.833 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:52.876 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:52.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:52.977 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:53.010 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:53.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:53.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:53.150 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:53.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:53.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:53.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:53.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:53.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:53.418 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:53.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:53.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:53.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:53.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 83 16 129995 129979 97.864 83.0625 0.0236603 0.0102137 2024-09-14T04:28:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:53.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:53 smithi114 ceph-mon[103029]: 14.17 scrub starts 2024-09-14T04:28:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:53.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:53 smithi114 ceph-mon[106969]: 14.17 scrub starts 2024-09-14T04:28:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:53.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:53 smithi160 ceph-mon[88407]: 14.17 scrub starts 2024-09-14T04:28:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:28:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:53.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:53.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:53.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:53.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:28:53.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:53.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:53.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:53.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:53.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:54.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:54.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:54.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:54.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:54.250 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:54.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:54.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:54.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:54.386 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:54.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:54.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:54.519 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:54.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:54.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 84 16 131333 131317 97.6944 83.625 0.00623183 0.0102316 2024-09-14T04:28:54.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:54 smithi114 ceph-mon[103029]: pgmap v1534: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 84 MiB/s wr, 1.34k op/s; 3157/374253 objects degraded (0.844%); 3738/374253 objects misplaced (0.999%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T04:28:54.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:54 smithi114 ceph-mon[106969]: pgmap v1534: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 84 MiB/s wr, 1.34k op/s; 3157/374253 objects degraded (0.844%); 3738/374253 objects misplaced (0.999%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T04:28:54.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:54 smithi160 ceph-mon[88407]: pgmap v1534: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 84 MiB/s wr, 1.34k op/s; 3157/374253 objects degraded (0.844%); 3738/374253 objects misplaced (0.999%); 4.4 MiB/s, 69 objects/s recovering 2024-09-14T04:28:54.661 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:54.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:54.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:54.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:54.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:54.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:54.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:54.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:54.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:55.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:55.087 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:55.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:55.189 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:55.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:55.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:55.328 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:55.349 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:55.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:55.451 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:55.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:55.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:55.550 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 85 16 132533 132517 97.4273 75 0.00912589 0.0102569 2024-09-14T04:28:55.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:55.606 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:55.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:55.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:55.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:55 smithi114 ceph-mon[106969]: 14.17 scrub starts 2024-09-14T04:28:55.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:55 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3157/374253 objects degraded (0.844%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:55.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:55 smithi114 ceph-mon[103029]: 14.17 scrub starts 2024-09-14T04:28:55.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:55 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3157/374253 objects degraded (0.844%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:55 smithi160 ceph-mon[88407]: 14.17 scrub starts 2024-09-14T04:28:55.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:55 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3157/374253 objects degraded (0.844%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:28:55.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:55.732 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:55.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:55.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:55.834 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:55.856 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:55.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:55.957 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:55.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:56.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:28:56.107 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:56.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:28:56.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:56.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:56.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:56.333 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:56.354 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:56.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:56.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:56.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:56.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:56.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 86 16 132881 132865 96.5473 21.75 0.0282661 0.0103534 2024-09-14T04:28:56.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:56.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:56 smithi114 ceph-mon[106969]: 14.17 scrub starts 2024-09-14T04:28:56.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:56 smithi114 ceph-mon[106969]: pgmap v1535: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 8.0 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 84 MiB/s wr, 1.34k op/s; 3157/394224 objects degraded (0.801%); 3309/394224 objects misplaced (0.839%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T04:28:56.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:56 smithi114 ceph-mon[106969]: osdmap e408: 8 total, 8 up, 5 in 2024-09-14T04:28:56.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:56 smithi114 ceph-mon[106969]: 14.14 scrub starts 2024-09-14T04:28:56.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:56 smithi114 ceph-mon[103029]: 14.17 scrub starts 2024-09-14T04:28:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:56 smithi114 ceph-mon[103029]: pgmap v1535: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 8.0 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 84 MiB/s wr, 1.34k op/s; 3157/394224 objects degraded (0.801%); 3309/394224 objects misplaced (0.839%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T04:28:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:56 smithi114 ceph-mon[103029]: osdmap e408: 8 total, 8 up, 5 in 2024-09-14T04:28:56.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:56 smithi114 ceph-mon[103029]: 14.14 scrub starts 2024-09-14T04:28:56.625 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:56.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:56 smithi160 ceph-mon[88407]: 14.17 scrub starts 2024-09-14T04:28:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:56 smithi160 ceph-mon[88407]: pgmap v1535: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 8.0 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 84 MiB/s wr, 1.34k op/s; 3157/394224 objects degraded (0.801%); 3309/394224 objects misplaced (0.839%); 5.1 MiB/s, 81 objects/s recovering 2024-09-14T04:28:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:56 smithi160 ceph-mon[88407]: osdmap e408: 8 total, 8 up, 5 in 2024-09-14T04:28:56.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:56 smithi160 ceph-mon[88407]: 14.14 scrub starts 2024-09-14T04:28:56.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:56.763 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:56.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:56.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:56.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:56.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:56.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:57.011 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:57.046 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:57.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:57.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:57.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:57.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:57.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:57.326 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:57.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:57.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:57.463 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:57.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:57.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 87 16 134131 134115 96.3355 78.125 0.00282686 0.010375 2024-09-14T04:28:57.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:57.589 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:57 smithi114 ceph-mon[106969]: osdmap e409: 8 total, 8 up, 5 in 2024-09-14T04:28:57.590 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:57 smithi114 ceph-mon[103029]: osdmap e409: 8 total, 8 up, 5 in 2024-09-14T04:28:57.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:57.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:57.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:57.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:57 smithi160 ceph-mon[88407]: osdmap e409: 8 total, 8 up, 5 in 2024-09-14T04:28:57.704 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:57.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:57.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:28:57.881 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:57.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:28:57.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:58.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:58.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:58.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:58.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:58.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:58.261 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:58.293 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:58.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:58.302 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:28:58.303 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:28:58.303 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:28:58.303 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:28:58.303 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:28:58.303 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:28:58.303 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:28:58.303 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:28:58.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:28:58.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:28:58.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:28:58.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:28:58.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:28:58.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:28:58.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:28:58.304 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:28:58.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:58.443 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:58.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:58.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:58.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 88 16 135348 135332 96.1046 76.0625 0.00728787 0.0104012 2024-09-14T04:28:58.580 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:58.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:58.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:58 smithi114 ceph-mon[106969]: pgmap v1538: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 8.0 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 60 MiB/s wr, 967 op/s; 3157/394224 objects degraded (0.801%); 3309/394224 objects misplaced (0.839%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:28:58.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:58 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3712580554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:58.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:58 smithi114 ceph-mon[103029]: pgmap v1538: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 8.0 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 60 MiB/s wr, 967 op/s; 3157/394224 objects degraded (0.801%); 3309/394224 objects misplaced (0.839%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:28:58.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:58 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3712580554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:58 smithi160 ceph-mon[88407]: pgmap v1538: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 107 active+clean; 8.0 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 60 MiB/s wr, 967 op/s; 3157/394224 objects degraded (0.801%); 3309/394224 objects misplaced (0.839%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:28:58.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:58 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3712580554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:28:58.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:58.722 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:58.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:58.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:58.824 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:58.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:28:58.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:58.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:58.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:28:59.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:28:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:59.137 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:59.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:28:59.238 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:59.277 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:59.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:59.379 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:28:59.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:59.417 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:59.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:28:59.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:28:59.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout: 89 16 136579 136563 95.8892 76.9375 0.0131696 0.0104245 2024-09-14T04:28:59.557 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:59.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:28:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:28:59 smithi160 ceph-mon[88407]: 14.14 scrub starts 2024-09-14T04:28:59.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:59.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:59.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:59.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:59.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:28:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:59.824 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:28:59.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:28:59.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:28:59 smithi114 ceph-mon[106969]: 14.14 scrub starts 2024-09-14T04:28:59.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:28:59 smithi114 ceph-mon[103029]: 14.14 scrub starts 2024-09-14T04:28:59.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:28:59.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:28:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:28:59.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:00.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:00.096 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:00.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:00.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:00.234 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:00.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:00.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:00.374 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:00.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:00.379 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:29:00.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:00.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:00.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:00.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total time run: 90.0086 2024-09-14T04:29:00.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Total writes made: 137783 2024-09-14T04:29:00.551 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Write size: 65536 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Object size: 65536 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Bandwidth (MB/sec): 95.6735 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Bandwidth: 44.3543 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max bandwidth (MB/sec): 230.312 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min bandwidth (MB/sec): 0 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average IOPS: 1530 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev IOPS: 709.678 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max IOPS: 3685 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min IOPS: 0 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Average Latency(s): 0.0104482 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Stddev Latency(s): 0.0391172 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Max latency(s): 2.12324 2024-09-14T04:29:00.552 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Min latency(s): 0.00118 2024-09-14T04:29:00.557 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Cleaning up (deleting benchmark objects) 2024-09-14T04:29:00.559 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:29:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:29:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:29:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:00 smithi160 ceph-mon[88407]: pgmap v1539: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 8.1 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 60 MiB/s wr, 953 op/s; 3157/397158 objects degraded (0.795%); 3157/397158 objects misplaced (0.795%); 4.5 MiB/s, 72 objects/s recovering 2024-09-14T04:29:00.559 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:00 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 3157/397158 objects degraded (0.795%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:00.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:00.663 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:00.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:00.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:00.771 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:00 smithi114 ceph-mon[103029]: pgmap v1539: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 8.1 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 60 MiB/s wr, 953 op/s; 3157/397158 objects degraded (0.795%); 3157/397158 objects misplaced (0.795%); 4.5 MiB/s, 72 objects/s recovering 2024-09-14T04:29:00.771 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:00 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 3157/397158 objects degraded (0.795%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:00.771 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:00 smithi114 ceph-mon[106969]: pgmap v1539: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 8.1 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 60 MiB/s wr, 953 op/s; 3157/397158 objects degraded (0.795%); 3157/397158 objects misplaced (0.795%); 4.5 MiB/s, 72 objects/s recovering 2024-09-14T04:29:00.772 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:00 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 3157/397158 objects degraded (0.795%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:00.786 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:00.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:00.887 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:00.920 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:00.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:01.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:01.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:01.058 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:01.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:01.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:01.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:01.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:01.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:01.324 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:01.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:01.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:01.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:01.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:01.563 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:01.584 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:01.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:01.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:01 smithi114 ceph-mon[103029]: 14.10 scrub starts 2024-09-14T04:29:01.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:01 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-14T04:29:01.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:29:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:29:01] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-14T04:29:01.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:01 smithi114 ceph-mon[106969]: 14.10 scrub starts 2024-09-14T04:29:01.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:01 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-14T04:29:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:01 smithi160 ceph-mon[88407]: 14.10 scrub starts 2024-09-14T04:29:01.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:01 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-14T04:29:01.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:01.709 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:01.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:01.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:01.834 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:01.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:01.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:01.958 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:01.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:02.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:02.082 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:02.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:02.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:02.205 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:02.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:02.307 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:02.329 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:02.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:02.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:02.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:02.452 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:02.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:02.553 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:02.575 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:02.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:02 smithi160 ceph-mon[88407]: pgmap v1540: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 93 MiB/s wr, 1.48k op/s; 5930/409881 objects misplaced (1.447%); 6.0 MiB/s, 96 objects/s recovering 2024-09-14T04:29:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:02 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3157/397158 objects degraded (0.795%), 1 pg degraded) 2024-09-14T04:29:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:02 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-14T04:29:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:02 smithi160 ceph-mon[88407]: osdmap e410: 8 total, 8 up, 5 in 2024-09-14T04:29:02.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:02 smithi160 ceph-mon[88407]: 14.10 scrub ok 2024-09-14T04:29:02.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:02.697 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:02.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:02.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:02.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:02.821 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:02.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:02.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:02 smithi114 ceph-mon[106969]: pgmap v1540: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 93 MiB/s wr, 1.48k op/s; 5930/409881 objects misplaced (1.447%); 6.0 MiB/s, 96 objects/s recovering 2024-09-14T04:29:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:02 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3157/397158 objects degraded (0.795%), 1 pg degraded) 2024-09-14T04:29:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:02 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-14T04:29:02.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:02 smithi114 ceph-mon[106969]: osdmap e410: 8 total, 8 up, 5 in 2024-09-14T04:29:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:02 smithi114 ceph-mon[106969]: 14.10 scrub ok 2024-09-14T04:29:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:02 smithi114 ceph-mon[103029]: pgmap v1540: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 93 MiB/s wr, 1.48k op/s; 5930/409881 objects misplaced (1.447%); 6.0 MiB/s, 96 objects/s recovering 2024-09-14T04:29:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:02 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3157/397158 objects degraded (0.795%), 1 pg degraded) 2024-09-14T04:29:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:02 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-14T04:29:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:02 smithi114 ceph-mon[103029]: osdmap e410: 8 total, 8 up, 5 in 2024-09-14T04:29:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:02 smithi114 ceph-mon[103029]: 14.10 scrub ok 2024-09-14T04:29:02.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:02.945 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:02.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:03.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:03.069 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:03.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:03.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:03.191 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:03.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:03.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:03.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:03.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:03.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:03.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:03.445 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:03.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:03.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:03.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:03.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:03.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:03.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:03.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:03.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:03.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:03.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:03.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:03 smithi114 ceph-mon[106969]: osdmap e411: 8 total, 8 up, 5 in 2024-09-14T04:29:03.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-14T04:29:03.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:03 smithi114 ceph-mon[103029]: osdmap e411: 8 total, 8 up, 5 in 2024-09-14T04:29:03.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-14T04:29:03.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:03 smithi160 ceph-mon[88407]: osdmap e411: 8 total, 8 up, 5 in 2024-09-14T04:29:03.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-14T04:29:03.934 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:03.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:03.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:04.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:04.078 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:04.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:04.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:04.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:04.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:04.303 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:04.325 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:04.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:04.427 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:04.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:04.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:04.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:04.572 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:04.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:04.673 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:04.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:04.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:04.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:04 smithi114 ceph-mon[103029]: pgmap v1543: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 48 MiB/s wr, 762 op/s; 5930/409881 objects misplaced (1.447%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:29:04.749 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:04 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-14T04:29:04.749 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:04 smithi114 ceph-mon[103029]: 14.13 starting backfill to osd.2 from (0'0,0'0] MAX to 409'6421 2024-09-14T04:29:04.749 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:04 smithi114 ceph-mon[103029]: osdmap e412: 8 total, 8 up, 5 in 2024-09-14T04:29:04.749 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:04 smithi114 ceph-mon[106969]: pgmap v1543: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 48 MiB/s wr, 762 op/s; 5930/409881 objects misplaced (1.447%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:29:04.749 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:04 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-14T04:29:04.749 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:04 smithi114 ceph-mon[106969]: 14.13 starting backfill to osd.2 from (0'0,0'0] MAX to 409'6421 2024-09-14T04:29:04.749 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:04 smithi114 ceph-mon[106969]: osdmap e412: 8 total, 8 up, 5 in 2024-09-14T04:29:04.757 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:29:04.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:04.818 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:04.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:04 smithi160 ceph-mon[88407]: pgmap v1543: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 48 MiB/s wr, 762 op/s; 5930/409881 objects misplaced (1.447%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:29:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:04 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-14T04:29:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:04 smithi160 ceph-mon[88407]: 14.13 starting backfill to osd.2 from (0'0,0'0] MAX to 409'6421 2024-09-14T04:29:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:04 smithi160 ceph-mon[88407]: osdmap e412: 8 total, 8 up, 5 in 2024-09-14T04:29:04.920 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:04.942 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:04.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:05.043 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:05.067 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:05.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:05.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:05.192 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:05.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:05.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:05.317 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:05.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:05.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:05.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:05.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:05.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:05.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:05.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:05.570 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:05.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:05.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:05 smithi114 ceph-mon[106969]: osdmap e413: 8 total, 8 up, 5 in 2024-09-14T04:29:05.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:05 smithi114 ceph-mon[103029]: osdmap e413: 8 total, 8 up, 5 in 2024-09-14T04:29:05.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:05.705 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:05.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:05.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:05.845 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:05.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:05 smithi160 ceph-mon[88407]: osdmap e413: 8 total, 8 up, 5 in 2024-09-14T04:29:05.947 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:05.982 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:05.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:06.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:06.118 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:06.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:06.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:06.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:06.254 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:06.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:06.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:06.384 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:06.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:06.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:06.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:06.521 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:06.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:06.623 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:06.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:06.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:06.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:06.758 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:06.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:29:06.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:06.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:06.830 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:06 smithi114 ceph-mon[106969]: pgmap v1546: 109 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 255 B/s rd, 0 op/s; 9419/408702 objects misplaced (2.305%) 2024-09-14T04:29:06.830 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:06 smithi114 ceph-mon[106969]: osdmap e414: 8 total, 8 up, 5 in 2024-09-14T04:29:06.830 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:06 smithi114 ceph-mon[103029]: pgmap v1546: 109 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 255 B/s rd, 0 op/s; 9419/408702 objects misplaced (2.305%) 2024-09-14T04:29:06.830 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:06 smithi114 ceph-mon[103029]: osdmap e414: 8 total, 8 up, 5 in 2024-09-14T04:29:06.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:06.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:06 smithi160 ceph-mon[88407]: pgmap v1546: 109 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 255 B/s rd, 0 op/s; 9419/408702 objects misplaced (2.305%) 2024-09-14T04:29:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:06 smithi160 ceph-mon[88407]: osdmap e414: 8 total, 8 up, 5 in 2024-09-14T04:29:06.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:06.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:06.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:07.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:07.087 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:07.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:07.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:07.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:07.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:07.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:07.362 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:07.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:07.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:07.497 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:07.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:07.600 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:07.633 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:07.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:07.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:07.768 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:07.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:07.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:07.904 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:07.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:08.008 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:08.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:08.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:08.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:08.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:08.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:08.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:08.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:08.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:08.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "size": 20, 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 600, 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:167869 13.d 13:b00b4514:::benchmark_data_smithi114_288325_object72804:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e365)", 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:25:53.295087+0000", 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 194.61138679999999, 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6035530549999999, 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 167869 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295087+0000", 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295087+0000", 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295087+0000", 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9299999999999999e-07 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295089+0000", 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5039999999999999e-06 2024-09-14T04:29:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295094+0000", 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0200000000000002e-06 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295098+0000", 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9700000000000001e-06 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295313+0000", 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000214682 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295318+0000", 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1100000000000002e-06 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.870000+0000", 2024-09-14T04:29:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.574682446 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.870623+0000", 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00062337099999999995 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4,5", 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.871395+0000", 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00077126099999999997 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.882119+0000", 2024-09-14T04:29:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.010724094 2024-09-14T04:29:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.882209+0000", 2024-09-14T04:29:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.0181999999999998e-05 2024-09-14T04:29:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.882424+0000", 2024-09-14T04:29:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000214985 2024-09-14T04:29:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.897552+0000", 2024-09-14T04:29:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015127936999999999 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.897808+0000", 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000255894 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.898640+0000", 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00083220600000000005 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:167875 13.d 13:b49abc78:::benchmark_data_smithi114_288325_object72810:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e365)", 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:25:53.295103+0000", 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 194.61137082900001, 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6047621809999999, 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:29:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 167875 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295103+0000", 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295103+0000", 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295103+0000", 2024-09-14T04:29:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.0100000000000001e-07 2024-09-14T04:29:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295105+0000", 2024-09-14T04:29:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.53e-06 2024-09-14T04:29:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295110+0000", 2024-09-14T04:29:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8570000000000003e-06 2024-09-14T04:29:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295114+0000", 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0799999999999999e-06 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295335+0000", 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000221117 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295340+0000", 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9980000000000003e-06 2024-09-14T04:29:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.874083+0000", 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5787434789999999 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.874684+0000", 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00060070900000000005 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4,5", 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.875361+0000", 2024-09-14T04:29:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00067701999999999997 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.882253+0000", 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0068919799999999998 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.882628+0000", 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00037491599999999999 2024-09-14T04:29:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.882859+0000", 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00023148299999999999 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.898803+0000", 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015943743 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.899061+0000", 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025792000000000002 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.899865+0000", 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00080404800000000002 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:167876 13.d 13:b175965f:::benchmark_data_smithi114_288325_object72811:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e365)", 2024-09-14T04:29:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:25:53.295119+0000", 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 194.61135527299999, 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6063566709999999, 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 167876 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295119+0000", 2024-09-14T04:29:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295119+0000", 2024-09-14T04:29:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295119+0000", 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.9999999999999999e-07 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295121+0000", 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8929999999999999e-06 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295125+0000", 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6990000000000002e-06 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295129+0000", 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.839e-06 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295357+0000", 2024-09-14T04:29:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000227848 2024-09-14T04:29:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295362+0000", 2024-09-14T04:29:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.062e-06 2024-09-14T04:29:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.877610+0000", 2024-09-14T04:29:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.58224747 2024-09-14T04:29:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.878176+0000", 2024-09-14T04:29:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00056616499999999998 2024-09-14T04:29:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4,5", 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.878931+0000", 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00075519499999999995 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.882948+0000", 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0040167529999999996 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.888670+0000", 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0057219319999999999 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.888868+0000", 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000198722 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.900451+0000", 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011582245 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.900692+0000", 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000241046 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.901475+0000", 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00078350200000000005 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.46760.0:167881 13.d 13:b08b2dbe:::benchmark_data_smithi114_288325_object72816:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e365)", 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:25:53.295134+0000", 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 194.611339759, 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.607552495, 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.46760", 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:29:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 167881 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295134+0000", 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295134+0000", 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295134+0000", 2024-09-14T04:29:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.1800000000000002e-07 2024-09-14T04:29:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295136+0000", 2024-09-14T04:29:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5e-06 2024-09-14T04:29:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295140+0000", 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6469999999999997e-06 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295144+0000", 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.9319999999999998e-06 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295387+0000", 2024-09-14T04:29:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00024238999999999999 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:53.295393+0000", 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8660000000000002e-06 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.883623+0000", 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5882305410000002 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.884229+0000", 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00060627100000000002 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 1,4,5", 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.884992+0000", 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00076255500000000005 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.888948+0000", 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039562290000000003 2024-09-14T04:29:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.889111+0000", 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00016254299999999999 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.897321+0000", 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0082102330000000008 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.901625+0000", 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0043040889999999997 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.901879+0000", 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00025369200000000002 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:25:54.902687+0000", 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000807689 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139057 14.14 14:286155e3:::benchmark_data_smithi114_329146_object1271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.514980+0000", 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.3914941580000004, 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0436499279999998, 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139057 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.514980+0000", 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.514980+0000", 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.514981+0000", 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.268e-06 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.514985+0000", 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1779999999999996e-06 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.514996+0000", 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0577999999999999e-05 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.515001+0000", 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1830000000000002e-06 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.515053+0000", 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.2228999999999997e-05 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.515061+0000", 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8269999999999998e-06 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.419611+0000", 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.90454986999999998 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.419623+0000", 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2489e-05 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.549991+0000", 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1303674429999999 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.550138+0000", 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00014744399999999999 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.550504+0000", 2024-09-14T04:29:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000365476 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.556113+0000", 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0056093699999999998 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.556123+0000", 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0434999999999999e-05 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.556165+0000", 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1763e-05 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.556214+0000", 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8686000000000001e-05 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.558630+0000", 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0024156889999999999 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139068 14.14 14:2a88ddf3:::benchmark_data_smithi114_329146_object1282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.528567+0000", 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.3779066540000002, 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.0343132339999999, 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139068 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.528567+0000", 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.528567+0000", 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.528568+0000", 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2750000000000001e-06 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.528573+0000", 2024-09-14T04:29:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4220000000000002e-06 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.528584+0000", 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1090000000000001e-05 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.528591+0000", 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8870000000000003e-06 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.528676+0000", 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4882000000000005e-05 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.528687+0000", 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1513e-05 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420139+0000", 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.89145137100000005 2024-09-14T04:29:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420148+0000", 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1789999999999997e-06 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.554037+0000", 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.133889302 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.554141+0000", 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000104385 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.554490+0000", 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000348427 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.560900+0000", 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0064100160000000002 2024-09-14T04:29:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.560960+0000", 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0427999999999998e-05 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.560976+0000", 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5238999999999999e-05 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.561027+0000", 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.1125999999999998e-05 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.562880+0000", 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018536920000000001 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139181 14.14 14:2dd45d8a:::benchmark_data_smithi114_329146_object1395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.638784+0000", 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.2676902730000004, 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.928275561, 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139181 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.638784+0000", 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.638784+0000", 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.638784+0000", 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0100000000000005e-07 2024-09-14T04:29:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.638786+0000", 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.1519999999999999e-06 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.638793+0000", 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5980000000000002e-06 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.638798+0000", 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4890000000000003e-06 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.639138+0000", 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00033967399999999999 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.639146+0000", 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.4400000000000005e-06 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420188+0000", 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.78104164399999998 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420199+0000", 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0706999999999999e-05 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.558692+0000", 2024-09-14T04:29:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1384934229999999 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.558796+0000", 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00010408000000000001 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.559195+0000", 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00039905300000000002 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.564954+0000", 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0057586060000000003 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.564971+0000", 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.6895e-05 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.564997+0000", 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.6186e-05 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.565034+0000", 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7214e-05 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.567059+0000", 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020248990000000001 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139189 14.14 14:2c2257ff:::benchmark_data_smithi114_329146_object1403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.643946+0000", 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.2625280730000004, 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9312714309999999, 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139189 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.643946+0000", 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.643946+0000", 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.643947+0000", 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5799999999999998e-07 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.643951+0000", 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.0319999999999997e-06 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.643962+0000", 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1482e-05 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.643970+0000", 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.515e-06 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.644570+0000", 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000600692 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.644579+0000", 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.5690000000000005e-06 2024-09-14T04:29:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420230+0000", 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.77565076200000005 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420237+0000", 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7109999999999995e-06 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.562936+0000", 2024-09-14T04:29:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.142698778 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.563044+0000", 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000107808 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.563360+0000", 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00031619200000000002 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.569287+0000", 2024-09-14T04:29:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0059271030000000004 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.569334+0000", 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6717e-05 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.573050+0000", 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0037157760000000001 2024-09-14T04:29:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.573111+0000", 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1779000000000006e-05 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.575217+0000", 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021056569999999999 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139216 14.14 14:2b215c42:::benchmark_data_smithi114_329146_object1430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.664193+0000", 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.2422803399999998, 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9213482690000001, 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139216 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.664193+0000", 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.664193+0000", 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.664194+0000", 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.7700000000000004e-07 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.664197+0000", 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.8449999999999999e-06 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.664205+0000", 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0139999999999998e-06 2024-09-14T04:29:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.664210+0000", 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5639999999999996e-06 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.664628+0000", 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00041726799999999999 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.664637+0000", 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2639999999999997e-06 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420342+0000", 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.75570465600000003 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420349+0000", 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2200000000000003e-06 2024-09-14T04:29:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.567106+0000", 2024-09-14T04:29:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1467573899999999 2024-09-14T04:29:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.567207+0000", 2024-09-14T04:29:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000100933 2024-09-14T04:29:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.567599+0000", 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00039158299999999998 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.579213+0000", 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.011614352 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.579258+0000", 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.4827999999999998e-05 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.583378+0000", 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0041197359999999997 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.583438+0000", 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.0701e-05 2024-09-14T04:29:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.585542+0000", 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021033380000000002 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139236 14.14 14:2dcf7b55:::benchmark_data_smithi114_329146_object1450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.680698+0000", 2024-09-14T04:29:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.2257755369999996, 2024-09-14T04:29:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.9064602150000001, 2024-09-14T04:29:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139236 2024-09-14T04:29:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.680698+0000", 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.680698+0000", 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.680699+0000", 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0330000000000001e-06 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.680704+0000", 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9609999999999996e-06 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.680716+0000", 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1551e-05 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.680723+0000", 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.951e-06 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.680791+0000", 2024-09-14T04:29:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8713000000000004e-05 2024-09-14T04:29:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.680802+0000", 2024-09-14T04:29:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0206e-05 2024-09-14T04:29:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420384+0000", 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.73958238700000001 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420392+0000", 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.9219999999999997e-06 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.569366+0000", 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1489743969999999 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.569517+0000", 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000151111 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.569783+0000", 2024-09-14T04:29:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00026587200000000001 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.579222+0000", 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0094385030000000009 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.579313+0000", 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1014000000000006e-05 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.585606+0000", 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0062927549999999997 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.585653+0000", 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.7067000000000002e-05 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.587158+0000", 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0015057720000000001 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139261 14.14 14:2f58bb91:::benchmark_data_smithi114_329146_object1475:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.703584+0000", 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.2028897389999997, 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8967905059999999, 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139261 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.703584+0000", 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.703584+0000", 2024-09-14T04:29:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.703586+0000", 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.587e-06 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.703590+0000", 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3610000000000003e-06 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.703601+0000", 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1147e-05 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.703607+0000", 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3509999999999998e-06 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.703675+0000", 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.7039000000000006e-05 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.703686+0000", 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0961999999999999e-05 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420424+0000", 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.71673812199999998 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420431+0000", 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4630000000000004e-06 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.571303+0000", 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1508720160000001 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.571414+0000", 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00011132 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.571637+0000", 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00022293599999999999 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.579275+0000", 2024-09-14T04:29:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0076377550000000004 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.583244+0000", 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039686959999999999 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.598469+0000", 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.015225272 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.598519+0000", 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.0158000000000003e-05 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.600375+0000", 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018553210000000001 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139269 14.14 14:2dc32470:::benchmark_data_smithi114_329146_object1483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.708374+0000", 2024-09-14T04:29:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.198099515, 2024-09-14T04:29:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.897026879, 2024-09-14T04:29:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139269 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.708374+0000", 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.708374+0000", 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.708375+0000", 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2200000000000003e-07 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.708378+0000", 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.689e-06 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.708388+0000", 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8989999999999996e-06 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.708394+0000", 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5270000000000004e-06 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.709232+0000", 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00083837600000000005 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.709241+0000", 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7630000000000006e-06 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420458+0000", 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.71121694199999996 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420465+0000", 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.2420000000000003e-06 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.575262+0000", 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1547968790000001 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.575361+0000", 2024-09-14T04:29:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8838999999999993e-05 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.575679+0000", 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00031750700000000002 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.581291+0000", 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0056119509999999996 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.583308+0000", 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.002017643 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.603304+0000", 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.019995472 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.603359+0000", 2024-09-14T04:29:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.5284999999999997e-05 2024-09-14T04:29:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.605401+0000", 2024-09-14T04:29:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0020421430000000002 2024-09-14T04:29:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139286 14.14 14:2b77bdc6:::benchmark_data_smithi114_329146_object1500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.718802+0000", 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.1876719199999997, 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8885359209999999, 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139286 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.718802+0000", 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.718802+0000", 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.718802+0000", 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9999999999999997e-07 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.718806+0000", 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.2219999999999998e-06 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.718825+0000", 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8995999999999999e-05 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.718832+0000", 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4050000000000003e-06 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.718907+0000", 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.4554999999999993e-05 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.718916+0000", 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.8300000000000008e-06 2024-09-14T04:29:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420491+0000", 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.70157498600000001 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420499+0000", 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.0600000000000002e-06 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.577488+0000", 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.156989762 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.577644+0000", 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00015586800000000001 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.577954+0000", 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00030959299999999999 2024-09-14T04:29:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.583257+0000", 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.005303655 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.587220+0000", 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0039628379999999998 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.605461+0000", 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.018240321 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.605517+0000", 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6570999999999998e-05 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.607338+0000", 2024-09-14T04:29:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018206590000000001 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139295 14.14 14:2dc01352:::benchmark_data_smithi114_329146_object1509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.729829+0000", 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.1766449960000003, 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.879322471, 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139295 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.729829+0000", 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.729829+0000", 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.729830+0000", 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.333e-06 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.729835+0000", 2024-09-14T04:29:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.5689999999999998e-06 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.729844+0000", 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.1710000000000008e-06 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.729852+0000", 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.8180000000000004e-06 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.729900+0000", 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8281000000000001e-05 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.729911+0000", 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1388e-05 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420536+0000", 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.69062509400000005 2024-09-14T04:29:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420544+0000", 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.109e-06 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.579352+0000", 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1588081770000001 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.579503+0000", 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000151001 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.579763+0000", 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00026047599999999999 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.585562+0000", 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0057988969999999999 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.587265+0000", 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017026770000000001 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.607403+0000", 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.020137774000000001 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.607461+0000", 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.8298000000000003e-05 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.609151+0000", 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.001690408 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139309 14.14 14:2f974f88:::benchmark_data_smithi114_329146_object1523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.742594+0000", 2024-09-14T04:29:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.1638802579999998, 2024-09-14T04:29:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8683846119999998, 2024-09-14T04:29:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139309 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.742594+0000", 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.742594+0000", 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.742594+0000", 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.7899999999999997e-07 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.742599+0000", 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.3939999999999998e-06 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.742610+0000", 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1430999999999999e-05 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.742636+0000", 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 2.5718e-05 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.742691+0000", 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4868999999999999e-05 2024-09-14T04:29:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.742700+0000", 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.2010000000000005e-06 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420578+0000", 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.67787808599999999 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420586+0000", 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.0120000000000005e-06 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.581332+0000", 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1607461720000001 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.581469+0000", 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013676300000000001 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.581680+0000", 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021051899999999999 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.587176+0000", 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0054959309999999999 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.587310+0000", 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013474200000000001 2024-09-14T04:29:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.609207+0000", 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.021896270999999998 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.609266+0000", 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.9395999999999998e-05 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.610978+0000", 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0017122280000000001 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139326 14.14 14:2e01eb66:::benchmark_data_smithi114_329146_object1540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.759098+0000", 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.1473758109999999, 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8549977690000001, 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139326 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.759098+0000", 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.759098+0000", 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.759099+0000", 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.2839999999999999e-06 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.759104+0000", 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.25e-06 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.759117+0000", 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.3047999999999999e-05 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.759123+0000", 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.8109999999999997e-06 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.759185+0000", 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1820000000000001e-05 2024-09-14T04:29:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.759196+0000", 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1249e-05 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420619+0000", 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.66142222900000003 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420625+0000", 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.5529999999999998e-06 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.587342+0000", 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.166716595 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.587475+0000", 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013361300000000001 2024-09-14T04:29:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.587886+0000", 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00041025399999999998 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.593487+0000", 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0056009709999999997 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.593548+0000", 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.1767999999999995e-05 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.611857+0000", 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.018308919999999999 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.611914+0000", 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6406999999999997e-05 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.614096+0000", 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0021819970000000002 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139328 14.14 14:28ebda45:::benchmark_data_smithi114_329146_object1542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.760547+0000", 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.1459266499999998, 2024-09-14T04:29:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8581003250000001, 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139328 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.760547+0000", 2024-09-14T04:29:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.760547+0000", 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.760548+0000", 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3200000000000005e-07 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.760552+0000", 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.7910000000000002e-06 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.760559+0000", 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.9990000000000002e-06 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.760563+0000", 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.391e-06 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.760609+0000", 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.6223000000000001e-05 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.760615+0000", 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4979999999999997e-06 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420657+0000", 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.66004215899999996 2024-09-14T04:29:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420664+0000", 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.3470000000000001e-06 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.589908+0000", 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1692442060000001 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.590039+0000", 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00013085500000000001 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.590387+0000", 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00034732699999999998 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.594965+0000", 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0045782039999999998 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.603153+0000", 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0081886930000000004 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.616673+0000", 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.013519432 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.616728+0000", 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4931e-05 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.618647+0000", 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0019196370000000001 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139343 14.14 14:28fa2083:::benchmark_data_smithi114_329146_object1557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.777692+0000", 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.128781493, 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.8530002620000001, 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139343 2024-09-14T04:29:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.777692+0000", 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.777692+0000", 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.777693+0000", 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.102e-06 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.777698+0000", 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1679999999999997e-06 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.777708+0000", 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.0207000000000001e-05 2024-09-14T04:29:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.777713+0000", 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.8829999999999997e-06 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.777762+0000", 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9277999999999998e-05 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.777769+0000", 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.1400000000000002e-06 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420696+0000", 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.64292654699999996 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420702+0000", 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.3119999999999999e-06 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.591987+0000", 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.17128465 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.592147+0000", 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000160326 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.592388+0000", 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000240688 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.596664+0000", 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0042767459999999997 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.603215+0000", 2024-09-14T04:29:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0065502670000000002 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.628374+0000", 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.025159519000000002 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.628452+0000", 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 7.7997000000000006e-05 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.630693+0000", 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0022404320000000001 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139381 14.14 14:2fea8b9f:::benchmark_data_smithi114_329146_object1595:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.844801+0000", 2024-09-14T04:29:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.061672314, 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7878972019999999, 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139381 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.844801+0000", 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.844801+0000", 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.844803+0000", 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.578e-06 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.844807+0000", 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.1729999999999997e-06 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.844822+0000", 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.5058999999999999e-05 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.844828+0000", 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.4410000000000001e-06 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.844892+0000", 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4424000000000005e-05 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.844903+0000", 2024-09-14T04:29:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1019000000000001e-05 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420729+0000", 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.57582571000000005 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420732+0000", 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 3.5829999999999998e-06 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.593588+0000", 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.172855269 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.593682+0000", 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 9.4207000000000001e-05 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.593893+0000", 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00021136500000000001 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.603104+0000", 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0092111640000000009 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.603259+0000", 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000154458 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.630761+0000", 2024-09-14T04:29:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.027502109 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.630897+0000", 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000135644 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.632699+0000", 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0018019990000000001 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "osd_op(client.47180.0:139389 14.14 14:299529e9:::benchmark_data_smithi114_329146_object1603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-09-14T04:29:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-09-14T04:29:03.867250+0000", 2024-09-14T04:29:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 4.039223797, 2024-09-14T04:29:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.7682811090000001, 2024-09-14T04:29:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-09-14T04:29:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-09-14T04:29:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "client_info": { 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "client": "client.47180", 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139389 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated", 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.867250+0000", 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read", 2024-09-14T04:29:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.867250+0000", 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled", 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.867252+0000", 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.854e-06 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read", 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.867258+0000", 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 5.6869999999999999e-06 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched", 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.867269+0000", 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1344e-05 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "queued_for_pg", 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.867278+0000", 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 8.8540000000000003e-06 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.867328+0000", 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 4.9879999999999997e-05 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:03.867334+0000", 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.4840000000000001e-06 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420754+0000", 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.553419467 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for peered", 2024-09-14T04:29:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:04.420760+0000", 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 6.46e-06 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "reached_pg", 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.595000+0000", 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 1.1742394890000001 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "started", 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.595145+0000", 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000145088 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.595346+0000", 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.00020105000000000001 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "op_commit", 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.603167+0000", 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0078218070000000001 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.611045+0000", 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0078774729999999994 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.632945+0000", 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.021900000999999999 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "commit_sent", 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.633174+0000", 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.000229149 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "done", 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-09-14T04:29:05.635531+0000", 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 0.0023570219999999998 2024-09-14T04:29:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-14T04:29:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-14T04:29:08.510 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:29:08.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:08.541 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:08.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:08.643 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:08.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:08.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:08.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:08.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:08.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:08.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:08.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:08.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:09.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:09.091 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:09.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:09.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:08 smithi114 ceph-mon[106969]: pgmap v1548: 109 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 212 B/s rd, 0 op/s; 9419/408702 objects misplaced (2.305%) 2024-09-14T04:29:09.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:09.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:09.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:08 smithi114 ceph-mon[103029]: pgmap v1548: 109 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 212 B/s rd, 0 op/s; 9419/408702 objects misplaced (2.305%) 2024-09-14T04:29:09.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:09.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:08 smithi160 ceph-mon[88407]: pgmap v1548: 109 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 212 B/s rd, 0 op/s; 9419/408702 objects misplaced (2.305%) 2024-09-14T04:29:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:09.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:09.240 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:09.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:09.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:09.380 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:09.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:09.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:09.503 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:09.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:09.604 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:09.626 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:09.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:09.728 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:09.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:09.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:09.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:09.865 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:09.898 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:09.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:10.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:10.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:10.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:10.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:10.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:10.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:10.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:10.328 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:10.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:10.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:10.465 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:10.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:10.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:10.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:10.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:29:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:29:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:29:10.704 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:10.742 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:10.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:10.844 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:10.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:10.866 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:10.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:10.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:10 smithi114 ceph-mon[106969]: pgmap v1549: 109 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 170 B/s rd, 0 op/s; 9419/406653 objects misplaced (2.316%) 2024-09-14T04:29:10.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:10 smithi114 ceph-mon[103029]: pgmap v1549: 109 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 170 B/s rd, 0 op/s; 9419/406653 objects misplaced (2.316%) 2024-09-14T04:29:10.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:11.001 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:11.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:11.103 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:11.141 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:11.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:10 smithi160 ceph-mon[88407]: pgmap v1549: 109 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.3 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 170 B/s rd, 0 op/s; 9419/406653 objects misplaced (2.316%) 2024-09-14T04:29:11.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:11.276 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:11.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:11.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:11.416 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:11.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:11.517 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:11.549 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:11.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:11.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:11.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:11.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:11.721 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:29:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:29:11] "GET /metrics HTTP/1.1" 200 36560 "" "Prometheus/2.51.0" 2024-09-14T04:29:11.795 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:11.827 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:11.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:11.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:11.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:11.964 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:11.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:12.003 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:11 smithi114 ceph-mon[106969]: Health check failed: Degraded data redundancy: 4216/393189 objects degraded (1.072%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:12.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:11 smithi114 ceph-mon[103029]: Health check failed: Degraded data redundancy: 4216/393189 objects degraded (1.072%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:12.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-14T04:29:12.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:12.083 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:11 smithi160 ceph-mon[88407]: Health check failed: Degraded data redundancy: 4216/393189 objects degraded (1.072%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:12.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:12.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:12.236 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:12.267 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:12.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:12.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:12.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:12.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:12.497 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:12.502 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:12.529 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:12.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:12.630 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:12.674 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:12.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:12.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:12.815 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:12.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:12.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:12 smithi114 ceph-mon[106969]: pgmap v1550: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.0 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 131 B/s rd, 0 B/s wr, 483 op/s; 4216/393189 objects degraded (1.072%); 8765/393189 objects misplaced (2.229%) 2024-09-14T04:29:12.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:12 smithi114 ceph-mon[103029]: pgmap v1550: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.0 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 131 B/s rd, 0 B/s wr, 483 op/s; 4216/393189 objects degraded (1.072%); 8765/393189 objects misplaced (2.229%) 2024-09-14T04:29:12.899 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:12 smithi160 ceph-mon[88407]: pgmap v1550: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.0 GiB data, 31 GiB used, 416 GiB / 447 GiB avail; 131 B/s rd, 0 B/s wr, 483 op/s; 4216/393189 objects degraded (1.072%); 8765/393189 objects misplaced (2.229%) 2024-09-14T04:29:12.916 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:12.945 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:29:12.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:12.953 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:12.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:13.054 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:13.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:13.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:13.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:13.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:13.235 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:13.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:13.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:13.371 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:13.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:13.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:13.506 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:13.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:13.607 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:13.640 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:13.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:13.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:13.789 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:13.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:13.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:13 smithi114 ceph-mon[103029]: pgmap v1551: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.0 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 0 B/s wr, 300 op/s; 4216/393189 objects degraded (1.072%); 8765/393189 objects misplaced (2.229%) 2024-09-14T04:29:13.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:13 smithi114 ceph-mon[106969]: pgmap v1551: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.0 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 0 B/s wr, 300 op/s; 4216/393189 objects degraded (1.072%); 8765/393189 objects misplaced (2.229%) 2024-09-14T04:29:13.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:13.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:13 smithi160 ceph-mon[88407]: pgmap v1551: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 8.0 GiB data, 30 GiB used, 417 GiB / 447 GiB avail; 0 B/s wr, 300 op/s; 4216/393189 objects degraded (1.072%); 8765/393189 objects misplaced (2.229%) 2024-09-14T04:29:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:13.929 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:13.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:14.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:14.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:14.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:14.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:14.203 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:14.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:14.304 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:14.344 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:14.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:14.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:14.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:14.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:14.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:14.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:14.622 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:14.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:14.724 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:14.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:14.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:14.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:14.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:14.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:14.974 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:29:14.974 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:29:14.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.974 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.47180.0:151500 14.a 14:53cac391:::benchmark_data_smithi114_329146_object13714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-14T04:29:14.974 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:14.947480+0000", 2024-09-14T04:29:14.974 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0061228080000000004, 2024-09-14T04:29:14.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0061630189999999996, 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.47180", 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 151500 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.947480+0000", 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.947480+0000", 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.947478+0000", 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.947486+0000", 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2409999999999998e-06 2024-09-14T04:29:14.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.947497+0000", 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1355000000000001e-05 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.947503+0000", 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8780000000000003e-06 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:14.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.947640+0000", 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013752 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.947758+0000", 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117516 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,4", 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.947902+0000", 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014446500000000001 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.950596+0000", 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026936019999999998 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:151501 14.5 e414/395 14:a6fef9fb:::benchmark_data_smithi114_329146_object13715:head v 414'6822, mlcod=414'6820)", 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:14.951886+0000", 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0017164140000000001, 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001842347, 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:14.979 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951886+0000", 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951886+0000", 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951885+0000", 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951893+0000", 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2119999999999997e-06 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951903+0000", 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0121e-05 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:14.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951908+0000", 2024-09-14T04:29:14.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1250000000000001e-06 2024-09-14T04:29:14.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:14.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951976+0000", 2024-09-14T04:29:14.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7923999999999995e-05 2024-09-14T04:29:14.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951994+0000", 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8434e-05 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.47180.0:151504 14.a 14.cc112e2a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:14.951955+0000", 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0016475680000000001, 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001828288, 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:14.983 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.47180", 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 151504 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951955+0000", 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951955+0000", 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951954+0000", 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951960+0000", 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1970000000000001e-06 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951971+0000", 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1473e-05 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.951978+0000", 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7610000000000002e-06 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:151502 14.2 e414/395 14:40c70194:::benchmark_data_smithi114_329146_object13716:head v 414'6870, mlcod=414'6868)", 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:14.952392+0000", 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0012107839999999999, 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00144783, 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952392+0000", 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952392+0000", 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952391+0000", 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952397+0000", 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.156e-06 2024-09-14T04:29:14.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952411+0000", 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3393999999999999e-05 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952416+0000", 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.057e-06 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953724+0000", 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001308303 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953741+0000", 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7373000000000001e-05 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.47180.0:151505 14.9 14:961a2768:::benchmark_data_smithi114_329146_object13719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-14T04:29:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:14.952688+0000", 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00091503999999999999, 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012071779999999999, 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.47180", 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 151505 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952688+0000", 2024-09-14T04:29:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952688+0000", 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952687+0000", 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952691+0000", 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.746e-06 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952705+0000", 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4021e-05 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952711+0000", 2024-09-14T04:29:14.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5130000000000004e-06 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952867+0000", 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000156355 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952979+0000", 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011157799999999999 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,4", 2024-09-14T04:29:14.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953409+0000", 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043050899999999998 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.47180.0:151509 14.19 14.afabeb79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:14.952723+0000", 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00087989800000000005, 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001236469, 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:29:14.995 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.47180", 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/3347816065", 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 151509 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952723+0000", 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952723+0000", 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952722+0000", 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952725+0000", 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.486e-06 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952730+0000", 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7420000000000003e-06 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.952734+0000", 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4529999999999998e-06 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.998 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.47180.0:151500 14.a e414/385 ondisk, result = 0)", 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:14.953454+0000", 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00014869500000000001, 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00055188899999999996, 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953454+0000", 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:14.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953454+0000", 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953453+0000", 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953458+0000", 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3540000000000002e-06 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:15.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953465+0000", 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6450000000000005e-06 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953470+0000", 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5950000000000001e-06 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953650+0000", 2024-09-14T04:29:15.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017979700000000001 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:14.953656+0000", 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.336e-06 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 7 2024-09-14T04:29:15.002 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:29:15.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:15.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:15.047 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:15.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:15.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:15.185 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:15.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:15.190 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 14 pg_num to 36 2024-09-14T04:29:15.213 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:15 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1075384623' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "36"} : dispatch 2024-09-14T04:29:15.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:15 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1075384623' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "36"} : dispatch 2024-09-14T04:29:15.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:15.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:15.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:15.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:15.425 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:15.461 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:15.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:15.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:15.600 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:15.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:15 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1075384623' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "36"} : dispatch 2024-09-14T04:29:15.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:15.731 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:15.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:15.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:15.867 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:15.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:15.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:16.008 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:16.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:16.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:16.143 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:16.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:16.244 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:16.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:16.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:16.322 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:16 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1075384623' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "36"}]': finished 2024-09-14T04:29:16.322 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:16 smithi114 ceph-mon[106969]: osdmap e415: 8 total, 8 up, 5 in 2024-09-14T04:29:16.322 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:16 smithi114 ceph-mon[106969]: pgmap v1553: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 4216/373674 objects degraded (1.128%); 8132/373674 objects misplaced (2.176%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:29:16.322 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:16 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-14T04:29:16.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:16 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1075384623' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "36"}]': finished 2024-09-14T04:29:16.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:16 smithi114 ceph-mon[103029]: osdmap e415: 8 total, 8 up, 5 in 2024-09-14T04:29:16.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:16 smithi114 ceph-mon[103029]: pgmap v1553: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 4216/373674 objects degraded (1.128%); 8132/373674 objects misplaced (2.176%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:29:16.323 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:16 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-14T04:29:16.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:16.435 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:16.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:16.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:16.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:16.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:16.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:16 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1075384623' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num", "val": "36"}]': finished 2024-09-14T04:29:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:16 smithi160 ceph-mon[88407]: osdmap e415: 8 total, 8 up, 5 in 2024-09-14T04:29:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:16 smithi160 ceph-mon[88407]: pgmap v1553: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 4216/373674 objects degraded (1.128%); 8132/373674 objects misplaced (2.176%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:29:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:16 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-14T04:29:16.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:16.681 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:16.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:16.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:16.804 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:16.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:16.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:16.928 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:16.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:16.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-14T04:29:17.030 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:17.054 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:17.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:17.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:17.177 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:17.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:17.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:17.268 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:17.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:17.309 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:17.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:17.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:17.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:17.446 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:17.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:17.547 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:17.578 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:17 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-14T04:29:17.578 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:17 smithi160 ceph-mon[88407]: osdmap e416: 8 total, 8 up, 5 in 2024-09-14T04:29:17.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:17.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:17.607 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:17 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-14T04:29:17.607 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:17 smithi114 ceph-mon[106969]: osdmap e416: 8 total, 8 up, 5 in 2024-09-14T04:29:17.608 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:17 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-14T04:29:17.608 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:17 smithi114 ceph-mon[103029]: osdmap e416: 8 total, 8 up, 5 in 2024-09-14T04:29:17.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:17.715 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:17.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:17.784 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:29:17.784 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:29:17.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:17.818 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:29:17.818 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:29:17.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:17.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:17.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:17.849 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:29:17.849 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:29:17.887 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:29:17.887 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:29:17.928 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:29:17.928 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:29:17.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:17.959 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:29:17.959 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:29:17.989 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:17.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:17.991 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:29:17.992 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:29:18.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:29:18.026 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:29:18.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:18.126 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:18.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:18.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:18.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:18.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:18.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:18.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:18.399 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:18.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:18.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:18.497 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[106969]: pgmap v1555: 119 pgs: 10 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 0 B/s wr, 1.37k op/s; 4216/373674 objects degraded (1.128%); 8132/373674 objects misplaced (2.176%); 3.6 MiB/s, 57 objects/s recovering 2024-09-14T04:29:18.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[106969]: osdmap e417: 8 total, 8 up, 5 in 2024-09-14T04:29:18.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[106969]: 14.3 scrub starts 2024-09-14T04:29:18.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[106969]: 14.b scrub starts 2024-09-14T04:29:18.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2833452545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:29:18.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[106969]: 14.3 scrub ok 2024-09-14T04:29:18.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[103029]: pgmap v1555: 119 pgs: 10 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 0 B/s wr, 1.37k op/s; 4216/373674 objects degraded (1.128%); 8132/373674 objects misplaced (2.176%); 3.6 MiB/s, 57 objects/s recovering 2024-09-14T04:29:18.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[103029]: osdmap e417: 8 total, 8 up, 5 in 2024-09-14T04:29:18.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[103029]: 14.3 scrub starts 2024-09-14T04:29:18.498 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[103029]: 14.b scrub starts 2024-09-14T04:29:18.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2833452545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:29:18.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:18 smithi114 ceph-mon[103029]: 14.3 scrub ok 2024-09-14T04:29:18.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:18.539 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:18.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:18 smithi160 ceph-mon[88407]: pgmap v1555: 119 pgs: 10 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 106 active+clean; 7.6 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 0 B/s wr, 1.37k op/s; 4216/373674 objects degraded (1.128%); 8132/373674 objects misplaced (2.176%); 3.6 MiB/s, 57 objects/s recovering 2024-09-14T04:29:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:18 smithi160 ceph-mon[88407]: osdmap e417: 8 total, 8 up, 5 in 2024-09-14T04:29:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:18 smithi160 ceph-mon[88407]: 14.3 scrub starts 2024-09-14T04:29:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:18 smithi160 ceph-mon[88407]: 14.b scrub starts 2024-09-14T04:29:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:18 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2833452545' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:29:18.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:18 smithi160 ceph-mon[88407]: 14.3 scrub ok 2024-09-14T04:29:18.641 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:18.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:18.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:18.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:18.778 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:18.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:18.814 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:18.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:18.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:18.952 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:18.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:19.054 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:19.086 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:19.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:19.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:19.223 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:19.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:19.248 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:29:19.286 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:29:19.286 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:29:19.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:19.365 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:19.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:19.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[106969]: 14.d scrub starts 2024-09-14T04:29:19.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[106969]: 14.f scrub starts 2024-09-14T04:29:19.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[106969]: 14.0 scrub starts 2024-09-14T04:29:19.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[106969]: 14.b scrub ok 2024-09-14T04:29:19.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3302901180' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:29:19.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:29:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[103029]: 14.d scrub starts 2024-09-14T04:29:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[103029]: 14.f scrub starts 2024-09-14T04:29:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[103029]: 14.0 scrub starts 2024-09-14T04:29:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[103029]: 14.b scrub ok 2024-09-14T04:29:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3302901180' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:29:19.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:19 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:29:19.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:19.499 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:19.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:19.601 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:19.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:19.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:19 smithi160 ceph-mon[88407]: 14.d scrub starts 2024-09-14T04:29:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:19 smithi160 ceph-mon[88407]: 14.f scrub starts 2024-09-14T04:29:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:19 smithi160 ceph-mon[88407]: 14.0 scrub starts 2024-09-14T04:29:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:19 smithi160 ceph-mon[88407]: 14.b scrub ok 2024-09-14T04:29:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:19 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3302901180' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:29:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:19 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:29:19.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:19.770 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:19.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:19.871 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:19.907 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:19.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:19.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:20.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:20.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:20.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:20.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-14T04:29:20.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:20.173 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:20.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:20.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:20.303 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:20.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:20.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-mon[88407]: 14.0 scrub ok 2024-09-14T04:29:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-mon[88407]: pgmap v1557: 119 pgs: 8 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 108 active+clean; 7.5 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 4216/371118 objects degraded (1.136%); 8132/371118 objects misplaced (2.191%); 2.3 MiB/s, 36 objects/s recovering 2024-09-14T04:29:20.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4216/371118 objects degraded (1.136%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:29:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:29:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-mon[88407]: osdmap e418: 8 total, 8 up, 5 in 2024-09-14T04:29:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-mon[88407]: 14.1 scrub starts 2024-09-14T04:29:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-mon[88407]: 14.f scrub ok 2024-09-14T04:29:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-mon[88407]: 14.c scrub starts 2024-09-14T04:29:20.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-mon[88407]: 14.1 scrub ok 2024-09-14T04:29:20.409 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:29:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:29:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:29:20.437 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:20.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:20.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:20.484 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:20.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:20.574 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:20.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[106969]: 14.0 scrub ok 2024-09-14T04:29:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[106969]: pgmap v1557: 119 pgs: 8 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 108 active+clean; 7.5 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 4216/371118 objects degraded (1.136%); 8132/371118 objects misplaced (2.191%); 2.3 MiB/s, 36 objects/s recovering 2024-09-14T04:29:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4216/371118 objects degraded (1.136%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:29:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:29:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[106969]: osdmap e418: 8 total, 8 up, 5 in 2024-09-14T04:29:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[106969]: 14.1 scrub starts 2024-09-14T04:29:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[106969]: 14.f scrub ok 2024-09-14T04:29:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[106969]: 14.c scrub starts 2024-09-14T04:29:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[106969]: 14.1 scrub ok 2024-09-14T04:29:20.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[103029]: 14.0 scrub ok 2024-09-14T04:29:20.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[103029]: pgmap v1557: 119 pgs: 8 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 108 active+clean; 7.5 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 0 B/s wr, 1.23k op/s; 4216/371118 objects degraded (1.136%); 8132/371118 objects misplaced (2.191%); 2.3 MiB/s, 36 objects/s recovering 2024-09-14T04:29:20.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4216/371118 objects degraded (1.136%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:20.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:29:20.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:29:20.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[103029]: osdmap e418: 8 total, 8 up, 5 in 2024-09-14T04:29:20.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[103029]: 14.1 scrub starts 2024-09-14T04:29:20.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[103029]: 14.f scrub ok 2024-09-14T04:29:20.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[103029]: 14.c scrub starts 2024-09-14T04:29:20.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:20 smithi114 ceph-mon[103029]: 14.1 scrub ok 2024-09-14T04:29:20.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:20.711 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:20.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:20.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:20.854 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:20.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:20.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:20.991 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:20.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:21.026 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 4, 5, 2] out_osds: [1, 7, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:29:21.026 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:29:21.026 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-14T04:29:21.026 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-14T04:29:21.026 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-14T04:29:21.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-14T04:29:21.093 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:21.112 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:29:21.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:21.120 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:21.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:21.221 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:21.249 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:21.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:21.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:21.351 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:21 smithi114 ceph-mon[103029]: 14.c scrub ok 2024-09-14T04:29:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:21 smithi114 ceph-mon[103029]: 14.1f scrub starts 2024-09-14T04:29:21.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:21 smithi114 ceph-mon[103029]: 14.20 scrub starts 2024-09-14T04:29:21.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:21 smithi114 ceph-mon[106969]: 14.c scrub ok 2024-09-14T04:29:21.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:21 smithi114 ceph-mon[106969]: 14.1f scrub starts 2024-09-14T04:29:21.371 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:21 smithi114 ceph-mon[106969]: 14.20 scrub starts 2024-09-14T04:29:21.385 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:21.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:21.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:21.526 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:21.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:21.628 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:21.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:21 smithi160 ceph-mon[88407]: 14.c scrub ok 2024-09-14T04:29:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:21 smithi160 ceph-mon[88407]: 14.1f scrub starts 2024-09-14T04:29:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:21 smithi160 ceph-mon[88407]: 14.20 scrub starts 2024-09-14T04:29:21.659 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:21.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:21.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:21.761 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:21.769 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:29:21.793 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:29:21.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:21.820 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:21.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:21.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:29:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:29:21] "GET /metrics HTTP/1.1" 200 36560 "" "Prometheus/2.51.0" 2024-09-14T04:29:21.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:21.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:21.951 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:21.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:21.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:29:22.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:22.082 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:22.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:22.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:22.216 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:22.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:22.317 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:22.351 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:22.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:22.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:22.491 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:22.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:22.593 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[106969]: pgmap v1559: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 113 active+clean; 7.3 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 4216/356436 objects degraded (1.183%); 7389/356436 objects misplaced (2.073%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:29:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[106969]: 14.20 scrub ok 2024-09-14T04:29:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[106969]: 14.21 scrub starts 2024-09-14T04:29:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[106969]: 14.1f scrub ok 2024-09-14T04:29:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[106969]: 14.e scrub starts 2024-09-14T04:29:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[106969]: 14.21 scrub ok 2024-09-14T04:29:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[103029]: pgmap v1559: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 113 active+clean; 7.3 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 4216/356436 objects degraded (1.183%); 7389/356436 objects misplaced (2.073%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:29:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[103029]: 14.20 scrub ok 2024-09-14T04:29:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[103029]: 14.21 scrub starts 2024-09-14T04:29:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[103029]: 14.1f scrub ok 2024-09-14T04:29:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[103029]: 14.e scrub starts 2024-09-14T04:29:22.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:22 smithi114 ceph-mon[103029]: 14.21 scrub ok 2024-09-14T04:29:22.627 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:22.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:22 smithi160 ceph-mon[88407]: pgmap v1559: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 113 active+clean; 7.3 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 4216/356436 objects degraded (1.183%); 7389/356436 objects misplaced (2.073%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:29:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:22 smithi160 ceph-mon[88407]: 14.20 scrub ok 2024-09-14T04:29:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:22 smithi160 ceph-mon[88407]: 14.21 scrub starts 2024-09-14T04:29:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:22 smithi160 ceph-mon[88407]: 14.1f scrub ok 2024-09-14T04:29:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:22 smithi160 ceph-mon[88407]: 14.e scrub starts 2024-09-14T04:29:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:22 smithi160 ceph-mon[88407]: 14.21 scrub ok 2024-09-14T04:29:22.729 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:22.766 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:22.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:22.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:22.907 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:22.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:23.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:23.046 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:23.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:23.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:23.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:23.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:23.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:23.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:23.331 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:23.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:23.432 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:23.455 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:23.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:23.557 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:23.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:23.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:23.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:23 smithi114 ceph-mon[106969]: 14.2 scrub starts 2024-09-14T04:29:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:23 smithi114 ceph-mon[106969]: 14.1c scrub starts 2024-09-14T04:29:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:23 smithi114 ceph-mon[106969]: 14.e scrub ok 2024-09-14T04:29:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:23 smithi114 ceph-mon[103029]: 14.2 scrub starts 2024-09-14T04:29:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:23 smithi114 ceph-mon[103029]: 14.1c scrub starts 2024-09-14T04:29:23.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:23 smithi114 ceph-mon[103029]: 14.e scrub ok 2024-09-14T04:29:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:23 smithi160 ceph-mon[88407]: 14.2 scrub starts 2024-09-14T04:29:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:23 smithi160 ceph-mon[88407]: 14.1c scrub starts 2024-09-14T04:29:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:23 smithi160 ceph-mon[88407]: 14.e scrub ok 2024-09-14T04:29:23.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:23.736 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:23.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:23.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:23.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:23.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:23.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:24.034 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:24.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:24.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:24.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:24.178 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:24.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:24.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:24.321 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:24.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:24.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:24.530 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:29:24.530 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:29:24.555 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:24.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:24.657 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:24.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:24.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:24.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:24.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:24.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:24.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:24 smithi160 ceph-mon[88407]: pgmap v1560: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 113 active+clean; 7.3 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 4216/356436 objects degraded (1.183%); 7389/356436 objects misplaced (2.073%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:29:24.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:24 smithi160 ceph-mon[88407]: 14.1b scrub starts 2024-09-14T04:29:24.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:24 smithi160 ceph-mon[88407]: 14.1c scrub ok 2024-09-14T04:29:24.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:24 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4216/356436 objects degraded (1.183%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:24.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:24.964 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:24.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:25.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:25.083 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:24 smithi114 ceph-mon[103029]: pgmap v1560: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 113 active+clean; 7.3 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 4216/356436 objects degraded (1.183%); 7389/356436 objects misplaced (2.073%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:29:25.083 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:24 smithi114 ceph-mon[103029]: 14.1b scrub starts 2024-09-14T04:29:25.083 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:24 smithi114 ceph-mon[103029]: 14.1c scrub ok 2024-09-14T04:29:25.083 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:24 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4216/356436 objects degraded (1.183%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:25.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:24 smithi114 ceph-mon[106969]: pgmap v1560: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 113 active+clean; 7.3 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 4216/356436 objects degraded (1.183%); 7389/356436 objects misplaced (2.073%); 2.4 MiB/s, 38 objects/s recovering 2024-09-14T04:29:25.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:24 smithi114 ceph-mon[106969]: 14.1b scrub starts 2024-09-14T04:29:25.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:24 smithi114 ceph-mon[106969]: 14.1c scrub ok 2024-09-14T04:29:25.084 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:24 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4216/356436 objects degraded (1.183%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:25.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:25.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:25.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-14T04:29:25.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:25.239 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:25.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:25.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:25.383 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:25.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:25.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:25.534 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:25.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:25.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:25.628 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:25.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:25.677 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:25.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:25.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:25.816 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:25.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:25.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:25.957 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:25 smithi160 ceph-mon[88407]: 14.1b scrub ok 2024-09-14T04:29:25.957 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:25 smithi160 ceph-mon[88407]: 14.1e scrub starts 2024-09-14T04:29:25.962 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:25.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:26.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:26.102 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:26.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:26.115 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:25 smithi114 ceph-mon[106969]: 14.1b scrub ok 2024-09-14T04:29:26.115 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:25 smithi114 ceph-mon[106969]: 14.1e scrub starts 2024-09-14T04:29:26.116 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:25 smithi114 ceph-mon[103029]: 14.1b scrub ok 2024-09-14T04:29:26.116 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:25 smithi114 ceph-mon[103029]: 14.1e scrub starts 2024-09-14T04:29:26.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:26.241 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:26.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:26.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:26.380 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:26.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:26.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:26.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:26.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:26.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:26.657 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:26.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:26.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:26.797 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:26.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:26.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:26.931 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:26.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:27.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:27.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:27.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:27.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:26 smithi160 ceph-mon[88407]: pgmap v1561: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 114 active+clean; 6.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 4216/334692 objects degraded (1.260%); 6700/334692 objects misplaced (2.002%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:29:27.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:26 smithi160 ceph-mon[88407]: 14.1d deep-scrub starts 2024-09-14T04:29:27.094 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:26 smithi160 ceph-mon[88407]: 14.1e scrub ok 2024-09-14T04:29:27.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:26 smithi114 ceph-mon[103029]: pgmap v1561: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 114 active+clean; 6.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 4216/334692 objects degraded (1.260%); 6700/334692 objects misplaced (2.002%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:29:27.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:26 smithi114 ceph-mon[103029]: 14.1d deep-scrub starts 2024-09-14T04:29:27.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:26 smithi114 ceph-mon[103029]: 14.1e scrub ok 2024-09-14T04:29:27.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:26 smithi114 ceph-mon[106969]: pgmap v1561: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 114 active+clean; 6.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 4216/334692 objects degraded (1.260%); 6700/334692 objects misplaced (2.002%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:29:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:26 smithi114 ceph-mon[106969]: 14.1d deep-scrub starts 2024-09-14T04:29:27.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:26 smithi114 ceph-mon[106969]: 14.1e scrub ok 2024-09-14T04:29:27.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:27.199 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:27.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:27.301 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:27.340 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:27.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:27.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:27.474 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:27.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:27.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:27.612 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:27.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:27.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:27.747 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:27.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:27.800 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:29:27.800 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:29:27.849 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:27.889 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:27.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:27.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:28.019 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:28.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:28.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:28.158 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:28.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:28.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:28.289 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:28.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:28.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:28.426 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:28.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:28.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:28.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:28.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:28.613 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-14T04:29:28.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:28.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:28.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:28.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:28.833 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:28.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:28.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:28 smithi160 ceph-mon[88407]: pgmap v1562: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 114 active+clean; 6.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 4216/334692 objects degraded (1.260%); 6700/334692 objects misplaced (2.002%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:29:28.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:28.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:28.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:29.038 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:29.064 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:29.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:29.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:29.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:28 smithi114 ceph-mon[103029]: pgmap v1562: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 114 active+clean; 6.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 4216/334692 objects degraded (1.260%); 6700/334692 objects misplaced (2.002%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:29:29.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:28 smithi114 ceph-mon[106969]: pgmap v1562: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 114 active+clean; 6.8 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 4216/334692 objects degraded (1.260%); 6700/334692 objects misplaced (2.002%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:29:29.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:29.230 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:29.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:29.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:29.368 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:29.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:29.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:29.496 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:29.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:29.598 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:29.630 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:29.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:29.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:29.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:29.753 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:29.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:29.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:29.889 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:29.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:29.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:29 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4216/334692 objects degraded (1.260%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:29.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:30.025 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:30.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:30.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:29 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4216/334692 objects degraded (1.260%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:30.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:29 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4216/334692 objects degraded (1.260%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:30.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:30.163 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:30.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:30.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:30.299 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:30.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:30.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:30.330 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:29:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:29:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:29:30.401 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:30.431 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:30.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:30.533 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:30.572 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:30.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:30.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:30.699 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:30.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:30.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:30.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:30.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:30.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:30.936 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:30.967 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:30.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:30.992 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:30 smithi160 ceph-mon[88407]: pgmap v1563: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 114 active+clean; 6.7 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 4216/331227 objects degraded (1.273%); 6700/331227 objects misplaced (2.023%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:29:31.018 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:29:31.018 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:29:31.069 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:31.105 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:31.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:31.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:30 smithi114 ceph-mon[106969]: pgmap v1563: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 114 active+clean; 6.7 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 4216/331227 objects degraded (1.273%); 6700/331227 objects misplaced (2.023%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:29:31.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:30 smithi114 ceph-mon[103029]: pgmap v1563: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 114 active+clean; 6.7 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 4216/331227 objects degraded (1.273%); 6700/331227 objects misplaced (2.023%); 3.1 MiB/s, 50 objects/s recovering 2024-09-14T04:29:31.208 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:31.238 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:31.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:31.340 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:31.377 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:31.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:31.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:31.511 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:31.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:31.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:31.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:29:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:29:31] "GET /metrics HTTP/1.1" 200 36575 "" "Prometheus/2.51.0" 2024-09-14T04:29:31.633 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:31.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:31.735 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:31.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-14T04:29:31.778 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:31.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:31.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:31.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:31.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:31.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:32.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:32.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:32.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:32.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:32.198 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:32.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:32.255 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:32.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:32.327 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:32.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:32.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:32.463 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:32.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:32.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:32.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:32.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:32.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:32.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:32.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:32.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:32.842 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:32.876 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:32.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:32.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:32.962 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:32 smithi160 ceph-mon[88407]: pgmap v1564: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 6.4 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 4216/312252 objects degraded (1.350%); 5970/312252 objects misplaced (1.912%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:29:32.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:33.010 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:33.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:33.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:33.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:32 smithi114 ceph-mon[103029]: pgmap v1564: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 6.4 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 4216/312252 objects degraded (1.350%); 5970/312252 objects misplaced (1.912%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:29:33.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:32 smithi114 ceph-mon[106969]: pgmap v1564: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 6.4 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 4216/312252 objects degraded (1.350%); 5970/312252 objects misplaced (1.912%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:29:33.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:33.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:33.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:33.270 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:33.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:33.372 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:33.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:33.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:33.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:33.562 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:33.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:33.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:33.700 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:33.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:33.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:33.844 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:33.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:33.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:33.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:33.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:34.041 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:29:34.041 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:34.041 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:34.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:34.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:29:34.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:34.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:34.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:29:34.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:34.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:34.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:34.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:34.289 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:34.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:34.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:34.447 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:34.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:34.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:34.589 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:34.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:34.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:34.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:34.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:34.778 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:29:34.779 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:29:34.829 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:34.860 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:34.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:34 smithi160 ceph-mon[88407]: pgmap v1565: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 6.4 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 4216/312252 objects degraded (1.350%); 5970/312252 objects misplaced (1.912%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:29:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:34 smithi160 ceph-mon[88407]: 14.22 scrub starts 2024-09-14T04:29:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:34 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:34 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:34.909 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:34 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4216/312252 objects degraded (1.350%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:34.960 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:34.982 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:34.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:35.084 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:35.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:34 smithi114 ceph-mon[103029]: pgmap v1565: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 6.4 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 4216/312252 objects degraded (1.350%); 5970/312252 objects misplaced (1.912%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:29:35.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:34 smithi114 ceph-mon[103029]: 14.22 scrub starts 2024-09-14T04:29:35.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:34 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:35.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:34 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:35.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:34 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4216/312252 objects degraded (1.350%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:35.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:34 smithi114 ceph-mon[106969]: pgmap v1565: 119 pgs: 1 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 6.4 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 4216/312252 objects degraded (1.350%); 5970/312252 objects misplaced (1.912%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:29:35.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:34 smithi114 ceph-mon[106969]: 14.22 scrub starts 2024-09-14T04:29:35.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:34 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:35.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:34 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:35.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:34 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4216/312252 objects degraded (1.350%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:35.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:35.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:35.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:35.267 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:35.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:35.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:35.398 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:35.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:35.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:35.500 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:35.536 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:35.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:35.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:35.672 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:35.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:35.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-14T04:29:35.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:35.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:35.842 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:35.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:35.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:35.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:35.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:35.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:36.078 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:36.119 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:36.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:36.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:36.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:36.251 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:36.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:36.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:36.390 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:36.418 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:36.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:36.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:36.559 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:36.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:36.661 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:36.699 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:36.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:36.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:36.841 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:36.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:36.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:36 smithi160 ceph-mon[88407]: pgmap v1566: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 4216/291264 objects degraded (1.447%); 5224/291264 objects misplaced (1.794%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:29:36.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:36.971 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:36.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:37.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:37.095 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:37.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:37.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:37.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:36 smithi114 ceph-mon[103029]: pgmap v1566: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 4216/291264 objects degraded (1.447%); 5224/291264 objects misplaced (1.794%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:29:37.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:36 smithi114 ceph-mon[106969]: pgmap v1566: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 4216/291264 objects degraded (1.447%); 5224/291264 objects misplaced (1.794%); 3.9 MiB/s, 62 objects/s recovering 2024-09-14T04:29:37.197 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:37.238 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:37.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:37.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:37.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:37.477 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:37.516 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:37.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:37.617 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:37.658 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:37.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:37.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:37.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:37.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:37.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:37.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:37.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:37.933 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:37.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:38.035 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:38.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:38.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:38.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:38.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:38.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:38.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:38.294 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:38.328 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:38.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:38.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:38.461 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:38.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:38.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:38.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:38.599 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:38.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:38.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:38.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:38.741 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:38.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:38.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:38.871 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:38.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:38.973 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:39.009 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:39.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:39.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:39.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:38 smithi114 ceph-mon[106969]: pgmap v1567: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 777 op/s; 4216/291264 objects degraded (1.447%); 5224/291264 objects misplaced (1.794%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:29:39.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:38 smithi114 ceph-mon[103029]: pgmap v1567: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 777 op/s; 4216/291264 objects degraded (1.447%); 5224/291264 objects misplaced (1.794%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:29:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:39.133 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:39.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:38 smithi160 ceph-mon[88407]: pgmap v1567: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.9 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 0 B/s wr, 777 op/s; 4216/291264 objects degraded (1.447%); 5224/291264 objects misplaced (1.794%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:29:39.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:39.269 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:39.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:39.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:39.405 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:39.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:39.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:39.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:39.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:39.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:39.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:39.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:39.677 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:39.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:39.779 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:39.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:39.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:39.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:39.948 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:39.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:40.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:40.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:40.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:40.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:40.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:39 smithi114 ceph-mon[106969]: pgmap v1568: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 864 op/s; 4216/288111 objects degraded (1.463%); 5224/288111 objects misplaced (1.813%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:29:40.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:39 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4216/291264 objects degraded (1.447%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:39 smithi114 ceph-mon[103029]: pgmap v1568: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 864 op/s; 4216/288111 objects degraded (1.463%); 5224/288111 objects misplaced (1.813%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:29:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:39 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4216/291264 objects degraded (1.447%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:39 smithi160 ceph-mon[88407]: pgmap v1568: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.9 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 0 B/s wr, 864 op/s; 4216/288111 objects degraded (1.463%); 5224/288111 objects misplaced (1.813%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:29:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:39 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4216/291264 objects degraded (1.447%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:40.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:40.217 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:40.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:40.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:40.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:40.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:40.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:40.459 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:40.486 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:40.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:40.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:40.588 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:40.641 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:40.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:29:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:29:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:29:40.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:40.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:40.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:40.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:40.916 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:40.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:41.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:41.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:41.040 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:41.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:41.142 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:41.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:41.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:41.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:41.330 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:41 smithi160 ceph-mon[88407]: 14.d deep-scrub starts 2024-09-14T04:29:41.337 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:41.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:41.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:41.486 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:41.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:41.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:41.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:41 smithi114 ceph-mon[106969]: 14.d deep-scrub starts 2024-09-14T04:29:41.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:29:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:29:41] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-09-14T04:29:41.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:41 smithi114 ceph-mon[103029]: 14.d deep-scrub starts 2024-09-14T04:29:41.621 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:41.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:41.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:41.771 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:41.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:41.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:41.806 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:29:41.806 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:29:41.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:41.896 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:41.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:41.997 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:42.039 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:42.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:42.061 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:29:42.141 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:42.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:42.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:42.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:42.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:42.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:42.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:42.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:42.457 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:42.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:42.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:42.506 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:42 smithi114 ceph-mon[106969]: pgmap v1569: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.27k op/s; 4216/270162 objects degraded (1.561%); 4390/270162 objects misplaced (1.625%); 4.4 MiB/s, 70 objects/s recovering 2024-09-14T04:29:42.507 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:42 smithi114 ceph-mon[103029]: pgmap v1569: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.27k op/s; 4216/270162 objects degraded (1.561%); 4390/270162 objects misplaced (1.625%); 4.4 MiB/s, 70 objects/s recovering 2024-09-14T04:29:42.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:42.598 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:42.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:42 smithi160 ceph-mon[88407]: pgmap v1569: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.27k op/s; 4216/270162 objects degraded (1.561%); 4390/270162 objects misplaced (1.625%); 4.4 MiB/s, 70 objects/s recovering 2024-09-14T04:29:42.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:42.733 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:42.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:42.835 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:42.870 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:42.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:42.913 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:29:42.913 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:29:42.950 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:29:42.951 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:29:42.971 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:42.986 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:29:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:29:43.000 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:43.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:43.022 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:29:43.022 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:29:43.054 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:29:43.054 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:29:43.088 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:29:43.088 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:29:43.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:43.126 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:29:43.127 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:29:43.137 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:43.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:43.155 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:29:43.155 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:29:43.239 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:43.268 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:43.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:43.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:43.403 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:43.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:43.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:43.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:43.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:43.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:43 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2982107050' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:29:43.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:43 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2982107050' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:29:43.645 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:43.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:43 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2982107050' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:29:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:43.682 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:43.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:43.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:43.819 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:43.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:43.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:43.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:43.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:44.058 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:44.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:44.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:44.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:44.171 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-14T04:29:44.196 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:44.227 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:44.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:44.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:44.354 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:44 smithi114 ceph-mon[103029]: pgmap v1570: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 4216/270162 objects degraded (1.561%); 4390/270162 objects misplaced (1.625%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T04:29:44.354 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:44 smithi114 ceph-mon[103029]: 14.2 scrub starts 2024-09-14T04:29:44.367 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:44.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:44 smithi160 ceph-mon[88407]: pgmap v1570: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 4216/270162 objects degraded (1.561%); 4390/270162 objects misplaced (1.625%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T04:29:44.409 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:44 smithi160 ceph-mon[88407]: 14.2 scrub starts 2024-09-14T04:29:44.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:44.471 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:29:44.471 INFO:teuthology.orchestra.run.smithi114.stdout: "success": "" 2024-09-14T04:29:44.471 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:29:44.517 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:44.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:44.619 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:44.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:44 smithi114 ceph-mon[106969]: pgmap v1570: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.5 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 0 B/s wr, 1.17k op/s; 4216/270162 objects degraded (1.561%); 4390/270162 objects misplaced (1.625%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T04:29:44.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:44 smithi114 ceph-mon[106969]: 14.2 scrub starts 2024-09-14T04:29:44.651 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:44.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:44.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:44.781 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:44.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:44.854 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:44.882 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:44.932 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:44.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:45.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:45.070 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:45.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:45.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:45.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:45.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:45.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:45.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:45.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:45.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:45.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:45.446 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:45.478 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:45.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:45.580 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:45.589 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:45 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4216/270162 objects degraded (1.561%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:45.589 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:45 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4216/270162 objects degraded (1.561%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:45.618 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:45.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:45.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:45 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4216/270162 objects degraded (1.561%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:45.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:45.759 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:45.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:45.861 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:45.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:45.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:46.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:46.028 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:46.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:46.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:46.130 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:46.164 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:46.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:46.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:46.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:46.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:46.432 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:46.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:46.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:46.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:46.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:46.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:46 smithi114 ceph-mon[103029]: pgmap v1571: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.74k op/s; 4216/249633 objects degraded (1.689%); 3517/249633 objects misplaced (1.409%); 4.8 MiB/s, 77 objects/s recovering 2024-09-14T04:29:46.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:46 smithi114 ceph-mon[103029]: 14.1d scrub starts 2024-09-14T04:29:46.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:46 smithi114 ceph-mon[106969]: pgmap v1571: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.74k op/s; 4216/249633 objects degraded (1.689%); 3517/249633 objects misplaced (1.409%); 4.8 MiB/s, 77 objects/s recovering 2024-09-14T04:29:46.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:46 smithi114 ceph-mon[106969]: 14.1d scrub starts 2024-09-14T04:29:46.635 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:46 smithi160 ceph-mon[88407]: pgmap v1571: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.74k op/s; 4216/249633 objects degraded (1.689%); 3517/249633 objects misplaced (1.409%); 4.8 MiB/s, 77 objects/s recovering 2024-09-14T04:29:46.635 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:46 smithi160 ceph-mon[88407]: 14.1d scrub starts 2024-09-14T04:29:46.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:46.698 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:46.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:46.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:46.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:46.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:46.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:46.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:46.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:47.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:47.095 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:47.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:47.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:47.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:47.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:47.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:47.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:47.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:47.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:47.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:47.492 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:47.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:47.593 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:47.628 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:47.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:29:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:47.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:47.730 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:47.776 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:47.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:47.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:47.918 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:47.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:48.020 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:48.047 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:48.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:48.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:48.181 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:48.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:48.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:48.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:48.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:48.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:48.364 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:48 smithi160 ceph-mon[88407]: pgmap v1572: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.16k op/s; 4216/249633 objects degraded (1.689%); 3517/249633 objects misplaced (1.409%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T04:29:48.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:48.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:48.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:48.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:48.594 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:48.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:48.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:48.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:48 smithi114 ceph-mon[103029]: pgmap v1572: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.16k op/s; 4216/249633 objects degraded (1.689%); 3517/249633 objects misplaced (1.409%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T04:29:48.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:48 smithi114 ceph-mon[106969]: pgmap v1572: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.1 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.16k op/s; 4216/249633 objects degraded (1.689%); 3517/249633 objects misplaced (1.409%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T04:29:48.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:48.728 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:48.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:48.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:48.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:48.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:48.852 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:48.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:48.953 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:48.986 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:48.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:49.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:49.104 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:29:49.105 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-14T04:29:49.105 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-14T04:29:49.105 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-14T04:29:49.105 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:29:49.132 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:49.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:49.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:49.266 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:49.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:49.368 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:49.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:49.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:49.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:49.535 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:29:49.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:49.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:49.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:49 smithi160 ceph-mon[88407]: 14.22 deep-scrub starts 2024-09-14T04:29:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:49 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/959200711' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:29:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:49 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:29:49.689 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:49.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:49.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:49.827 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:49.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:49.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:49 smithi114 ceph-mon[106969]: 14.22 deep-scrub starts 2024-09-14T04:29:49.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:49 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/959200711' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:29:49.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:49 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:29:49.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:49 smithi114 ceph-mon[103029]: 14.22 deep-scrub starts 2024-09-14T04:29:49.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:49 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/959200711' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:29:49.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:49 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:29:49.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:49.964 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:49.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:50.069 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:50.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-14T04:29:50.107 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:50.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:50.209 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:50.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:50.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:50.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:50.372 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:50.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:50.380 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:50 smithi160 ceph-mon[88407]: pgmap v1573: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.0 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.25k op/s; 4216/246222 objects degraded (1.712%); 3517/246222 objects misplaced (1.428%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T04:29:50.380 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:50 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:29:50.380 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:50 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:29:50.380 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:50 smithi160 ceph-mon[88407]: osdmap e419: 8 total, 8 up, 5 in 2024-09-14T04:29:50.380 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:50 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4216/246222 objects degraded (1.712%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:50.380 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:29:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:29:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:29:50.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:50.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:50.512 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:50.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:50.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:50.614 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:29:50.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:50.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:50.717 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:50.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:50 smithi114 ceph-mon[103029]: pgmap v1573: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.0 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.25k op/s; 4216/246222 objects degraded (1.712%); 3517/246222 objects misplaced (1.428%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T04:29:50.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:50 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:29:50.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:50 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:29:50.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:50 smithi114 ceph-mon[103029]: osdmap e419: 8 total, 8 up, 5 in 2024-09-14T04:29:50.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:50 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4216/246222 objects degraded (1.712%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:50.748 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:50 smithi114 ceph-mon[106969]: pgmap v1573: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 5.0 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 0 B/s wr, 1.25k op/s; 4216/246222 objects degraded (1.712%); 3517/246222 objects misplaced (1.428%); 3.4 MiB/s, 54 objects/s recovering 2024-09-14T04:29:50.749 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:50 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:29:50.749 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:50 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:29:50.749 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:50 smithi114 ceph-mon[106969]: osdmap e419: 8 total, 8 up, 5 in 2024-09-14T04:29:50.749 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:50 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4216/246222 objects degraded (1.712%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:50.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:50.776 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:50.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:50.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:50.878 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:50.900 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:50.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:51.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:51.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:51.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:51.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:51.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:51.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:51.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:51.273 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:51.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:51.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:51.402 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:51.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:51.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:51.537 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:51.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:51.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:29:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:29:51] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-09-14T04:29:51.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:51 smithi114 ceph-mon[106969]: osdmap e420: 8 total, 8 up, 5 in 2024-09-14T04:29:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:51 smithi114 ceph-mon[103029]: osdmap e420: 8 total, 8 up, 5 in 2024-09-14T04:29:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:51.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:51.639 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:51 smithi160 ceph-mon[88407]: osdmap e420: 8 total, 8 up, 5 in 2024-09-14T04:29:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:51.673 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 4, 5, 2] out_osds: [1, 7, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:29:51.674 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:29:51.674 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-14T04:29:51.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:29:51.679 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:51.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:51.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:51.817 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:51.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:51.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:51.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:51.940 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:51.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:51.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:52.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:52.071 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:52.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:52.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:52.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:52.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:52.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:52.338 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:52.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:52.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:52.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:52.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:52.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:52.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:29:52.593 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:52.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:52.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:52 smithi160 ceph-mon[88407]: pgmap v1576: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 4.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.74k op/s; 4216/228345 objects degraded (1.846%); 2416/228345 objects misplaced (1.058%); 6.1 MiB/s, 97 objects/s recovering 2024-09-14T04:29:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:52 smithi160 ceph-mon[88407]: osdmap e421: 8 total, 8 up, 5 in 2024-09-14T04:29:52.681 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:29:52.681 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-14T04:29:52.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-14T04:29:52.681 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:29:52.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:51067 13.7 13:e1fcd919:::benchmark_data_smithi114_288325_object51066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:36.458700+0000", 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 436.210013661, 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.446806431, 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 51067 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.682 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.458700+0000", 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.458700+0000", 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.458698+0000", 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.458735+0000", 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7429999999999999e-05 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.458754+0000", 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.874e-05 2024-09-14T04:29:52.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.458760+0000", 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5250000000000002e-06 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.458943+0000", 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018258599999999999 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.459391+0000", 2024-09-14T04:29:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000448317 2024-09-14T04:29:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.459756+0000", 2024-09-14T04:29:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00036486100000000002 2024-09-14T04:29:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.462183+0000", 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002426409 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.462213+0000", 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0219e-05 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:52.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.905216+0000", 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.443003694 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.905341+0000", 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012450199999999999 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.905506+0000", 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000165014 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.688 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:51088 13.7 13:e1ff0abf:::benchmark_data_smithi114_288325_object51087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:36.468749+0000", 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 436.19996455699999, 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.438709297, 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:29:52.689 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 51088 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.468749+0000", 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.468749+0000", 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.468747+0000", 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.468823+0000", 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5839000000000004e-05 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.468836+0000", 2024-09-14T04:29:52.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3389e-05 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.468843+0000", 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0140000000000001e-06 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.468894+0000", 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0285999999999999e-05 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.469029+0000", 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000135656 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.469171+0000", 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014124499999999999 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.471202+0000", 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020315530000000002 2024-09-14T04:29:52.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.471234+0000", 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1488000000000001e-05 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.907365+0000", 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43613151700000002 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.907420+0000", 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5161999999999998e-05 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.907458+0000", 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7592000000000002e-05 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:51098 13.7 13:eef7fd99:::benchmark_data_smithi114_288325_object51097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-09-14T04:29:52.695 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:22:36.475134+0000", 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 436.19357933700002, 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.432434186, 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 51098 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.475134+0000", 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.475134+0000", 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.475132+0000", 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:29:52.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.475191+0000", 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8411e-05 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.475208+0000", 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7340999999999999e-05 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.475214+0000", 2024-09-14T04:29:52.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3849999999999999e-06 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.475263+0000", 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8464999999999998e-05 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.475405+0000", 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014205899999999999 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.475543+0000", 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000137942 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.477428+0000", 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001885269 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.477923+0000", 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00049441399999999999 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.907492+0000", 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42956972999999998 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.907538+0000", 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5451000000000002e-05 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:22:36.907568+0000", 2024-09-14T04:29:52.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0301e-05 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:74213 13.7 13:eed348cb:::benchmark_data_smithi114_288325_object74212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:07.977241+0000", 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.69147213399998, 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19712896199999999, 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:29:52.702 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 74213 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.977241+0000", 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.977241+0000", 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.977239+0000", 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999981 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.977283+0000", 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3764999999999999e-05 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.977295+0000", 2024-09-14T04:29:52.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1788e-05 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.977300+0000", 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2909999999999998e-06 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.979068+0000", 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017677490000000001 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.979212+0000", 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000144182 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.979360+0000", 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014774100000000001 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.982034+0000", 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026737169999999999 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.986949+0000", 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049149190000000002 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174280+0000", 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.187331888 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174332+0000", 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1391000000000003e-05 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174370+0000", 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8220000000000003e-05 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:74207 13.1 e336/332 13:8731ea14:::benchmark_data_smithi114_288325_object74206:head v 336'4698, mlcod=0'0)", 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:07.986404+0000", 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.68230887700003, 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18812584399999999, 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.708 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.986404+0000", 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.986404+0000", 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.986403+0000", 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.986429+0000", 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5482000000000001e-05 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.986445+0000", 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6277999999999999e-05 2024-09-14T04:29:52.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.986452+0000", 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8709999999999999e-06 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.995386+0000", 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0089339320000000003 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:07.995410+0000", 2024-09-14T04:29:52.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4258000000000001e-05 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174501+0000", 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17909064299999999 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174530+0000", 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9339000000000001e-05 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.712 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:74225 13.1 e336/332 13:85e8fbe0:::benchmark_data_smithi114_288325_object74224:head v 336'4699, mlcod=0'0)", 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:08.000506+0000", 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.66820714800002, 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18192824199999999, 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.000506+0000", 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.000506+0000", 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.000506+0000", 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.000532+0000", 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6047999999999999e-05 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.000539+0000", 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5360000000000004e-06 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.000544+0000", 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3540000000000002e-06 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.001012+0000", 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00046866900000000001 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.001033+0000", 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1101e-05 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.182406+0000", 2024-09-14T04:29:52.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18137299500000001 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.182434+0000", 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7889e-05 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:74227 13.1 e336/332 13:803dfdbf:::benchmark_data_smithi114_288325_object74226:head v 336'4700, mlcod=0'0)", 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:08.002520+0000", 2024-09-14T04:29:52.717 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.666193147, 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19063039000000001, 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.002520+0000", 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.002520+0000", 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.002518+0000", 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.002550+0000", 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1433000000000002e-05 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.002567+0000", 2024-09-14T04:29:52.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6942000000000001e-05 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.002574+0000", 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.759e-06 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174551+0000", 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17197776300000001 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174572+0000", 2024-09-14T04:29:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0429e-05 2024-09-14T04:29:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.193108+0000", 2024-09-14T04:29:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018536568 2024-09-14T04:29:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.193150+0000", 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2098999999999998e-05 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.46760.0:74230 13.7 13:ebb1f78f:::benchmark_data_smithi114_288325_object74229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:08.003874+0000", 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.66483957200001, 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.170820373, 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-14T04:29:52.722 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.46760", 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.114:0/643775923", 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 74230 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.003874+0000", 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.003874+0000", 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.003871+0000", 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.003906+0000", 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4836999999999999e-05 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.003923+0000", 2024-09-14T04:29:52.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6631000000000001e-05 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.003930+0000", 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5089999999999996e-06 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.004031+0000", 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010090900000000001 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.004179+0000", 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000147971 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 4,6", 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.004335+0000", 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015563600000000001 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007993+0000", 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0036585110000000001 2024-09-14T04:29:52.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174401+0000", 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16640787900000001 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174589+0000", 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018762499999999999 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174661+0000", 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2379999999999995e-05 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174694+0000", 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2768000000000002e-05 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:74231 13.a e336/332 13:5286a5b0:::benchmark_data_smithi114_288325_object74230:head v 336'4579, mlcod=0'0)", 2024-09-14T04:29:52.728 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:08.005340+0000", 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.66337281699998, 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17107301799999999, 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.005340+0000", 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.005340+0000", 2024-09-14T04:29:52.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.005339+0000", 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.005378+0000", 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8917e-05 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.005391+0000", 2024-09-14T04:29:52.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2751e-05 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.005397+0000", 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1210000000000003e-06 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.005442+0000", 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4898000000000003e-05 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.005465+0000", 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2901e-05 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.176308+0000", 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.170843148 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.176413+0000", 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000105729 2024-09-14T04:29:52.732 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.46760.0:74213 13.7 e336/321 ondisk, result = 0)", 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:08.007638+0000", 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.66107525699999, 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.166737364, 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007638+0000", 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007638+0000", 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007638+0000", 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967296 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007640+0000", 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1749999999999999e-06 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007645+0000", 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8269999999999997e-06 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007648+0000", 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9330000000000001e-06 2024-09-14T04:29:52.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174265+0000", 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16661748200000001 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174274+0000", 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6289999999999999e-06 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174375+0000", 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101494 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:74214 13.f e336/332 13:f48f2afe:::benchmark_data_smithi114_288325_object74213:head v 336'4665, mlcod=0'0)", 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:08.007654+0000", 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.66105893299999, 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17903832, 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007654+0000", 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007654+0000", 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007654+0000", 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967296 2024-09-14T04:29:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007683+0000", 2024-09-14T04:29:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8586999999999999e-05 2024-09-14T04:29:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007689+0000", 2024-09-14T04:29:52.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.348e-06 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.007692+0000", 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.433e-06 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174421+0000", 2024-09-14T04:29:52.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16672884299999999 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174442+0000", 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1098e-05 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.185606+0000", 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011164019000000001 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.186693+0000", 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001086182 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:74229 13.d e336/321 13:b0380dbb:::benchmark_data_smithi114_288325_object74228:head v 336'4726, mlcod=336'4724)", 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:08.008438+0000", 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.66027570300002, 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17209349299999999, 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.743 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.008438+0000", 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.008438+0000", 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.008437+0000", 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.008458+0000", 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1730999999999999e-05 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174235+0000", 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.165776863 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174246+0000", 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0594000000000001e-05 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174785+0000", 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00053911900000000001 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174847+0000", 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2372999999999994e-05 2024-09-14T04:29:52.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.180467+0000", 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0056195330000000003 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.180531+0000", 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4193e-05 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:74234 13.e e336/324 13:770ae5bc:::benchmark_data_smithi114_288325_object74233:head v 336'4500, mlcod=336'4498)", 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:08.011501+0000", 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.65721176199997, 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16576596499999999, 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.011501+0000", 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.011501+0000", 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.011500+0000", 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.011528+0000", 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7685e-05 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174228+0000", 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.162700347 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174243+0000", 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4948e-05 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174533+0000", 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028954600000000001 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174554+0000", 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0786000000000001e-05 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.177237+0000", 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026836910000000002 2024-09-14T04:29:52.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.177267+0000", 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0037999999999999e-05 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:74236 13.c e336/332 13:3fce542c:::benchmark_data_smithi114_288325_object74235:head v 336'4620, mlcod=0'0)", 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:23:08.014822+0000", 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 404.65389133600002, 2024-09-14T04:29:52.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.166362334, 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.014822+0000", 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.014822+0000", 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.014820+0000", 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.014859+0000", 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8528000000000001e-05 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174404+0000", 2024-09-14T04:29:52.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15954510799999999 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174410+0000", 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9089999999999997e-06 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174446+0000", 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6062000000000001e-05 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.174470+0000", 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4017999999999999e-05 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.181119+0000", 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0066487339999999999 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:23:08.181184+0000", 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5557999999999998e-05 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.46760.0:141601 13.8 e350/347 13:163e3e1c:::benchmark_data_smithi114_288325_object46536:head v 350'8773, mlcod=329'2463)", 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:25:02.483897+0000", 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 290.18481616299999, 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.094969081999999996, 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-14T04:29:52.757 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:02.483897+0000", 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:02.483897+0000", 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:02.483896+0000", 2024-09-14T04:29:52.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999986 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:02.483906+0000", 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0288999999999999e-05 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:02.483926+0000", 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-05 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:02.483933+0000", 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.272e-06 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:02.484004+0000", 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0721000000000003e-05 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:02.484036+0000", 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2449000000000001e-05 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:02.578802+0000", 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.094765617999999996 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:25:02.578866+0000", 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4525999999999999e-05 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:52.761 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:52.762 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:29:52.763 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:52.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:52.846 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:52 smithi114 ceph-mon[103029]: pgmap v1576: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 4.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.74k op/s; 4216/228345 objects degraded (1.846%); 2416/228345 objects misplaced (1.058%); 6.1 MiB/s, 97 objects/s recovering 2024-09-14T04:29:52.846 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:52 smithi114 ceph-mon[103029]: osdmap e421: 8 total, 8 up, 5 in 2024-09-14T04:29:52.847 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:52 smithi114 ceph-mon[106969]: pgmap v1576: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 4.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.74k op/s; 4216/228345 objects degraded (1.846%); 2416/228345 objects misplaced (1.058%); 6.1 MiB/s, 97 objects/s recovering 2024-09-14T04:29:52.847 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:52 smithi114 ceph-mon[106969]: osdmap e421: 8 total, 8 up, 5 in 2024-09-14T04:29:52.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:52.899 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:52.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:53.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:53.033 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:53.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:53.135 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:53.167 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:53.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:53.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:53.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:53.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:53.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:53.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:53.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:53.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:53.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:53.588 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:53.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:53.589 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:29:53.689 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:53.733 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:53.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:53.833 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:53.864 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:53.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:53.966 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:54.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:54.013 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:54.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:54.085 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:54.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:54.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:54.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:54.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:29:54.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:29:54.140 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:54.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:54.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:54.282 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:54.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:54.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:54.425 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:54.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:54.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:54.566 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:54.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:54.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:54.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:54.706 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:54.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:54.808 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:54.847 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:54.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:54 smithi160 ceph-mon[88407]: pgmap v1578: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 4.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.18k op/s; 4216/228345 objects degraded (1.846%); 2416/228345 objects misplaced (1.058%); 4.7 MiB/s, 74 objects/s recovering 2024-09-14T04:29:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:54 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 4216/228345 objects degraded (1.846%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:54.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:54.986 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:54.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:55.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:55.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:54 smithi114 ceph-mon[103029]: pgmap v1578: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 4.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.18k op/s; 4216/228345 objects degraded (1.846%); 2416/228345 objects misplaced (1.058%); 4.7 MiB/s, 74 objects/s recovering 2024-09-14T04:29:55.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:54 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 4216/228345 objects degraded (1.846%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:55.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:54 smithi114 ceph-mon[106969]: pgmap v1578: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 115 active+clean; 4.6 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 0 B/s wr, 1.18k op/s; 4216/228345 objects degraded (1.846%); 2416/228345 objects misplaced (1.058%); 4.7 MiB/s, 74 objects/s recovering 2024-09-14T04:29:55.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:54 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 4216/228345 objects degraded (1.846%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:29:55.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:55.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:55.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:55.223 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:55.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:55.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:55.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:55.379 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:55.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:55.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:55.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:55.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:55.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:55.657 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:55.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:55.758 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:55.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:55.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:55.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:55.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:55.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:56.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:56.074 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:56.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:56.175 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:56.204 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:56.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:56.308 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:56.351 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:56.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:56.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:56.488 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:56.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:208066 14.1b e421/416 14:d92c268c:::benchmark_data_smithi114_329146_object70280:head v 421'11329, mlcod=421'11327)", 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:56.482321+0000", 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0069060550000000004, 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0069334970000000003, 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:56.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.482321+0000", 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.482321+0000", 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.482321+0000", 2024-09-14T04:29:56.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.482324+0000", 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3909999999999998e-06 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.482329+0000", 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1649999999999997e-06 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:56.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.482332+0000", 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0180000000000002e-06 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.483991+0000", 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0016585689999999999 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.484008+0000", 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6555e-05 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.494 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:208073 14.13 e421/411)", 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:56.486340+0000", 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0028872559999999999, 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029721080000000002, 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486340+0000", 2024-09-14T04:29:56.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486340+0000", 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486339+0000", 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486347+0000", 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6850000000000001e-06 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486362+0000", 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4941e-05 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486368+0000", 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.207e-06 2024-09-14T04:29:56.497 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:208067 14.19 e421/395 14:98359a2d:::benchmark_data_smithi114_329146_object70281:head v 421'8637, mlcod=421'8635)", 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:56.486425+0000", 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.002802003, 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002930556, 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:56.498 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486425+0000", 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486425+0000", 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486425+0000", 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:56.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486431+0000", 2024-09-14T04:29:56.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2179999999999996e-06 2024-09-14T04:29:56.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:56.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486440+0000", 2024-09-14T04:29:56.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5580000000000006e-06 2024-09-14T04:29:56.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486445+0000", 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8629999999999999e-06 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486567+0000", 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012175700000000001 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:56.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486585+0000", 2024-09-14T04:29:56.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8091e-05 2024-09-14T04:29:56.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.502 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:56.502 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.502 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:208068 14.7 e421/395)", 2024-09-14T04:29:56.502 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:56.486458+0000", 2024-09-14T04:29:56.502 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0027695430000000002, 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029482969999999999, 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486458+0000", 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486458+0000", 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486457+0000", 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486461+0000", 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.619e-06 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486468+0000", 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6830000000000002e-06 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486471+0000", 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7859999999999998e-06 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:208074 14.1e e421/416 14:7bd8be9c:::benchmark_data_smithi114_329146_object70288:head v 421'11337, mlcod=421'11332)", 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:56.486491+0000", 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0027358980000000001, 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002957571, 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486491+0000", 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:56.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486491+0000", 2024-09-14T04:29:56.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:56.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486491+0000", 2024-09-14T04:29:56.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486496+0000", 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6759999999999998e-06 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486502+0000", 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6420000000000001e-06 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:56.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486506+0000", 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6169999999999998e-06 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486907+0000", 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00040156300000000002 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.486924+0000", 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6554000000000001e-05 2024-09-14T04:29:56.509 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:208077 14.1c e421/416)", 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:56.488797+0000", 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00042972900000000001, 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00070140000000000003, 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:56.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488797+0000", 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488797+0000", 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488797+0000", 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488803+0000", 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6220000000000003e-06 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488820+0000", 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6492000000000001e-05 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488825+0000", 2024-09-14T04:29:56.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3709999999999999e-06 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:208076 14.a e421/416 14:567bb6c5:::benchmark_data_smithi114_329146_object70290:head v 421'11398, mlcod=421'11396)", 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:56.488836+0000", 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00039111300000000001, 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00070559500000000005, 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488836+0000", 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488836+0000", 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488836+0000", 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:56.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488840+0000", 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9899999999999999e-06 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488846+0000", 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3620000000000002e-06 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488849+0000", 2024-09-14T04:29:56.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.959e-06 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488922+0000", 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2879000000000005e-05 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.488940+0000", 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8088e-05 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:56.516 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(14.14 421/413 [PushOp(14:2f75d0a1:::benchmark_data_smithi114_329146_object73467:head, version: 401'4208, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(14:2f75d0a1:::benchmark_data_smithi114_329146_object73467:head@401'4208, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:56.470862+0000", 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.018365210999999999, 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018754943, 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470862+0000", 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470862+0000", 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470862+0000", 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470912+0000", 2024-09-14T04:29:56.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0584999999999998e-05 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470927+0000", 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5288e-05 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470936+0000", 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.653e-06 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:208042 14.14 e421/413 14:2af8fe41:::benchmark_data_smithi114_329146_object70256:head v 421'8334, mlcod=389'97)", 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:56.470954+0000", 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.018273576999999999, 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018707793, 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470954+0000", 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470954+0000", 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470953+0000", 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470958+0000", 2024-09-14T04:29:56.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2889999999999999e-06 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470964+0000", 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0079999999999999e-06 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470967+0000", 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.755e-06 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:56.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.477954+0000", 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0069867749999999998 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.477979+0000", 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4978999999999999e-05 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop(client.47180.0:208053 14.14 e421/413 14:2d882b0d:::benchmark_data_smithi114_329146_object70267:head v 421'8335, mlcod=389'97)", 2024-09-14T04:29:56.523 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-14T04:29:56.470971+0000", 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.018256377000000001, 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018746704999999999, 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470971+0000", 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-14T04:29:56.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470971+0000", 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470971+0000", 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967296 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470973+0000", 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7149999999999998e-06 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470978+0000", 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.583e-06 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.470981+0000", 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9510000000000002e-06 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.484947+0000", 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013966127 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-14T04:29:56.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-14T04:29:56.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-14T04:29:56.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-14T04:29:56.484969+0000", 2024-09-14T04:29:56.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2082999999999998e-05 2024-09-14T04:29:56.527 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.527 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-14T04:29:56.527 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.527 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-14T04:29:56.527 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-14T04:29:56.527 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 10 2024-09-14T04:29:56.527 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-14T04:29:56.589 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:56.623 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:56.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:56.725 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:56.763 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:56.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:56.864 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:56.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:56 smithi114 ceph-mon[106969]: pgmap v1579: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 4.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.97k op/s; 1826/210732 objects degraded (0.867%); 4283/210732 objects misplaced (2.032%); 7.6 MiB/s, 121 objects/s recovering 2024-09-14T04:29:56.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:56 smithi114 ceph-mon[103029]: pgmap v1579: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 4.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.97k op/s; 1826/210732 objects degraded (0.867%); 4283/210732 objects misplaced (2.032%); 7.6 MiB/s, 121 objects/s recovering 2024-09-14T04:29:56.899 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:56.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:57.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:29:57.033 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:57.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:57.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:29:57.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:57.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:56 smithi160 ceph-mon[88407]: pgmap v1579: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 4.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.97k op/s; 1826/210732 objects degraded (0.867%); 4283/210732 objects misplaced (2.032%); 7.6 MiB/s, 121 objects/s recovering 2024-09-14T04:29:57.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:57.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:57.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:57.313 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:57.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:57.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:57.457 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:57.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:57.559 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:57.597 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:57.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:57.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:57.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:57.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:57.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:57.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:57.860 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:57.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:57.962 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:57.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:57.995 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:57.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:58.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:29:58.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:58.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:29:58.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:58.265 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:58.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:58.367 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:58.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:58.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:58.486 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:29:58.487 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:29:58.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-14T04:29:58.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:29:58.515 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:29:58.515 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:29:58.543 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:29:58.543 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:29:58.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:58.554 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:58.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:29:58.566 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:29:58.566 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:29:58.594 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:29:58.594 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:29:58.620 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:29:58.621 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:29:58.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:58.660 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:29:58.661 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:29:58.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:58.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:58.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:58.693 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:29:58.693 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:29:58.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:58.826 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:58.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:58.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:58 smithi114 ceph-mon[106969]: pgmap v1580: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 4.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.50k op/s; 1826/210732 objects degraded (0.867%); 4283/210732 objects misplaced (2.032%); 5.8 MiB/s, 92 objects/s recovering 2024-09-14T04:29:58.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:58 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2655318503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:29:58.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:58 smithi114 ceph-mon[103029]: pgmap v1580: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 4.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.50k op/s; 1826/210732 objects degraded (0.867%); 4283/210732 objects misplaced (2.032%); 5.8 MiB/s, 92 objects/s recovering 2024-09-14T04:29:58.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:58 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2655318503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:29:58.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:58 smithi160 ceph-mon[88407]: pgmap v1580: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 4.3 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 1.50k op/s; 1826/210732 objects degraded (0.867%); 4283/210732 objects misplaced (2.032%); 5.8 MiB/s, 92 objects/s recovering 2024-09-14T04:29:58.910 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:58 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2655318503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:29:58.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:58.961 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:58.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:58.961 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:29:59.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:29:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:59.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:59.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:29:59.195 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:59.225 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:59.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:59.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:59.327 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:29:59.355 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:59.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:29:59.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:59.493 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:59.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:59.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:59.595 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:29:59.634 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:29:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:59.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:29:59.726 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:29:59 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1826/210732 objects degraded (0.867%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:59.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:29:59 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1826/210732 objects degraded (0.867%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:29:59.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:29:59.773 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:59.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:29:59.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:29:59.912 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:29:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:29:59.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:29:59.961 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:29:59.992 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:00.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:00.031 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:29:59 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1826/210732 objects degraded (0.867%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:30:00.068 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:00.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:00.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:00.212 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:00.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:00.292 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:30:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:30:00.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:30:00.315 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:00.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:00.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:00.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:00.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:00.484 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:00.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:00.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:00.586 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:00.618 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:00.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:00.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:00.719 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:00.748 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:00.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:00.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:00.898 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:00.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:00.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:00 smithi114 ceph-mon[106969]: pgmap v1581: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 4.2 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 795 op/s; 1826/207237 objects degraded (0.881%); 4283/207237 objects misplaced (2.067%); 2.0 MiB/s, 31 objects/s recovering 2024-09-14T04:30:00.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:00 smithi114 ceph-mon[106969]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 1826/207237 objects degraded (0.881%), 1 pg degraded 2024-09-14T04:30:00.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:00 smithi114 ceph-mon[106969]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-14T04:30:00.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:00 smithi114 ceph-mon[106969]: [WRN] PG_DEGRADED: Degraded data redundancy: 1826/207237 objects degraded (0.881%), 1 pg degraded 2024-09-14T04:30:00.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:00 smithi114 ceph-mon[106969]: pg 14.14 is active+recovering+undersized+degraded+remapped, acting [3,6] 2024-09-14T04:30:00.952 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:00 smithi114 ceph-mon[103029]: pgmap v1581: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 4.2 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 795 op/s; 1826/207237 objects degraded (0.881%); 4283/207237 objects misplaced (2.067%); 2.0 MiB/s, 31 objects/s recovering 2024-09-14T04:30:00.952 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:00 smithi114 ceph-mon[103029]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 1826/207237 objects degraded (0.881%), 1 pg degraded 2024-09-14T04:30:00.952 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:00 smithi114 ceph-mon[103029]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-14T04:30:00.952 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:00 smithi114 ceph-mon[103029]: [WRN] PG_DEGRADED: Degraded data redundancy: 1826/207237 objects degraded (0.881%), 1 pg degraded 2024-09-14T04:30:00.952 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:00 smithi114 ceph-mon[103029]: pg 14.14 is active+recovering+undersized+degraded+remapped, acting [3,6] 2024-09-14T04:30:00.964 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:00 smithi160 ceph-mon[88407]: pgmap v1581: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 4.2 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 795 op/s; 1826/207237 objects degraded (0.881%); 4283/207237 objects misplaced (2.067%); 2.0 MiB/s, 31 objects/s recovering 2024-09-14T04:30:00.964 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:00 smithi160 ceph-mon[88407]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 1826/207237 objects degraded (0.881%), 1 pg degraded 2024-09-14T04:30:00.964 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:00 smithi160 ceph-mon[88407]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-14T04:30:00.964 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:00 smithi160 ceph-mon[88407]: [WRN] PG_DEGRADED: Degraded data redundancy: 1826/207237 objects degraded (0.881%), 1 pg degraded 2024-09-14T04:30:00.964 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:00 smithi160 ceph-mon[88407]: pg 14.14 is active+recovering+undersized+degraded+remapped, acting [3,6] 2024-09-14T04:30:01.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:01.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:01.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:01.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:01.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:01.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:01.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:01.298 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:01.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:01.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:01.394 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:30:01.394 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:30:01.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:01.428 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:01.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:01.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:01.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:30:01.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:01.569 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:01.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:01.671 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:01.709 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:01.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:01.811 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:01.848 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:01.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:01.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:30:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:30:01] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-09-14T04:30:01.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:01.989 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:01.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:02.090 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:02.121 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:02.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:02.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:02.254 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:02.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:02.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:02.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:02.389 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:02.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:02.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:02.529 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:02.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:02.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:02.675 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:02.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:02.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:02.777 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:02.804 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:02.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:02.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:02.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:02.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:02.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:03.040 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:03.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:03.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:02 smithi114 ceph-mon[103029]: pgmap v1582: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 1826/188994 objects degraded (0.966%); 3853/188994 objects misplaced (2.039%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T04:30:03.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:02 smithi114 ceph-mon[106969]: pgmap v1582: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 1826/188994 objects degraded (0.966%); 3853/188994 objects misplaced (2.039%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T04:30:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:02 smithi160 ceph-mon[88407]: pgmap v1582: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 1826/188994 objects degraded (0.966%); 3853/188994 objects misplaced (2.039%); 3.1 MiB/s, 49 objects/s recovering 2024-09-14T04:30:03.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:03.198 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:03.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:03.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:03.323 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:03.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:03.425 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:03.454 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:03.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:03.556 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:03.591 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:03.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:03.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:03.692 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:03.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:03.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:03.863 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:03.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:03.965 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:03.987 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:03.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:04.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:04.111 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:04.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:04.213 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:04.248 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:04.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:04.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:04.382 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:04.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:04.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:04.518 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:04.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:04.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:04.620 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:04.650 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:04.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:04.667 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-14T04:30:04.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:04.780 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:04.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:04.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:04.903 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:04.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:04 smithi160 ceph-mon[88407]: pgmap v1583: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 1826/188994 objects degraded (0.966%); 3853/188994 objects misplaced (2.039%); 2.6 MiB/s, 41 objects/s recovering 2024-09-14T04:30:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:04 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1826/188994 objects degraded (0.966%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:30:05.005 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:05.053 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:05.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:05.091 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:30:05.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:04 smithi114 ceph-mon[106969]: pgmap v1583: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 1826/188994 objects degraded (0.966%); 3853/188994 objects misplaced (2.039%); 2.6 MiB/s, 41 objects/s recovering 2024-09-14T04:30:05.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:04 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1826/188994 objects degraded (0.966%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:30:05.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:04 smithi114 ceph-mon[103029]: pgmap v1583: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.11k op/s; 1826/188994 objects degraded (0.966%); 3853/188994 objects misplaced (2.039%); 2.6 MiB/s, 41 objects/s recovering 2024-09-14T04:30:05.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:04 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1826/188994 objects degraded (0.966%), 1 pg degraded (PG_DEGRADED) 2024-09-14T04:30:05.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:05.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:05.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:05.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:05.334 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:05.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:05.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:05.462 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:05.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:05.564 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:05.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:05.586 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:05.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:05.687 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:05.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:05.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:05.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:05.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:05.862 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:05.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:05.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:06.005 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:06.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:06.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:06.148 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:06.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:06.250 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:06.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:06.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:06.427 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:06.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:06.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:06.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:06.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:06.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:06.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:06.693 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:06.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:06.795 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:06.838 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:06.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:06.843 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:06 smithi160 ceph-mon[88407]: pgmap v1584: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.63k op/s; 1826/169566 objects degraded (1.077%); 3431/169566 objects misplaced (2.023%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T04:30:06.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:06.969 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:06.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:07.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:07.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:06 smithi114 ceph-mon[103029]: pgmap v1584: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.63k op/s; 1826/169566 objects degraded (1.077%); 3431/169566 objects misplaced (2.023%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T04:30:07.081 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:06 smithi114 ceph-mon[106969]: pgmap v1584: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.63k op/s; 1826/169566 objects degraded (1.077%); 3431/169566 objects misplaced (2.023%); 3.7 MiB/s, 59 objects/s recovering 2024-09-14T04:30:07.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:07.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:07.194 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:30:07.194 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:30:07.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:07.235 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:07.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:07.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:07.376 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:07.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:07.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:07.508 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:07.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:07.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:07.645 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:07.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:07.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:07.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:07.901 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:07.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:07.941 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:07.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:07.979 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:07.979 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:08.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:08.080 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:08.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:08.169 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-14T04:30:08.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:08.246 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:08.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:08.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:08.383 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:08.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:08.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:08.533 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:08.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:08.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:08.667 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:08.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:08.743 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:30:08.768 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:08.796 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:08.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:08.898 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:08.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:09.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:09.069 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:09.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:09.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:08 smithi114 ceph-mon[106969]: pgmap v1585: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.14k op/s; 1826/169566 objects degraded (1.077%); 3431/169566 objects misplaced (2.023%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:30:09.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3769889461' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:30:09.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:08 smithi114 ceph-mon[103029]: pgmap v1585: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.14k op/s; 1826/169566 objects degraded (1.077%); 3431/169566 objects misplaced (2.023%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:30:09.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3769889461' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:30:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:08 smithi160 ceph-mon[88407]: pgmap v1585: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.14k op/s; 1826/169566 objects degraded (1.077%); 3431/169566 objects misplaced (2.023%); 2.2 MiB/s, 35 objects/s recovering 2024-09-14T04:30:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3769889461' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:30:09.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:09.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:09.307 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:09.333 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:09.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:09.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:09.434 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:09.472 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:09.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:09.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:09.611 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:09.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:09.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:09.748 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:09.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:09.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:09.891 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:09.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:09.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:10.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:09 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:30:10.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:09 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3769889461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:30:10.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:09 smithi114 ceph-mon[103029]: osdmap e422: 8 total, 8 up, 5 in 2024-09-14T04:30:10.021 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:09 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1826/169566 objects degraded (1.077%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:10.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:09 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:30:10.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:09 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3769889461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:30:10.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:09 smithi114 ceph-mon[106969]: osdmap e422: 8 total, 8 up, 5 in 2024-09-14T04:30:10.022 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:09 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1826/169566 objects degraded (1.077%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:10.023 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:10.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:10.124 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:10.153 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:10.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:10.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:09 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:30:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:09 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3769889461' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:30:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:09 smithi160 ceph-mon[88407]: osdmap e422: 8 total, 8 up, 5 in 2024-09-14T04:30:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:09 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1826/169566 objects degraded (1.077%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:10.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:10.291 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:10.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:10.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:10.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:10.429 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:10.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:10.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:10.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:30:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:30:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:30:10.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:10.702 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:10.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:10.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:10.832 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:10 smithi114 ceph-mon[106969]: pgmap v1587: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.36k op/s; 1826/166467 objects degraded (1.097%); 3431/166467 objects misplaced (2.061%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:30:10.833 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:10 smithi114 ceph-mon[103029]: pgmap v1587: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.36k op/s; 1826/166467 objects degraded (1.097%); 3431/166467 objects misplaced (2.061%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:30:10.837 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:10.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:10.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:10.976 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:10.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:11.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:11.115 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:11.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:10 smithi160 ceph-mon[88407]: pgmap v1587: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 1.36k op/s; 1826/166467 objects degraded (1.097%); 3431/166467 objects misplaced (2.061%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:30:11.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:11.247 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:11.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:11.349 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:11.387 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:11.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:11.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:11.534 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:11.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:11.591 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:30:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:30:11] "GET /metrics HTTP/1.1" 200 36571 "" "Prometheus/2.51.0" 2024-09-14T04:30:11.635 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:11.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:11.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:11.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:11.816 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:11.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:11.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:11.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:11.946 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:11.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:12.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:12.081 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:12.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:12.100 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:12.170 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:12.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:12.228 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:12.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:12.248 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:11 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:12.248 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:11 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:12.248 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:11 smithi160 ceph-mon[88407]: pgmap v1588: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.34k op/s; 1826/148674 objects degraded (1.228%); 2977/148674 objects misplaced (2.002%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:30:12.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:11 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:12.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:11 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:12.249 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:11 smithi114 ceph-mon[106969]: pgmap v1588: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.34k op/s; 1826/148674 objects degraded (1.228%); 2977/148674 objects misplaced (2.002%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:30:12.250 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:12.250 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:11 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:12.250 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:11 smithi114 ceph-mon[103029]: pgmap v1588: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.34k op/s; 1826/148674 objects degraded (1.228%); 2977/148674 objects misplaced (2.002%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:30:12.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:12.367 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:12.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:12.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:12.508 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:12.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:12.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:12.646 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:12.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:12.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:12.785 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:12.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:12.886 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:12.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:12.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:13.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:13.060 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:13.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:13.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:13.197 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:13.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:13.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:13.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:13.335 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:13.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:13.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:13.470 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:13.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:13.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:13.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:13.605 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:13.605 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:13.606 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:13.607 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:13.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:13.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:13.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:13.746 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:13.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:13.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:13.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:13.870 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:13.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:13.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:13.972 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:14.014 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:14.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:14.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:14.147 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:14.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:14.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:30:14.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:14.284 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:14.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:14.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:14 smithi114 ceph-mon[106969]: pgmap v1589: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.34k op/s; 1826/148674 objects degraded (1.228%); 2977/148674 objects misplaced (2.002%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:30:14.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:14 smithi114 ceph-mon[103029]: pgmap v1589: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.34k op/s; 1826/148674 objects degraded (1.228%); 2977/148674 objects misplaced (2.002%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:30:14.386 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:14.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:14.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:14.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:14.524 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:14.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:14.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:14.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:14 smithi160 ceph-mon[88407]: pgmap v1589: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 3.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 1.34k op/s; 1826/148674 objects degraded (1.228%); 2977/148674 objects misplaced (2.002%); 2.7 MiB/s, 43 objects/s recovering 2024-09-14T04:30:14.660 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:14.692 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:14.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:14.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:14.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:14.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:14.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:14.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:14.965 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:14.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:15.067 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:15.106 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:15.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:15.207 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:15.231 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:15.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:15.332 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:15.365 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:15.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:15.467 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:15.501 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:15.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:15.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:15.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:15 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1826/148674 objects degraded (1.228%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:15.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:15 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1826/148674 objects degraded (1.228%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:15.637 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:15.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:15.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:15 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1826/148674 objects degraded (1.228%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:15.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:15.771 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:15.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:15.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:15.905 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:15.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:16.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:16.042 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:16.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:16.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:16.180 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:16.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:16.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:16.282 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:16.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:16.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:16.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:16.444 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:16.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:16.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:16.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:16.585 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:16.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:16.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:16 smithi114 ceph-mon[106969]: pgmap v1590: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.41k op/s; 1826/127293 objects degraded (1.434%); 2542/127293 objects misplaced (1.997%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:30:16.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:16 smithi114 ceph-mon[103029]: pgmap v1590: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.41k op/s; 1826/127293 objects degraded (1.434%); 2542/127293 objects misplaced (1.997%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:30:16.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:16 smithi160 ceph-mon[88407]: pgmap v1590: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.41k op/s; 1826/127293 objects degraded (1.434%); 2542/127293 objects misplaced (1.997%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:30:16.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:16.721 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:16.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:16.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:16.822 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:16.850 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:16.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:16.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:16.997 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:16.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:17.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:17.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:17.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:17.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:17.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:17.260 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:17.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:17.362 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:17.397 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:17.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:17.499 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:17.533 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:17.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:17.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:17.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:17.670 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:17.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:17.771 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:17.808 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:17.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:17.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:17.949 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:17.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:18.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:18.086 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:18.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:18.187 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:18.224 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:18.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:18.326 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:18.359 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:18.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:18.370 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:18 smithi114 ceph-mon[106969]: pgmap v1591: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.41k op/s; 1826/127293 objects degraded (1.434%); 2542/127293 objects misplaced (1.997%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:30:18.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:18 smithi114 ceph-mon[103029]: pgmap v1591: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.41k op/s; 1826/127293 objects degraded (1.434%); 2542/127293 objects misplaced (1.997%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:30:18.460 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:18.490 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:30:18.494 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:30:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:18.496 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:18.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:18.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:18.637 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:18.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:18 smithi160 ceph-mon[88407]: pgmap v1591: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 1.41k op/s; 1826/127293 objects degraded (1.434%); 2542/127293 objects misplaced (1.997%); 2.8 MiB/s, 44 objects/s recovering 2024-09-14T04:30:18.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:18.769 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:18.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:18.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:18.913 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:18.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:19.014 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:19.044 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:19.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:19.145 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:19.176 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:19.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:19.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:19.278 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:19.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:19.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:19.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:19.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:19.448 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:19.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:19.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:19.581 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:19.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:19.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:19.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:19 smithi114 ceph-mon[106969]: from='client.37302 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:30:19.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:19 smithi114 ceph-mon[103029]: from='client.37302 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:30:19.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:19 smithi160 ceph-mon[88407]: from='client.37302 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:30:19.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:19.721 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:19.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:19.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:19.855 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:19.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:19.958 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:19.996 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:19.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:20.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:20.129 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:20.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:20.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:20.262 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:20.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:20.364 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:20.400 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:20.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:20.405 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2024-09-14T04:30:17.236057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2779551717,"num_objects":42431,"num_object_clones":0,"num_object_copies":127293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1826,"num_objects_misplaced":2542,"num_objects_unfound":0,"num_objects_dirty":42431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145096,"num_write_kb":3628704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":426711839,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":202525,"ondisk_log_size":202525,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":362,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":15427328,"kb_used_data":9018840,"kb_used_omap":20,"kb_used_meta":6408043,"kb_avail":453298432,"statfs":{"total":479975178240,"available":464177594368,"internally_reserved":0,"allocated":9235292160,"data_stored":9225197772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20935,"internal_metadata":6561836601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-923467776,"num_objects":-14091,"num_object_clones":0,"num_object_copies":-42273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-889,"num_objects_unfound":0,"num_objects_dirty":-14091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14091,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":29294592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003017"},"pg_stats":[{"pgid":"14.23","version":"422'8136","reported_seq":8272,"reported_epoch":422,"state":"active+recovery_wait","last_fresh":"2024-09-14T04:30:14.925666+0000","last_change":"2024-09-14T04:29:17.294908+0000","last_active":"2024-09-14T04:30:14.925666+0000","last_peered":"2024-09-14T04:30:14.925666+0000","last_clean":"2024-09-14T04:29:16.194741+0000","last_became_active":"2024-09-14T04:29:17.276054+0000","last_became_peered":"2024-09-14T04:29:17.276054+0000","last_unstale":"2024-09-14T04:30:14.925666+0000","last_undegraded":"2024-09-14T04:30:14.925666+0000","last_fullsized":"2024-09-14T04:30:14.925666+0000","mapping_epoch":416,"log_start":"390'1355","ondisk_log_start":"390'1355","created":416,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'4263","last_scrub_stamp":"2024-09-14T04:28:05.894214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:05.894214+0000","objects_scrubbed":2271,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42434560,"num_objects":648,"num_object_clones":0,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2578,"num_write_kb":66880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.22","version":"422'8118","reported_seq":8279,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:13.783397+0000","last_change":"2024-09-14T04:29:48.522753+0000","last_active":"2024-09-14T04:30:13.783397+0000","last_peered":"2024-09-14T04:30:13.783397+0000","last_clean":"2024-09-14T04:30:13.783397+0000","last_became_active":"2024-09-14T04:29:17.267099+0000","last_became_peered":"2024-09-14T04:29:17.267099+0000","last_unstale":"2024-09-14T04:30:13.783397+0000","last_undegraded":"2024-09-14T04:30:13.783397+0000","last_fullsized":"2024-09-14T04:30:13.783397+0000","mapping_epoch":416,"log_start":"390'529","ondisk_log_start":"390'529","created":416,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"396'4166","last_scrub_stamp":"2024-09-14T04:28:03.205488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:03.205488+0000","objects_scrubbed":2056,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.69549441300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":683,"num_object_clones":0,"num_object_copies":2049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2587,"num_write_kb":71744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.21","version":"422'7945","reported_seq":8106,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.830937+0000","last_change":"2024-09-14T04:29:22.062807+0000","last_active":"2024-09-14T04:30:14.830937+0000","last_peered":"2024-09-14T04:30:14.830937+0000","last_clean":"2024-09-14T04:30:14.830937+0000","last_became_active":"2024-09-14T04:29:17.267078+0000","last_became_peered":"2024-09-14T04:29:17.267078+0000","last_unstale":"2024-09-14T04:30:14.830937+0000","last_undegraded":"2024-09-14T04:30:14.830937+0000","last_fullsized":"2024-09-14T04:30:14.830937+0000","mapping_epoch":416,"log_start":"390'1036","ondisk_log_start":"390'1036","created":416,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"418'6866","last_scrub_stamp":"2024-09-14T04:29:22.062488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:22.062488+0000","objects_scrubbed":1734,"log_size":6909,"log_dups_size":0,"ondisk_log_size":6909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:48:13.716818+0000","scrub_duration":428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42532864,"num_objects":649,"num_object_clones":0,"num_object_copies":1947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1085,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.20","version":"422'8117","reported_seq":8283,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.760960+0000","last_change":"2024-09-14T04:29:21.224859+0000","last_active":"2024-09-14T04:30:14.760960+0000","last_peered":"2024-09-14T04:30:14.760960+0000","last_clean":"2024-09-14T04:30:14.760960+0000","last_became_active":"2024-09-14T04:29:17.255764+0000","last_became_peered":"2024-09-14T04:29:17.255764+0000","last_unstale":"2024-09-14T04:30:14.760960+0000","last_undegraded":"2024-09-14T04:30:14.760960+0000","last_fullsized":"2024-09-14T04:30:14.760960+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":416,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"418'6946","last_scrub_stamp":"2024-09-14T04:29:21.224528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:21.224528+0000","objects_scrubbed":1797,"log_size":8117,"log_dups_size":0,"ondisk_log_size":8117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:55:01.670841+0000","scrub_duration":500,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40697856,"num_objects":621,"num_object_clones":0,"num_object_copies":1863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1176,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1e","version":"422'12086","reported_seq":12229,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.791287+0000","last_change":"2024-09-14T04:29:25.799582+0000","last_active":"2024-09-14T04:30:14.791287+0000","last_peered":"2024-09-14T04:30:14.791287+0000","last_clean":"2024-09-14T04:30:14.791287+0000","last_became_active":"2024-09-14T04:29:17.276779+0000","last_became_peered":"2024-09-14T04:29:17.276779+0000","last_unstale":"2024-09-14T04:30:14.791287+0000","last_undegraded":"2024-09-14T04:30:14.791287+0000","last_fullsized":"2024-09-14T04:30:14.791287+0000","mapping_epoch":416,"log_start":"401'5283","ondisk_log_start":"401'5283","created":416,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"418'9993","last_scrub_stamp":"2024-09-14T04:29:25.799477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:25.799477+0000","objects_scrubbed":3509,"log_size":6803,"log_dups_size":0,"ondisk_log_size":6803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:47:10.179259+0000","scrub_duration":1026,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91095040,"num_objects":1390,"num_object_clones":0,"num_object_copies":4170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2119,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":681,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.783987+0000","last_change":"2024-09-14T04:25:42.350690+0000","last_active":"2024-09-14T04:30:08.783987+0000","last_peered":"2024-09-14T04:30:08.783987+0000","last_clean":"2024-09-14T04:30:08.783987+0000","last_became_active":"2024-09-14T04:25:42.350363+0000","last_became_peered":"2024-09-14T04:25:42.350363+0000","last_unstale":"2024-09-14T04:30:08.783987+0000","last_undegraded":"2024-09-14T04:30:08.783987+0000","last_fullsized":"2024-09-14T04:30:08.783987+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:11:38.039845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":141,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.759928+0000","last_change":"2024-09-14T04:25:42.287293+0000","last_active":"2024-09-14T04:30:08.759928+0000","last_peered":"2024-09-14T04:30:08.759928+0000","last_clean":"2024-09-14T04:30:08.759928+0000","last_became_active":"2024-09-14T04:25:42.237213+0000","last_became_peered":"2024-09-14T04:25:42.237213+0000","last_unstale":"2024-09-14T04:30:08.759928+0000","last_undegraded":"2024-09-14T04:30:08.759928+0000","last_fullsized":"2024-09-14T04:30:08.759928+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:49:14.895808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"422'12110","reported_seq":12255,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.915890+0000","last_change":"2024-09-14T04:29:21.638363+0000","last_active":"2024-09-14T04:30:14.915890+0000","last_peered":"2024-09-14T04:30:14.915890+0000","last_clean":"2024-09-14T04:30:14.915890+0000","last_became_active":"2024-09-14T04:29:17.263896+0000","last_became_peered":"2024-09-14T04:29:17.263896+0000","last_unstale":"2024-09-14T04:30:14.915890+0000","last_undegraded":"2024-09-14T04:30:14.915890+0000","last_fullsized":"2024-09-14T04:30:14.915890+0000","mapping_epoch":416,"log_start":"406'6874","ondisk_log_start":"406'6874","created":416,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"418'9773","last_scrub_stamp":"2024-09-14T04:29:21.638034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:21.638034+0000","objects_scrubbed":3687,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:33:22.586959+0000","scrub_duration":971,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86769664,"num_objects":1324,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2363,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":127,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387277+0000","last_change":"2024-09-14T04:27:55.395641+0000","last_active":"2024-09-14T04:29:51.387277+0000","last_peered":"2024-09-14T04:29:51.387277+0000","last_clean":"2024-09-14T04:29:51.387277+0000","last_became_active":"2024-09-14T04:27:55.395369+0000","last_became_peered":"2024-09-14T04:27:55.395369+0000","last_unstale":"2024-09-14T04:29:51.387277+0000","last_undegraded":"2024-09-14T04:29:51.387277+0000","last_fullsized":"2024-09-14T04:29:51.387277+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:09:44.628733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":506,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.769690+0000","last_change":"2024-09-14T04:26:18.735795+0000","last_active":"2024-09-14T04:30:08.769690+0000","last_peered":"2024-09-14T04:30:08.769690+0000","last_clean":"2024-09-14T04:30:08.769690+0000","last_became_active":"2024-09-14T04:26:18.735453+0000","last_became_peered":"2024-09-14T04:26:18.735453+0000","last_unstale":"2024-09-14T04:30:08.769690+0000","last_undegraded":"2024-09-14T04:30:08.769690+0000","last_fullsized":"2024-09-14T04:30:08.769690+0000","mapping_epoch":373,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:11:19.345201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1c","version":"422'12195","reported_seq":12339,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.748485+0000","last_change":"2024-09-14T04:29:23.757301+0000","last_active":"2024-09-14T04:30:14.748485+0000","last_peered":"2024-09-14T04:30:14.748485+0000","last_clean":"2024-09-14T04:30:14.748485+0000","last_became_active":"2024-09-14T04:29:17.275023+0000","last_became_peered":"2024-09-14T04:29:17.275023+0000","last_unstale":"2024-09-14T04:30:14.748485+0000","last_undegraded":"2024-09-14T04:30:14.748485+0000","last_fullsized":"2024-09-14T04:30:14.748485+0000","mapping_epoch":416,"log_start":"401'5323","ondisk_log_start":"401'5323","created":416,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"418'9949","last_scrub_stamp":"2024-09-14T04:29:23.756764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:23.756764+0000","objects_scrubbed":3590,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:37:41.827385+0000","scrub_duration":1025,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86376448,"num_objects":1318,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2272,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":39,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386939+0000","last_change":"2024-09-14T04:27:55.402319+0000","last_active":"2024-09-14T04:29:51.386939+0000","last_peered":"2024-09-14T04:29:51.386939+0000","last_clean":"2024-09-14T04:29:51.386939+0000","last_became_active":"2024-09-14T04:27:55.401998+0000","last_became_peered":"2024-09-14T04:27:55.401998+0000","last_unstale":"2024-09-14T04:29:51.386939+0000","last_undegraded":"2024-09-14T04:29:51.386939+0000","last_fullsized":"2024-09-14T04:29:51.386939+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:37.324383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":394,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387806+0000","last_change":"2024-09-14T04:26:15.737623+0000","last_active":"2024-09-14T04:29:51.387806+0000","last_peered":"2024-09-14T04:29:51.387806+0000","last_clean":"2024-09-14T04:29:51.387806+0000","last_became_active":"2024-09-14T04:26:15.731253+0000","last_became_peered":"2024-09-14T04:26:15.731253+0000","last_unstale":"2024-09-14T04:29:51.387806+0000","last_undegraded":"2024-09-14T04:29:51.387806+0000","last_fullsized":"2024-09-14T04:29:51.387806+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:25:47.213793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"422'11906","reported_seq":11965,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.076451+0000","last_change":"2024-09-14T04:29:45.511721+0000","last_active":"2024-09-14T04:30:14.076451+0000","last_peered":"2024-09-14T04:30:14.076451+0000","last_clean":"2024-09-14T04:30:14.076451+0000","last_became_active":"2024-09-14T04:29:17.277094+0000","last_became_peered":"2024-09-14T04:29:17.277094+0000","last_unstale":"2024-09-14T04:30:14.076451+0000","last_undegraded":"2024-09-14T04:30:14.076451+0000","last_fullsized":"2024-09-14T04:30:14.076451+0000","mapping_epoch":416,"log_start":"401'6131","ondisk_log_start":"401'6131","created":416,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89391116,"num_objects":1365,"num_object_clones":0,"num_object_copies":4095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7189,"num_write_kb":273665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":494,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.774031+0000","last_change":"2024-09-14T04:25:42.334456+0000","last_active":"2024-09-14T04:30:08.774031+0000","last_peered":"2024-09-14T04:30:08.774031+0000","last_clean":"2024-09-14T04:30:08.774031+0000","last_became_active":"2024-09-14T04:25:42.334177+0000","last_became_peered":"2024-09-14T04:25:42.334177+0000","last_unstale":"2024-09-14T04:30:08.774031+0000","last_undegraded":"2024-09-14T04:30:08.774031+0000","last_fullsized":"2024-09-14T04:30:08.774031+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:12:07.154369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":455,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387809+0000","last_change":"2024-09-14T04:26:15.744615+0000","last_active":"2024-09-14T04:29:51.387809+0000","last_peered":"2024-09-14T04:29:51.387809+0000","last_clean":"2024-09-14T04:29:51.387809+0000","last_became_active":"2024-09-14T04:26:15.726296+0000","last_became_peered":"2024-09-14T04:26:15.726296+0000","last_unstale":"2024-09-14T04:29:51.387809+0000","last_undegraded":"2024-09-14T04:29:51.387809+0000","last_fullsized":"2024-09-14T04:29:51.387809+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:46:10.656996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.2","version":"422'8129","reported_seq":8291,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.032748+0000","last_change":"2024-09-14T04:29:43.586381+0000","last_active":"2024-09-14T04:30:14.032748+0000","last_peered":"2024-09-14T04:30:14.032748+0000","last_clean":"2024-09-14T04:30:14.032748+0000","last_became_active":"2024-09-14T04:29:17.249747+0000","last_became_peered":"2024-09-14T04:29:17.249747+0000","last_unstale":"2024-09-14T04:30:14.032748+0000","last_undegraded":"2024-09-14T04:30:14.032748+0000","last_fullsized":"2024-09-14T04:30:14.032748+0000","mapping_epoch":385,"log_start":"390'529","ondisk_log_start":"390'529","created":385,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4166","last_scrub_stamp":"2024-09-14T04:28:03.205488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:03.205488+0000","objects_scrubbed":2056,"log_size":7600,"log_dups_size":0,"ondisk_log_size":7600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.69549441300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44040192,"num_objects":672,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2598,"num_write_kb":71744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":421,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387169+0000","last_change":"2024-09-14T04:26:15.750280+0000","last_active":"2024-09-14T04:29:51.387169+0000","last_peered":"2024-09-14T04:29:51.387169+0000","last_clean":"2024-09-14T04:29:51.387169+0000","last_became_active":"2024-09-14T04:26:15.736153+0000","last_became_peered":"2024-09-14T04:26:15.736153+0000","last_unstale":"2024-09-14T04:29:51.387169+0000","last_undegraded":"2024-09-14T04:29:51.387169+0000","last_fullsized":"2024-09-14T04:29:51.387169+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:00:25.087633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":117,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386377+0000","last_change":"2024-09-14T04:23:09.282698+0000","last_active":"2024-09-14T04:29:51.386377+0000","last_peered":"2024-09-14T04:29:51.386377+0000","last_clean":"2024-09-14T04:29:51.386377+0000","last_became_active":"2024-09-14T04:23:09.282181+0000","last_became_peered":"2024-09-14T04:23:09.282181+0000","last_unstale":"2024-09-14T04:29:51.386377+0000","last_undegraded":"2024-09-14T04:29:51.386377+0000","last_fullsized":"2024-09-14T04:29:51.386377+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:18:12.880523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"422'8120","reported_seq":8285,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.910520+0000","last_change":"2024-09-14T04:29:18.163580+0000","last_active":"2024-09-14T04:30:14.910520+0000","last_peered":"2024-09-14T04:30:14.910520+0000","last_clean":"2024-09-14T04:30:14.910520+0000","last_became_active":"2024-09-14T04:29:17.256289+0000","last_became_peered":"2024-09-14T04:29:17.256289+0000","last_unstale":"2024-09-14T04:30:14.910520+0000","last_undegraded":"2024-09-14T04:30:14.910520+0000","last_fullsized":"2024-09-14T04:30:14.910520+0000","mapping_epoch":385,"log_start":"390'1355","ondisk_log_start":"390'1355","created":385,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'6878","last_scrub_stamp":"2024-09-14T04:29:18.163357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:18.163357+0000","objects_scrubbed":1907,"log_size":6765,"log_dups_size":0,"ondisk_log_size":6765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:34:31.693567+0000","scrub_duration":542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43188224,"num_objects":659,"num_object_clones":0,"num_object_copies":1977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1248,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":98,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387102+0000","last_change":"2024-09-14T04:26:15.755751+0000","last_active":"2024-09-14T04:29:51.387102+0000","last_peered":"2024-09-14T04:29:51.387102+0000","last_clean":"2024-09-14T04:29:51.387102+0000","last_became_active":"2024-09-14T04:26:15.745572+0000","last_became_peered":"2024-09-14T04:26:15.745572+0000","last_unstale":"2024-09-14T04:29:51.387102+0000","last_undegraded":"2024-09-14T04:29:51.387102+0000","last_fullsized":"2024-09-14T04:29:51.387102+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:54:48.838252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":463,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.766732+0000","last_change":"2024-09-14T04:24:47.613225+0000","last_active":"2024-09-14T04:30:08.766732+0000","last_peered":"2024-09-14T04:30:08.766732+0000","last_clean":"2024-09-14T04:30:08.766732+0000","last_became_active":"2024-09-14T04:24:47.612805+0000","last_became_peered":"2024-09-14T04:24:47.612805+0000","last_unstale":"2024-09-14T04:30:08.766732+0000","last_undegraded":"2024-09-14T04:30:08.766732+0000","last_fullsized":"2024-09-14T04:30:08.766732+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:35.992716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.0","version":"422'8155","reported_seq":8324,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.748534+0000","last_change":"2024-09-14T04:29:19.186533+0000","last_active":"2024-09-14T04:30:14.748534+0000","last_peered":"2024-09-14T04:30:14.748534+0000","last_clean":"2024-09-14T04:30:14.748534+0000","last_became_active":"2024-09-14T04:29:17.244571+0000","last_became_peered":"2024-09-14T04:29:17.244571+0000","last_unstale":"2024-09-14T04:30:14.748534+0000","last_undegraded":"2024-09-14T04:30:14.748534+0000","last_fullsized":"2024-09-14T04:30:14.748534+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":385,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'6889","last_scrub_stamp":"2024-09-14T04:29:19.186188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:19.186188+0000","objects_scrubbed":1950,"log_size":8155,"log_dups_size":0,"ondisk_log_size":8155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:10:59.978926+0000","scrub_duration":497,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44367872,"num_objects":677,"num_object_clones":0,"num_object_copies":2031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1273,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":125,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387077+0000","last_change":"2024-09-14T04:25:03.878325+0000","last_active":"2024-09-14T04:29:51.387077+0000","last_peered":"2024-09-14T04:29:51.387077+0000","last_clean":"2024-09-14T04:29:51.387077+0000","last_became_active":"2024-09-14T04:25:03.861286+0000","last_became_peered":"2024-09-14T04:25:03.861286+0000","last_unstale":"2024-09-14T04:29:51.387077+0000","last_undegraded":"2024-09-14T04:29:51.387077+0000","last_fullsized":"2024-09-14T04:29:51.387077+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:57:29.738051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":436,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387978+0000","last_change":"2024-09-14T04:26:15.726841+0000","last_active":"2024-09-14T04:29:51.387978+0000","last_peered":"2024-09-14T04:29:51.387978+0000","last_clean":"2024-09-14T04:29:51.387978+0000","last_became_active":"2024-09-14T04:26:15.725967+0000","last_became_peered":"2024-09-14T04:26:15.725967+0000","last_unstale":"2024-09-14T04:29:51.387978+0000","last_undegraded":"2024-09-14T04:29:51.387978+0000","last_fullsized":"2024-09-14T04:29:51.387978+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:30:20.743135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1","version":"422'7989","reported_seq":8152,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.925032+0000","last_change":"2024-09-14T04:29:20.105985+0000","last_active":"2024-09-14T04:30:14.925032+0000","last_peered":"2024-09-14T04:30:14.925032+0000","last_clean":"2024-09-14T04:30:14.925032+0000","last_became_active":"2024-09-14T04:29:17.260825+0000","last_became_peered":"2024-09-14T04:29:17.260825+0000","last_unstale":"2024-09-14T04:30:14.925032+0000","last_undegraded":"2024-09-14T04:30:14.925032+0000","last_fullsized":"2024-09-14T04:30:14.925032+0000","mapping_epoch":385,"log_start":"390'1036","ondisk_log_start":"390'1036","created":385,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'6806","last_scrub_stamp":"2024-09-14T04:29:20.105723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:20.105723+0000","objects_scrubbed":1840,"log_size":6953,"log_dups_size":0,"ondisk_log_size":6953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:46.893168+0000","scrub_duration":441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42205184,"num_objects":644,"num_object_clones":0,"num_object_copies":1932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1196,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":107,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386768+0000","last_change":"2024-09-14T04:22:08.468750+0000","last_active":"2024-09-14T04:29:51.386768+0000","last_peered":"2024-09-14T04:29:51.386768+0000","last_clean":"2024-09-14T04:29:51.386768+0000","last_became_active":"2024-09-14T04:22:08.468217+0000","last_became_peered":"2024-09-14T04:22:08.468217+0000","last_unstale":"2024-09-14T04:29:51.386768+0000","last_undegraded":"2024-09-14T04:29:51.386768+0000","last_fullsized":"2024-09-14T04:29:51.386768+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:06.709182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":169,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.770658+0000","last_change":"2024-09-14T04:26:15.732663+0000","last_active":"2024-09-14T04:30:08.770658+0000","last_peered":"2024-09-14T04:30:08.770658+0000","last_clean":"2024-09-14T04:30:08.770658+0000","last_became_active":"2024-09-14T04:26:15.731910+0000","last_became_peered":"2024-09-14T04:26:15.731910+0000","last_unstale":"2024-09-14T04:30:08.770658+0000","last_undegraded":"2024-09-14T04:30:08.770658+0000","last_fullsized":"2024-09-14T04:30:08.770658+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:21:44.768223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.6","version":"422'9372","reported_seq":9501,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.819929+0000","last_change":"2024-09-14T04:28:03.114846+0000","last_active":"2024-09-14T04:30:14.819929+0000","last_peered":"2024-09-14T04:30:14.819929+0000","last_clean":"2024-09-14T04:30:14.819929+0000","last_became_active":"2024-09-14T04:28:02.388695+0000","last_became_peered":"2024-09-14T04:28:02.388695+0000","last_unstale":"2024-09-14T04:30:14.819929+0000","last_undegraded":"2024-09-14T04:30:14.819929+0000","last_fullsized":"2024-09-14T04:30:14.819929+0000","mapping_epoch":385,"log_start":"401'4621","ondisk_log_start":"401'4621","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4203","last_scrub_stamp":"2024-09-14T04:28:03.114713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:03.114713+0000","objects_scrubbed":2118,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:17:13.476881+0000","scrub_duration":513,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86966272,"num_objects":1327,"num_object_clones":0,"num_object_copies":3981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5183,"num_write_kb":140544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"76'100","reported_seq":100,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.766450+0000","last_change":"2024-09-14T04:27:43.391069+0000","last_active":"2024-09-14T04:30:08.766450+0000","last_peered":"2024-09-14T04:30:08.766450+0000","last_clean":"2024-09-14T04:30:08.766450+0000","last_became_active":"2024-09-14T04:27:12.689443+0000","last_became_peered":"2024-09-14T04:27:12.689443+0000","last_unstale":"2024-09-14T04:30:08.766450+0000","last_undegraded":"2024-09-14T04:30:08.766450+0000","last_fullsized":"2024-09-14T04:30:08.766450+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:43.390999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:43.390999+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:55:51.689081+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":722,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.771311+0000","last_change":"2024-09-14T04:26:15.751894+0000","last_active":"2024-09-14T04:30:08.771311+0000","last_peered":"2024-09-14T04:30:08.771311+0000","last_clean":"2024-09-14T04:30:08.771311+0000","last_became_active":"2024-09-14T04:26:15.731434+0000","last_became_peered":"2024-09-14T04:26:15.731434+0000","last_unstale":"2024-09-14T04:30:08.771311+0000","last_undegraded":"2024-09-14T04:30:08.771311+0000","last_fullsized":"2024-09-14T04:30:08.771311+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:17.854732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":547,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.784076+0000","last_change":"2024-09-14T04:17:48.083765+0000","last_active":"2024-09-14T04:30:08.784076+0000","last_peered":"2024-09-14T04:30:08.784076+0000","last_clean":"2024-09-14T04:30:08.784076+0000","last_became_active":"2024-09-14T04:17:48.083385+0000","last_became_peered":"2024-09-14T04:17:48.083385+0000","last_unstale":"2024-09-14T04:30:08.784076+0000","last_undegraded":"2024-09-14T04:30:08.784076+0000","last_fullsized":"2024-09-14T04:30:08.784076+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:13:49.268399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.7","version":"422'10013","reported_seq":10167,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.345771+0000","last_change":"2024-09-14T04:28:07.388643+0000","last_active":"2024-09-14T04:30:14.345771+0000","last_peered":"2024-09-14T04:30:14.345771+0000","last_clean":"2024-09-14T04:30:14.345771+0000","last_became_active":"2024-09-14T04:28:02.428617+0000","last_became_peered":"2024-09-14T04:28:02.428617+0000","last_unstale":"2024-09-14T04:30:14.345771+0000","last_undegraded":"2024-09-14T04:30:14.345771+0000","last_fullsized":"2024-09-14T04:30:14.345771+0000","mapping_epoch":385,"log_start":"405'5986","ondisk_log_start":"405'5986","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'4926","last_scrub_stamp":"2024-09-14T04:28:07.388588+0000","last_deep_scrub":"398'4926","last_deep_scrub_stamp":"2024-09-14T04:28:07.388588+0000","last_clean_scrub_stamp":"2024-09-14T04:28:07.388588+0000","objects_scrubbed":2217,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:39:05.724015+0000","scrub_duration":2.2206136920000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86376448,"num_objects":1318,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5089,"num_write_kb":134080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"76'100","reported_seq":63,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.385858+0000","last_change":"2024-09-14T04:27:46.144096+0000","last_active":"2024-09-14T04:29:51.385858+0000","last_peered":"2024-09-14T04:29:51.385858+0000","last_clean":"2024-09-14T04:29:51.385858+0000","last_became_active":"2024-09-14T04:27:12.688140+0000","last_became_peered":"2024-09-14T04:27:12.688140+0000","last_unstale":"2024-09-14T04:29:51.385858+0000","last_undegraded":"2024-09-14T04:29:51.385858+0000","last_fullsized":"2024-09-14T04:29:51.385858+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:46.144003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:46.144003+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:00:07.652536+0000","scrub_duration":0.0071911659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":417,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388303+0000","last_change":"2024-09-14T04:24:47.615423+0000","last_active":"2024-09-14T04:29:51.388303+0000","last_peered":"2024-09-14T04:29:51.388303+0000","last_clean":"2024-09-14T04:29:51.388303+0000","last_became_active":"2024-09-14T04:24:47.609944+0000","last_became_peered":"2024-09-14T04:24:47.609944+0000","last_unstale":"2024-09-14T04:29:51.388303+0000","last_undegraded":"2024-09-14T04:29:51.388303+0000","last_fullsized":"2024-09-14T04:29:51.388303+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:30:03.475490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":136,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.761707+0000","last_change":"2024-09-14T04:25:42.247602+0000","last_active":"2024-09-14T04:30:08.761707+0000","last_peered":"2024-09-14T04:30:08.761707+0000","last_clean":"2024-09-14T04:30:08.761707+0000","last_became_active":"2024-09-14T04:25:42.247304+0000","last_became_peered":"2024-09-14T04:25:42.247304+0000","last_unstale":"2024-09-14T04:30:08.761707+0000","last_undegraded":"2024-09-14T04:30:08.761707+0000","last_fullsized":"2024-09-14T04:30:08.761707+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:18:17.260700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"422'9643","reported_seq":9793,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.062980+0000","last_change":"2024-09-14T04:28:06.867085+0000","last_active":"2024-09-14T04:30:14.062980+0000","last_peered":"2024-09-14T04:30:14.062980+0000","last_clean":"2024-09-14T04:30:14.062980+0000","last_became_active":"2024-09-14T04:28:02.414511+0000","last_became_peered":"2024-09-14T04:28:02.414511+0000","last_unstale":"2024-09-14T04:30:14.062980+0000","last_undegraded":"2024-09-14T04:30:14.062980+0000","last_fullsized":"2024-09-14T04:30:14.062980+0000","mapping_epoch":385,"log_start":"401'5513","ondisk_log_start":"401'5513","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'4718","last_scrub_stamp":"2024-09-14T04:28:06.866991+0000","last_deep_scrub":"398'4718","last_deep_scrub_stamp":"2024-09-14T04:28:06.866991+0000","last_clean_scrub_stamp":"2024-09-14T04:28:06.866991+0000","objects_scrubbed":2158,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:10:32.942361+0000","scrub_duration":3.3310999840000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87752704,"num_objects":1339,"num_object_clones":0,"num_object_copies":4017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4941,"num_write_kb":131904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":279,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.758217+0000","last_change":"2024-09-14T04:25:53.296317+0000","last_active":"2024-09-14T04:30:08.758217+0000","last_peered":"2024-09-14T04:30:08.758217+0000","last_clean":"2024-09-14T04:30:08.758217+0000","last_became_active":"2024-09-14T04:25:53.296039+0000","last_became_peered":"2024-09-14T04:25:53.296039+0000","last_unstale":"2024-09-14T04:30:08.758217+0000","last_undegraded":"2024-09-14T04:30:08.758217+0000","last_fullsized":"2024-09-14T04:30:08.758217+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:26:31.287496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":299,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388479+0000","last_change":"2024-09-14T04:24:47.616564+0000","last_active":"2024-09-14T04:29:51.388479+0000","last_peered":"2024-09-14T04:29:51.388479+0000","last_clean":"2024-09-14T04:29:51.388479+0000","last_became_active":"2024-09-14T04:24:47.616196+0000","last_became_peered":"2024-09-14T04:24:47.616196+0000","last_unstale":"2024-09-14T04:29:51.388479+0000","last_undegraded":"2024-09-14T04:29:51.388479+0000","last_fullsized":"2024-09-14T04:29:51.388479+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:38:56.531650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"422'9435","reported_seq":9568,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.907168+0000","last_change":"2024-09-14T04:28:05.093393+0000","last_active":"2024-09-14T04:30:14.907168+0000","last_peered":"2024-09-14T04:30:14.907168+0000","last_clean":"2024-09-14T04:30:14.907168+0000","last_became_active":"2024-09-14T04:28:02.411968+0000","last_became_peered":"2024-09-14T04:28:02.411968+0000","last_unstale":"2024-09-14T04:30:14.907168+0000","last_undegraded":"2024-09-14T04:30:14.907168+0000","last_fullsized":"2024-09-14T04:30:14.907168+0000","mapping_epoch":385,"log_start":"406'5470","ondisk_log_start":"406'5470","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4326","last_scrub_stamp":"2024-09-14T04:28:05.093275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:05.093275+0000","objects_scrubbed":2220,"log_size":3965,"log_dups_size":0,"ondisk_log_size":3965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:00:40.272674+0000","scrub_duration":584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82706432,"num_objects":1262,"num_object_clones":0,"num_object_copies":3786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5120,"num_write_kb":133184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'100","reported_seq":58,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386381+0000","last_change":"2024-09-14T04:27:35.272869+0000","last_active":"2024-09-14T04:29:51.386381+0000","last_peered":"2024-09-14T04:29:51.386381+0000","last_clean":"2024-09-14T04:29:51.386381+0000","last_became_active":"2024-09-14T04:27:12.688026+0000","last_became_peered":"2024-09-14T04:27:12.688026+0000","last_unstale":"2024-09-14T04:29:51.386381+0000","last_undegraded":"2024-09-14T04:29:51.386381+0000","last_fullsized":"2024-09-14T04:29:51.386381+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:35.272785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:35.272785+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:00:40.802976+0000","scrub_duration":0.0029595770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":488,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.766632+0000","last_change":"2024-09-14T04:25:03.848027+0000","last_active":"2024-09-14T04:30:08.766632+0000","last_peered":"2024-09-14T04:30:08.766632+0000","last_clean":"2024-09-14T04:30:08.766632+0000","last_became_active":"2024-09-14T04:25:03.847621+0000","last_became_peered":"2024-09-14T04:25:03.847621+0000","last_unstale":"2024-09-14T04:30:08.766632+0000","last_undegraded":"2024-09-14T04:30:08.766632+0000","last_fullsized":"2024-09-14T04:30:08.766632+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:54:52.585026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":523,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.774064+0000","last_change":"2024-09-14T04:25:03.856805+0000","last_active":"2024-09-14T04:30:08.774064+0000","last_peered":"2024-09-14T04:30:08.774064+0000","last_clean":"2024-09-14T04:30:08.774064+0000","last_became_active":"2024-09-14T04:25:03.856459+0000","last_became_peered":"2024-09-14T04:25:03.856459+0000","last_unstale":"2024-09-14T04:30:08.774064+0000","last_undegraded":"2024-09-14T04:30:08.774064+0000","last_fullsized":"2024-09-14T04:30:08.774064+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:17:41.568556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.f","version":"422'12036","reported_seq":12184,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.925445+0000","last_change":"2024-09-14T04:29:19.684923+0000","last_active":"2024-09-14T04:30:14.925445+0000","last_peered":"2024-09-14T04:30:14.925445+0000","last_clean":"2024-09-14T04:30:14.925445+0000","last_became_active":"2024-09-14T04:29:17.255896+0000","last_became_peered":"2024-09-14T04:29:17.255896+0000","last_unstale":"2024-09-14T04:30:14.925445+0000","last_undegraded":"2024-09-14T04:30:14.925445+0000","last_fullsized":"2024-09-14T04:30:14.925445+0000","mapping_epoch":385,"log_start":"405'6697","ondisk_log_start":"405'6697","created":385,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'9686","last_scrub_stamp":"2024-09-14T04:29:19.684589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:19.684589+0000","objects_scrubbed":3672,"log_size":5339,"log_dups_size":0,"ondisk_log_size":5339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:56:56.505653+0000","scrub_duration":1044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85131264,"num_objects":1299,"num_object_clones":0,"num_object_copies":3897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2373,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":294,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387952+0000","last_change":"2024-09-14T04:27:34.456942+0000","last_active":"2024-09-14T04:29:51.387952+0000","last_peered":"2024-09-14T04:29:51.387952+0000","last_clean":"2024-09-14T04:29:51.387952+0000","last_became_active":"2024-09-14T04:26:15.726724+0000","last_became_peered":"2024-09-14T04:26:15.726724+0000","last_unstale":"2024-09-14T04:29:51.387952+0000","last_undegraded":"2024-09-14T04:29:51.387952+0000","last_fullsized":"2024-09-14T04:29:51.387952+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:34.456083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:34.456083+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:01:39.830454+0000","scrub_duration":0.0045693399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":147,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386628+0000","last_change":"2024-09-14T04:25:52.311202+0000","last_active":"2024-09-14T04:29:51.386628+0000","last_peered":"2024-09-14T04:29:51.386628+0000","last_clean":"2024-09-14T04:29:51.386628+0000","last_became_active":"2024-09-14T04:25:42.253580+0000","last_became_peered":"2024-09-14T04:25:42.253580+0000","last_unstale":"2024-09-14T04:29:51.386628+0000","last_undegraded":"2024-09-14T04:29:51.386628+0000","last_fullsized":"2024-09-14T04:29:51.386628+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:59:26.500812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":488,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.770108+0000","last_change":"2024-09-14T04:22:08.472386+0000","last_active":"2024-09-14T04:30:08.770108+0000","last_peered":"2024-09-14T04:30:08.770108+0000","last_clean":"2024-09-14T04:30:08.770108+0000","last_became_active":"2024-09-14T04:22:08.472100+0000","last_became_peered":"2024-09-14T04:22:08.472100+0000","last_unstale":"2024-09-14T04:30:08.770108+0000","last_undegraded":"2024-09-14T04:30:08.770108+0000","last_fullsized":"2024-09-14T04:30:08.770108+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:36:03.114984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":409,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388535+0000","last_change":"2024-09-14T04:26:15.727747+0000","last_active":"2024-09-14T04:29:51.388535+0000","last_peered":"2024-09-14T04:29:51.388535+0000","last_clean":"2024-09-14T04:29:51.388535+0000","last_became_active":"2024-09-14T04:26:15.726210+0000","last_became_peered":"2024-09-14T04:26:15.726210+0000","last_unstale":"2024-09-14T04:29:51.388535+0000","last_undegraded":"2024-09-14T04:29:51.388535+0000","last_fullsized":"2024-09-14T04:29:51.388535+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:01.720621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.c","version":"422'12251","reported_seq":12400,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.786500+0000","last_change":"2024-09-14T04:29:20.659421+0000","last_active":"2024-09-14T04:30:14.786500+0000","last_peered":"2024-09-14T04:30:14.786500+0000","last_clean":"2024-09-14T04:30:14.786500+0000","last_became_active":"2024-09-14T04:29:17.244241+0000","last_became_peered":"2024-09-14T04:29:17.244241+0000","last_unstale":"2024-09-14T04:30:14.786500+0000","last_undegraded":"2024-09-14T04:30:14.786500+0000","last_fullsized":"2024-09-14T04:30:14.786500+0000","mapping_epoch":385,"log_start":"401'5524","ondisk_log_start":"401'5524","created":385,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'9828","last_scrub_stamp":"2024-09-14T04:29:20.659177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:20.659177+0000","objects_scrubbed":3730,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:16:12.948692+0000","scrub_duration":973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84475904,"num_objects":1289,"num_object_clones":0,"num_object_copies":3867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2441,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"76'100","reported_seq":303,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388070+0000","last_change":"2024-09-14T04:27:36.436313+0000","last_active":"2024-09-14T04:29:51.388070+0000","last_peered":"2024-09-14T04:29:51.388070+0000","last_clean":"2024-09-14T04:29:51.388070+0000","last_became_active":"2024-09-14T04:27:12.685740+0000","last_became_peered":"2024-09-14T04:27:12.685740+0000","last_unstale":"2024-09-14T04:29:51.388070+0000","last_undegraded":"2024-09-14T04:29:51.388070+0000","last_fullsized":"2024-09-14T04:29:51.388070+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:36.436230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:36.436230+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:26:41.485866+0000","scrub_duration":0.0026608669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":176,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.766358+0000","last_change":"2024-09-14T04:26:15.727773+0000","last_active":"2024-09-14T04:30:08.766358+0000","last_peered":"2024-09-14T04:30:08.766358+0000","last_clean":"2024-09-14T04:30:08.766358+0000","last_became_active":"2024-09-14T04:26:15.727477+0000","last_became_peered":"2024-09-14T04:26:15.727477+0000","last_unstale":"2024-09-14T04:30:08.766358+0000","last_undegraded":"2024-09-14T04:30:08.766358+0000","last_fullsized":"2024-09-14T04:30:08.766358+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:09:08.222067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":543,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.766111+0000","last_change":"2024-09-14T04:26:15.724485+0000","last_active":"2024-09-14T04:30:08.766111+0000","last_peered":"2024-09-14T04:30:08.766111+0000","last_clean":"2024-09-14T04:30:08.766111+0000","last_became_active":"2024-09-14T04:26:15.723989+0000","last_became_peered":"2024-09-14T04:26:15.723989+0000","last_unstale":"2024-09-14T04:30:08.766111+0000","last_undegraded":"2024-09-14T04:30:08.766111+0000","last_fullsized":"2024-09-14T04:30:08.766111+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:27:04.257761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":143,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.765515+0000","last_change":"2024-09-14T04:26:15.753288+0000","last_active":"2024-09-14T04:30:08.765515+0000","last_peered":"2024-09-14T04:30:08.765515+0000","last_clean":"2024-09-14T04:30:08.765515+0000","last_became_active":"2024-09-14T04:26:15.753003+0000","last_became_peered":"2024-09-14T04:26:15.753003+0000","last_unstale":"2024-09-14T04:30:08.765515+0000","last_undegraded":"2024-09-14T04:30:08.765515+0000","last_fullsized":"2024-09-14T04:30:08.765515+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:05:37.332422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.d","version":"422'11869","reported_seq":11929,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.074786+0000","last_change":"2024-09-14T04:29:40.592300+0000","last_active":"2024-09-14T04:30:14.074786+0000","last_peered":"2024-09-14T04:30:14.074786+0000","last_clean":"2024-09-14T04:30:14.074786+0000","last_became_active":"2024-09-14T04:29:17.242718+0000","last_became_peered":"2024-09-14T04:29:17.242718+0000","last_unstale":"2024-09-14T04:30:14.074786+0000","last_undegraded":"2024-09-14T04:30:14.074786+0000","last_fullsized":"2024-09-14T04:30:14.074786+0000","mapping_epoch":385,"log_start":"401'6110","ondisk_log_start":"401'6110","created":385,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91815948,"num_objects":1401,"num_object_clones":0,"num_object_copies":4203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7151,"num_write_kb":273664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"76'100","reported_seq":305,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.758452+0000","last_change":"2024-09-14T04:27:44.819744+0000","last_active":"2024-09-14T04:30:08.758452+0000","last_peered":"2024-09-14T04:30:08.758452+0000","last_clean":"2024-09-14T04:30:08.758452+0000","last_became_active":"2024-09-14T04:27:12.689439+0000","last_became_peered":"2024-09-14T04:27:12.689439+0000","last_unstale":"2024-09-14T04:30:08.758452+0000","last_undegraded":"2024-09-14T04:30:08.758452+0000","last_fullsized":"2024-09-14T04:30:08.758452+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:44.819663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:44.819663+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:25:22.854494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":143,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.758390+0000","last_change":"2024-09-14T04:26:15.723403+0000","last_active":"2024-09-14T04:30:08.758390+0000","last_peered":"2024-09-14T04:30:08.758390+0000","last_clean":"2024-09-14T04:30:08.758390+0000","last_became_active":"2024-09-14T04:26:15.723058+0000","last_became_peered":"2024-09-14T04:26:15.723058+0000","last_unstale":"2024-09-14T04:30:08.758390+0000","last_undegraded":"2024-09-14T04:30:08.758390+0000","last_fullsized":"2024-09-14T04:30:08.758390+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:53:41.162301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":142,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.758913+0000","last_change":"2024-09-14T04:26:15.727903+0000","last_active":"2024-09-14T04:30:08.758913+0000","last_peered":"2024-09-14T04:30:08.758913+0000","last_clean":"2024-09-14T04:30:08.758913+0000","last_became_active":"2024-09-14T04:26:15.727050+0000","last_became_peered":"2024-09-14T04:26:15.727050+0000","last_unstale":"2024-09-14T04:30:08.758913+0000","last_undegraded":"2024-09-14T04:30:08.758913+0000","last_fullsized":"2024-09-14T04:30:08.758913+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:27:59.930610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":381,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387498+0000","last_change":"2024-09-14T04:26:15.737341+0000","last_active":"2024-09-14T04:29:51.387498+0000","last_peered":"2024-09-14T04:29:51.387498+0000","last_clean":"2024-09-14T04:29:51.387498+0000","last_became_active":"2024-09-14T04:26:15.736045+0000","last_became_peered":"2024-09-14T04:26:15.736045+0000","last_unstale":"2024-09-14T04:29:51.387498+0000","last_undegraded":"2024-09-14T04:29:51.387498+0000","last_fullsized":"2024-09-14T04:29:51.387498+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:08:08.143667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.e","version":"422'12162","reported_seq":12307,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.784090+0000","last_change":"2024-09-14T04:29:22.790334+0000","last_active":"2024-09-14T04:30:14.784090+0000","last_peered":"2024-09-14T04:30:14.784090+0000","last_clean":"2024-09-14T04:30:14.784090+0000","last_became_active":"2024-09-14T04:29:17.255592+0000","last_became_peered":"2024-09-14T04:29:17.255592+0000","last_unstale":"2024-09-14T04:30:14.784090+0000","last_undegraded":"2024-09-14T04:30:14.784090+0000","last_fullsized":"2024-09-14T04:30:14.784090+0000","mapping_epoch":385,"log_start":"401'5223","ondisk_log_start":"401'5223","created":385,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'9910","last_scrub_stamp":"2024-09-14T04:29:22.789621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:22.789621+0000","objects_scrubbed":3578,"log_size":6939,"log_dups_size":0,"ondisk_log_size":6939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:36:57.208359+0000","scrub_duration":1033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"76'100","reported_seq":58,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386098+0000","last_change":"2024-09-14T04:27:43.130461+0000","last_active":"2024-09-14T04:29:51.386098+0000","last_peered":"2024-09-14T04:29:51.386098+0000","last_clean":"2024-09-14T04:29:51.386098+0000","last_became_active":"2024-09-14T04:27:12.687762+0000","last_became_peered":"2024-09-14T04:27:12.687762+0000","last_unstale":"2024-09-14T04:29:51.386098+0000","last_undegraded":"2024-09-14T04:29:51.386098+0000","last_fullsized":"2024-09-14T04:29:51.386098+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:43.130258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:43.130258+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:26:17.918367+0000","scrub_duration":0.0054891430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":139,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386024+0000","last_change":"2024-09-14T04:27:06.910535+0000","last_active":"2024-09-14T04:29:51.386024+0000","last_peered":"2024-09-14T04:29:51.386024+0000","last_clean":"2024-09-14T04:29:51.386024+0000","last_became_active":"2024-09-14T04:26:15.729031+0000","last_became_peered":"2024-09-14T04:26:15.729031+0000","last_unstale":"2024-09-14T04:29:51.386024+0000","last_undegraded":"2024-09-14T04:29:51.386024+0000","last_fullsized":"2024-09-14T04:29:51.386024+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:32:12.734036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":71,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.757899+0000","last_change":"2024-09-14T04:27:55.393446+0000","last_active":"2024-09-14T04:30:08.757899+0000","last_peered":"2024-09-14T04:30:08.757899+0000","last_clean":"2024-09-14T04:30:08.757899+0000","last_became_active":"2024-09-14T04:27:55.393202+0000","last_became_peered":"2024-09-14T04:27:55.393202+0000","last_unstale":"2024-09-14T04:30:08.757899+0000","last_undegraded":"2024-09-14T04:30:08.757899+0000","last_fullsized":"2024-09-14T04:30:08.757899+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:00:14.696662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":136,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.759951+0000","last_change":"2024-09-14T04:25:42.246310+0000","last_active":"2024-09-14T04:30:08.759951+0000","last_peered":"2024-09-14T04:30:08.759951+0000","last_clean":"2024-09-14T04:30:08.759951+0000","last_became_active":"2024-09-14T04:25:42.245997+0000","last_became_peered":"2024-09-14T04:25:42.245997+0000","last_unstale":"2024-09-14T04:30:08.759951+0000","last_undegraded":"2024-09-14T04:30:08.759951+0000","last_fullsized":"2024-09-14T04:30:08.759951+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:01:43.109388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.8","version":"422'9268","reported_seq":9400,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.795148+0000","last_change":"2024-09-14T04:28:05.139226+0000","last_active":"2024-09-14T04:30:14.795148+0000","last_peered":"2024-09-14T04:30:14.795148+0000","last_clean":"2024-09-14T04:30:14.795148+0000","last_became_active":"2024-09-14T04:28:02.407635+0000","last_became_peered":"2024-09-14T04:28:02.407635+0000","last_unstale":"2024-09-14T04:30:14.795148+0000","last_undegraded":"2024-09-14T04:30:14.795148+0000","last_fullsized":"2024-09-14T04:30:14.795148+0000","mapping_epoch":385,"log_start":"401'4580","ondisk_log_start":"401'4580","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4184","last_scrub_stamp":"2024-09-14T04:28:05.139073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:05.139073+0000","objects_scrubbed":2132,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:33:39.466052+0000","scrub_duration":552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88539136,"num_objects":1351,"num_object_clones":0,"num_object_copies":4053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5093,"num_write_kb":137984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"76'100","reported_seq":303,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388196+0000","last_change":"2024-09-14T04:27:38.451495+0000","last_active":"2024-09-14T04:29:51.388196+0000","last_peered":"2024-09-14T04:29:51.388196+0000","last_clean":"2024-09-14T04:29:51.388196+0000","last_became_active":"2024-09-14T04:27:12.687176+0000","last_became_peered":"2024-09-14T04:27:12.687176+0000","last_unstale":"2024-09-14T04:29:51.388196+0000","last_undegraded":"2024-09-14T04:29:51.388196+0000","last_fullsized":"2024-09-14T04:29:51.388196+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:38.451408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:38.451408+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:26:08.295464+0000","scrub_duration":0.0034959740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":127,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387212+0000","last_change":"2024-09-14T04:25:42.253453+0000","last_active":"2024-09-14T04:29:51.387212+0000","last_peered":"2024-09-14T04:29:51.387212+0000","last_clean":"2024-09-14T04:29:51.387212+0000","last_became_active":"2024-09-14T04:25:42.252529+0000","last_became_peered":"2024-09-14T04:25:42.252529+0000","last_unstale":"2024-09-14T04:29:51.387212+0000","last_undegraded":"2024-09-14T04:29:51.387212+0000","last_fullsized":"2024-09-14T04:29:51.387212+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:19.539802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":143,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.760239+0000","last_change":"2024-09-14T04:26:15.723378+0000","last_active":"2024-09-14T04:30:08.760239+0000","last_peered":"2024-09-14T04:30:08.760239+0000","last_clean":"2024-09-14T04:30:08.760239+0000","last_became_active":"2024-09-14T04:26:15.723070+0000","last_became_peered":"2024-09-14T04:26:15.723070+0000","last_unstale":"2024-09-14T04:30:08.760239+0000","last_undegraded":"2024-09-14T04:30:08.760239+0000","last_fullsized":"2024-09-14T04:30:08.760239+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:38:14.395886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":604,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386171+0000","last_change":"2024-09-14T04:26:15.723597+0000","last_active":"2024-09-14T04:29:51.386171+0000","last_peered":"2024-09-14T04:29:51.386171+0000","last_clean":"2024-09-14T04:29:51.386171+0000","last_became_active":"2024-09-14T04:26:15.723289+0000","last_became_peered":"2024-09-14T04:26:15.723289+0000","last_unstale":"2024-09-14T04:29:51.386171+0000","last_undegraded":"2024-09-14T04:29:51.386171+0000","last_fullsized":"2024-09-14T04:29:51.386171+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:55:41.253830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"422'9517","reported_seq":9672,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.318219+0000","last_change":"2024-09-14T04:28:04.855497+0000","last_active":"2024-09-14T04:30:14.318219+0000","last_peered":"2024-09-14T04:30:14.318219+0000","last_clean":"2024-09-14T04:30:14.318219+0000","last_became_active":"2024-09-14T04:28:02.428275+0000","last_became_peered":"2024-09-14T04:28:02.428275+0000","last_unstale":"2024-09-14T04:30:14.318219+0000","last_undegraded":"2024-09-14T04:30:14.318219+0000","last_fullsized":"2024-09-14T04:30:14.318219+0000","mapping_epoch":385,"log_start":"405'5471","ondisk_log_start":"405'5471","created":385,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'4320","last_scrub_stamp":"2024-09-14T04:28:04.855410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:04.855410+0000","objects_scrubbed":2256,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:52:26.414450+0000","scrub_duration":0.70843562500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5217,"num_write_kb":136064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"76'100","reported_seq":303,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388385+0000","last_change":"2024-09-14T04:27:43.482881+0000","last_active":"2024-09-14T04:29:51.388385+0000","last_peered":"2024-09-14T04:29:51.388385+0000","last_clean":"2024-09-14T04:29:51.388385+0000","last_became_active":"2024-09-14T04:27:12.686451+0000","last_became_peered":"2024-09-14T04:27:12.686451+0000","last_unstale":"2024-09-14T04:29:51.388385+0000","last_undegraded":"2024-09-14T04:29:51.388385+0000","last_fullsized":"2024-09-14T04:29:51.388385+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:43.482790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:43.482790+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:48:43.699493+0000","scrub_duration":0.013209995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":143,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.759804+0000","last_change":"2024-09-14T04:26:15.734625+0000","last_active":"2024-09-14T04:30:08.759804+0000","last_peered":"2024-09-14T04:30:08.759804+0000","last_clean":"2024-09-14T04:30:08.759804+0000","last_became_active":"2024-09-14T04:26:15.734309+0000","last_became_peered":"2024-09-14T04:26:15.734309+0000","last_unstale":"2024-09-14T04:30:08.759804+0000","last_undegraded":"2024-09-14T04:30:08.759804+0000","last_fullsized":"2024-09-14T04:30:08.759804+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:58:57.797342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":89,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388107+0000","last_change":"2024-09-14T04:26:15.727947+0000","last_active":"2024-09-14T04:29:51.388107+0000","last_peered":"2024-09-14T04:29:51.388107+0000","last_clean":"2024-09-14T04:29:51.388107+0000","last_became_active":"2024-09-14T04:26:15.726608+0000","last_became_peered":"2024-09-14T04:26:15.726608+0000","last_unstale":"2024-09-14T04:29:51.388107+0000","last_undegraded":"2024-09-14T04:29:51.388107+0000","last_fullsized":"2024-09-14T04:29:51.388107+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:33:03.772215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":664,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386555+0000","last_change":"2024-09-14T04:26:15.732599+0000","last_active":"2024-09-14T04:29:51.386555+0000","last_peered":"2024-09-14T04:29:51.386555+0000","last_clean":"2024-09-14T04:29:51.386555+0000","last_became_active":"2024-09-14T04:26:15.725658+0000","last_became_peered":"2024-09-14T04:26:15.725658+0000","last_unstale":"2024-09-14T04:29:51.386555+0000","last_undegraded":"2024-09-14T04:29:51.386555+0000","last_fullsized":"2024-09-14T04:29:51.386555+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:48:20.963473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.a","version":"422'12143","reported_seq":12197,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.379938+0000","last_change":"2024-09-14T04:29:17.255515+0000","last_active":"2024-09-14T04:30:14.379938+0000","last_peered":"2024-09-14T04:30:14.379938+0000","last_clean":"2024-09-14T04:30:14.379938+0000","last_became_active":"2024-09-14T04:29:17.239884+0000","last_became_peered":"2024-09-14T04:29:17.239884+0000","last_unstale":"2024-09-14T04:30:14.379938+0000","last_undegraded":"2024-09-14T04:30:14.379938+0000","last_fullsized":"2024-09-14T04:30:14.379938+0000","mapping_epoch":385,"log_start":"405'6702","ondisk_log_start":"405'6702","created":385,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90439680,"num_objects":1380,"num_object_clones":0,"num_object_copies":4140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7313,"num_write_kb":278176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'100","reported_seq":305,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.761595+0000","last_change":"2024-09-14T04:27:43.802100+0000","last_active":"2024-09-14T04:30:08.761595+0000","last_peered":"2024-09-14T04:30:08.761595+0000","last_clean":"2024-09-14T04:30:08.761595+0000","last_became_active":"2024-09-14T04:27:12.687977+0000","last_became_peered":"2024-09-14T04:27:12.687977+0000","last_unstale":"2024-09-14T04:30:08.761595+0000","last_undegraded":"2024-09-14T04:30:08.761595+0000","last_fullsized":"2024-09-14T04:30:08.761595+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:43.801906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:43.801906+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:33:01.220893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":711,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.774152+0000","last_change":"2024-09-14T04:26:15.748334+0000","last_active":"2024-09-14T04:30:08.774152+0000","last_peered":"2024-09-14T04:30:08.774152+0000","last_clean":"2024-09-14T04:30:08.774152+0000","last_became_active":"2024-09-14T04:26:15.726174+0000","last_became_peered":"2024-09-14T04:26:15.726174+0000","last_unstale":"2024-09-14T04:30:08.774152+0000","last_undegraded":"2024-09-14T04:30:08.774152+0000","last_fullsized":"2024-09-14T04:30:08.774152+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:25:47.468961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":439,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388547+0000","last_change":"2024-09-14T04:24:47.618083+0000","last_active":"2024-09-14T04:29:51.388547+0000","last_peered":"2024-09-14T04:29:51.388547+0000","last_clean":"2024-09-14T04:29:51.388547+0000","last_became_active":"2024-09-14T04:24:47.617759+0000","last_became_peered":"2024-09-14T04:24:47.617759+0000","last_unstale":"2024-09-14T04:29:51.388547+0000","last_undegraded":"2024-09-14T04:29:51.388547+0000","last_fullsized":"2024-09-14T04:29:51.388547+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:08:35.614869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":241,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.766443+0000","last_change":"2024-09-14T04:26:15.726983+0000","last_active":"2024-09-14T04:30:08.766443+0000","last_peered":"2024-09-14T04:30:08.766443+0000","last_clean":"2024-09-14T04:30:08.766443+0000","last_became_active":"2024-09-14T04:26:15.726003+0000","last_became_peered":"2024-09-14T04:26:15.726003+0000","last_unstale":"2024-09-14T04:30:08.766443+0000","last_undegraded":"2024-09-14T04:30:08.766443+0000","last_fullsized":"2024-09-14T04:30:08.766443+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:18:15.328679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.b","version":"422'12117","reported_seq":12266,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.796792+0000","last_change":"2024-09-14T04:29:18.693970+0000","last_active":"2024-09-14T04:30:14.796792+0000","last_peered":"2024-09-14T04:30:14.796792+0000","last_clean":"2024-09-14T04:30:14.796792+0000","last_became_active":"2024-09-14T04:29:17.238165+0000","last_became_peered":"2024-09-14T04:29:17.238165+0000","last_unstale":"2024-09-14T04:30:14.796792+0000","last_undegraded":"2024-09-14T04:30:14.796792+0000","last_fullsized":"2024-09-14T04:30:14.796792+0000","mapping_epoch":385,"log_start":"401'5243","ondisk_log_start":"401'5243","created":385,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'9620","last_scrub_stamp":"2024-09-14T04:29:18.692579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:18.692579+0000","objects_scrubbed":3778,"log_size":6874,"log_dups_size":0,"ondisk_log_size":6874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:03:21.772514+0000","scrub_duration":994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82640896,"num_objects":1261,"num_object_clones":0,"num_object_copies":3783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"76'100","reported_seq":105,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.777689+0000","last_change":"2024-09-14T04:27:44.420656+0000","last_active":"2024-09-14T04:30:08.777689+0000","last_peered":"2024-09-14T04:30:08.777689+0000","last_clean":"2024-09-14T04:30:08.777689+0000","last_became_active":"2024-09-14T04:27:12.689883+0000","last_became_peered":"2024-09-14T04:27:12.689883+0000","last_unstale":"2024-09-14T04:30:08.777689+0000","last_undegraded":"2024-09-14T04:30:08.777689+0000","last_fullsized":"2024-09-14T04:30:08.777689+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:44.420585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:44.420585+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:51:18.088643+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":483,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387023+0000","last_change":"2024-09-14T04:26:15.760792+0000","last_active":"2024-09-14T04:29:51.387023+0000","last_peered":"2024-09-14T04:29:51.387023+0000","last_clean":"2024-09-14T04:29:51.387023+0000","last_became_active":"2024-09-14T04:26:15.745969+0000","last_became_peered":"2024-09-14T04:26:15.745969+0000","last_unstale":"2024-09-14T04:29:51.387023+0000","last_undegraded":"2024-09-14T04:29:51.387023+0000","last_fullsized":"2024-09-14T04:29:51.387023+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:41:56.094887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":439,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388439+0000","last_change":"2024-09-14T04:25:42.332231+0000","last_active":"2024-09-14T04:29:51.388439+0000","last_peered":"2024-09-14T04:29:51.388439+0000","last_clean":"2024-09-14T04:29:51.388439+0000","last_became_active":"2024-09-14T04:25:42.331514+0000","last_became_peered":"2024-09-14T04:25:42.331514+0000","last_unstale":"2024-09-14T04:29:51.388439+0000","last_undegraded":"2024-09-14T04:29:51.388439+0000","last_fullsized":"2024-09-14T04:29:51.388439+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:33:37.729778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":107,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386525+0000","last_change":"2024-09-14T04:22:08.465532+0000","last_active":"2024-09-14T04:29:51.386525+0000","last_peered":"2024-09-14T04:29:51.386525+0000","last_clean":"2024-09-14T04:29:51.386525+0000","last_became_active":"2024-09-14T04:22:08.464571+0000","last_became_peered":"2024-09-14T04:22:08.464571+0000","last_unstale":"2024-09-14T04:29:51.386525+0000","last_undegraded":"2024-09-14T04:29:51.386525+0000","last_fullsized":"2024-09-14T04:29:51.386525+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:25:23.109822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.19","version":"422'9409","reported_seq":9564,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.375782+0000","last_change":"2024-09-14T04:28:06.394086+0000","last_active":"2024-09-14T04:30:14.375782+0000","last_peered":"2024-09-14T04:30:14.375782+0000","last_clean":"2024-09-14T04:30:14.375782+0000","last_became_active":"2024-09-14T04:28:02.440766+0000","last_became_peered":"2024-09-14T04:28:02.440766+0000","last_unstale":"2024-09-14T04:30:14.375782+0000","last_undegraded":"2024-09-14T04:30:14.375782+0000","last_fullsized":"2024-09-14T04:30:14.375782+0000","mapping_epoch":395,"log_start":"405'5402","ondisk_log_start":"405'5402","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"397'4347","last_scrub_stamp":"2024-09-14T04:28:06.394033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:06.394033+0000","objects_scrubbed":2196,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:20:41.162067+0000","scrub_duration":0.23278294199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89522176,"num_objects":1366,"num_object_clones":0,"num_object_copies":4098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5062,"num_write_kb":135424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":186,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388403+0000","last_change":"2024-09-14T04:25:53.301718+0000","last_active":"2024-09-14T04:29:51.388403+0000","last_peered":"2024-09-14T04:29:51.388403+0000","last_clean":"2024-09-14T04:29:51.388403+0000","last_became_active":"2024-09-14T04:25:53.301421+0000","last_became_peered":"2024-09-14T04:25:53.301421+0000","last_unstale":"2024-09-14T04:29:51.388403+0000","last_undegraded":"2024-09-14T04:29:51.388403+0000","last_fullsized":"2024-09-14T04:29:51.388403+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:45:47.546722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":143,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388506+0000","last_change":"2024-09-14T04:25:42.375734+0000","last_active":"2024-09-14T04:29:51.388506+0000","last_peered":"2024-09-14T04:29:51.388506+0000","last_clean":"2024-09-14T04:29:51.388506+0000","last_became_active":"2024-09-14T04:25:42.259500+0000","last_became_peered":"2024-09-14T04:25:42.259500+0000","last_unstale":"2024-09-14T04:29:51.388506+0000","last_undegraded":"2024-09-14T04:29:51.388506+0000","last_fullsized":"2024-09-14T04:29:51.388506+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:27:37.236272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"422'9355","reported_seq":9486,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.762515+0000","last_change":"2024-09-14T04:28:05.927770+0000","last_active":"2024-09-14T04:30:14.762515+0000","last_peered":"2024-09-14T04:30:14.762515+0000","last_clean":"2024-09-14T04:30:14.762515+0000","last_became_active":"2024-09-14T04:28:02.447084+0000","last_became_peered":"2024-09-14T04:28:02.447084+0000","last_unstale":"2024-09-14T04:30:14.762515+0000","last_undegraded":"2024-09-14T04:30:14.762515+0000","last_fullsized":"2024-09-14T04:30:14.762515+0000","mapping_epoch":395,"log_start":"401'4634","ondisk_log_start":"401'4634","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"397'4212","last_scrub_stamp":"2024-09-14T04:28:05.927656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:05.927656+0000","objects_scrubbed":2194,"log_size":4721,"log_dups_size":0,"ondisk_log_size":4721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:57:21.528392+0000","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88735744,"num_objects":1354,"num_object_clones":0,"num_object_copies":4062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5146,"num_write_kb":137792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":616,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388220+0000","last_change":"2024-09-14T04:26:15.728258+0000","last_active":"2024-09-14T04:29:51.388220+0000","last_peered":"2024-09-14T04:29:51.388220+0000","last_clean":"2024-09-14T04:29:51.388220+0000","last_became_active":"2024-09-14T04:26:15.726521+0000","last_became_peered":"2024-09-14T04:26:15.726521+0000","last_unstale":"2024-09-14T04:29:51.388220+0000","last_undegraded":"2024-09-14T04:29:51.388220+0000","last_fullsized":"2024-09-14T04:29:51.388220+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:22:20.663544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":375,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.771145+0000","last_change":"2024-09-14T04:26:15.758946+0000","last_active":"2024-09-14T04:30:08.771145+0000","last_peered":"2024-09-14T04:30:08.771145+0000","last_clean":"2024-09-14T04:30:08.771145+0000","last_became_active":"2024-09-14T04:26:15.725634+0000","last_became_peered":"2024-09-14T04:26:15.725634+0000","last_unstale":"2024-09-14T04:30:08.771145+0000","last_undegraded":"2024-09-14T04:30:08.771145+0000","last_fullsized":"2024-09-14T04:30:08.771145+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:21:59.189929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"422'12111","reported_seq":12255,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.735490+0000","last_change":"2024-09-14T04:29:24.683097+0000","last_active":"2024-09-14T04:30:14.735490+0000","last_peered":"2024-09-14T04:30:14.735490+0000","last_clean":"2024-09-14T04:30:14.735490+0000","last_became_active":"2024-09-14T04:29:17.275023+0000","last_became_peered":"2024-09-14T04:29:17.275023+0000","last_unstale":"2024-09-14T04:30:14.735490+0000","last_undegraded":"2024-09-14T04:30:14.735490+0000","last_fullsized":"2024-09-14T04:30:14.735490+0000","mapping_epoch":416,"log_start":"401'5332","ondisk_log_start":"401'5332","created":416,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"418'9902","last_scrub_stamp":"2024-09-14T04:29:24.682679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:24.682679+0000","objects_scrubbed":3545,"log_size":6779,"log_dups_size":0,"ondisk_log_size":6779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:17:53.461744+0000","scrub_duration":941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":3957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2226,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":137,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387278+0000","last_change":"2024-09-14T04:25:42.256482+0000","last_active":"2024-09-14T04:29:51.387278+0000","last_peered":"2024-09-14T04:29:51.387278+0000","last_clean":"2024-09-14T04:29:51.387278+0000","last_became_active":"2024-09-14T04:25:42.256182+0000","last_became_peered":"2024-09-14T04:25:42.256182+0000","last_unstale":"2024-09-14T04:29:51.387278+0000","last_undegraded":"2024-09-14T04:29:51.387278+0000","last_fullsized":"2024-09-14T04:29:51.387278+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:04:58.123110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":655,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.766560+0000","last_change":"2024-09-14T04:25:03.864958+0000","last_active":"2024-09-14T04:30:08.766560+0000","last_peered":"2024-09-14T04:30:08.766560+0000","last_clean":"2024-09-14T04:30:08.766560+0000","last_became_active":"2024-09-14T04:25:03.864529+0000","last_became_peered":"2024-09-14T04:25:03.864529+0000","last_unstale":"2024-09-14T04:30:08.766560+0000","last_undegraded":"2024-09-14T04:30:08.766560+0000","last_fullsized":"2024-09-14T04:30:08.766560+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:12:25.237701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1a","version":"422'12182","reported_seq":12235,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.395220+0000","last_change":"2024-09-14T04:29:17.286968+0000","last_active":"2024-09-14T04:30:14.395220+0000","last_peered":"2024-09-14T04:30:14.395220+0000","last_clean":"2024-09-14T04:30:14.395220+0000","last_became_active":"2024-09-14T04:29:17.274801+0000","last_became_peered":"2024-09-14T04:29:17.274801+0000","last_unstale":"2024-09-14T04:30:14.395220+0000","last_undegraded":"2024-09-14T04:30:14.395220+0000","last_fullsized":"2024-09-14T04:30:14.395220+0000","mapping_epoch":416,"log_start":"406'6850","ondisk_log_start":"406'6850","created":416,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":5332,"log_dups_size":0,"ondisk_log_size":5332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87883776,"num_objects":1341,"num_object_clones":0,"num_object_copies":4023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7352,"num_write_kb":278176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":667,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387339+0000","last_change":"2024-09-14T04:26:15.736843+0000","last_active":"2024-09-14T04:29:51.387339+0000","last_peered":"2024-09-14T04:29:51.387339+0000","last_clean":"2024-09-14T04:29:51.387339+0000","last_became_active":"2024-09-14T04:26:15.734655+0000","last_became_peered":"2024-09-14T04:26:15.734655+0000","last_unstale":"2024-09-14T04:29:51.387339+0000","last_undegraded":"2024-09-14T04:29:51.387339+0000","last_fullsized":"2024-09-14T04:29:51.387339+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:47:05.810191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":71,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.759699+0000","last_change":"2024-09-14T04:27:55.394565+0000","last_active":"2024-09-14T04:30:08.759699+0000","last_peered":"2024-09-14T04:30:08.759699+0000","last_clean":"2024-09-14T04:30:08.759699+0000","last_became_active":"2024-09-14T04:27:55.394243+0000","last_became_peered":"2024-09-14T04:27:55.394243+0000","last_unstale":"2024-09-14T04:30:08.759699+0000","last_undegraded":"2024-09-14T04:30:08.759699+0000","last_fullsized":"2024-09-14T04:30:08.759699+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:36:52.365624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"422'9536","reported_seq":9667,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.888481+0000","last_change":"2024-09-14T04:28:06.796053+0000","last_active":"2024-09-14T04:30:14.888481+0000","last_peered":"2024-09-14T04:30:14.888481+0000","last_clean":"2024-09-14T04:30:14.888481+0000","last_became_active":"2024-09-14T04:28:02.446713+0000","last_became_peered":"2024-09-14T04:28:02.446713+0000","last_unstale":"2024-09-14T04:30:14.888481+0000","last_undegraded":"2024-09-14T04:30:14.888481+0000","last_fullsized":"2024-09-14T04:30:14.888481+0000","mapping_epoch":395,"log_start":"406'5587","ondisk_log_start":"406'5587","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"397'4350","last_scrub_stamp":"2024-09-14T04:28:06.795987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:06.795987+0000","objects_scrubbed":2241,"log_size":3949,"log_dups_size":0,"ondisk_log_size":3949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:47.373654+0000","scrub_duration":265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90112000,"num_objects":1375,"num_object_clones":0,"num_object_copies":4125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5186,"num_write_kb":138240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":446,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.388367+0000","last_change":"2024-09-14T04:26:15.731440+0000","last_active":"2024-09-14T04:29:51.388367+0000","last_peered":"2024-09-14T04:29:51.388367+0000","last_clean":"2024-09-14T04:29:51.388367+0000","last_became_active":"2024-09-14T04:26:15.726850+0000","last_became_peered":"2024-09-14T04:26:15.726850+0000","last_unstale":"2024-09-14T04:29:51.388367+0000","last_undegraded":"2024-09-14T04:29:51.388367+0000","last_fullsized":"2024-09-14T04:29:51.388367+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:00:48.044324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":501,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.766523+0000","last_change":"2024-09-14T04:26:15.727272+0000","last_active":"2024-09-14T04:30:08.766523+0000","last_peered":"2024-09-14T04:30:08.766523+0000","last_clean":"2024-09-14T04:30:08.766523+0000","last_became_active":"2024-09-14T04:26:15.726200+0000","last_became_peered":"2024-09-14T04:26:15.726200+0000","last_unstale":"2024-09-14T04:30:08.766523+0000","last_undegraded":"2024-09-14T04:30:08.766523+0000","last_fullsized":"2024-09-14T04:30:08.766523+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:38:30.447563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"422'9123","reported_seq":11454,"reported_epoch":422,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-14T04:30:14.948252+0000","last_change":"2024-09-14T04:29:50.175030+0000","last_active":"2024-09-14T04:30:14.948252+0000","last_peered":"2024-09-14T04:30:14.948252+0000","last_clean":"2024-09-14T04:29:01.810326+0000","last_became_active":"2024-09-14T04:29:05.545170+0000","last_became_peered":"2024-09-14T04:29:05.545170+0000","last_unstale":"2024-09-14T04:30:14.948252+0000","last_undegraded":"2024-09-14T04:29:05.449552+0000","last_fullsized":"2024-09-14T04:29:05.431242+0000","mapping_epoch":413,"log_start":"405'5133","ondisk_log_start":"405'5133","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":3990,"log_dups_size":0,"ondisk_log_size":3990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87425024,"num_objects":1334,"num_object_clones":0,"num_object_copies":4002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1826,"num_objects_misplaced":1215,"num_objects_unfound":0,"num_objects_dirty":1334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7126,"num_write_kb":270720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":73924608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,6],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1215},{"shards":"3,6","objects":-1707}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":497,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.770939+0000","last_change":"2024-09-14T04:26:15.745872+0000","last_active":"2024-09-14T04:30:08.770939+0000","last_peered":"2024-09-14T04:30:08.770939+0000","last_clean":"2024-09-14T04:30:08.770939+0000","last_became_active":"2024-09-14T04:26:15.745533+0000","last_became_peered":"2024-09-14T04:26:15.745533+0000","last_unstale":"2024-09-14T04:30:08.770939+0000","last_undegraded":"2024-09-14T04:30:08.770939+0000","last_fullsized":"2024-09-14T04:30:08.770939+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:28:40.179766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":516,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.766787+0000","last_change":"2024-09-14T04:24:47.612687+0000","last_active":"2024-09-14T04:30:08.766787+0000","last_peered":"2024-09-14T04:30:08.766787+0000","last_clean":"2024-09-14T04:30:08.766787+0000","last_became_active":"2024-09-14T04:24:47.612392+0000","last_became_peered":"2024-09-14T04:24:47.612392+0000","last_unstale":"2024-09-14T04:30:08.766787+0000","last_undegraded":"2024-09-14T04:30:08.766787+0000","last_fullsized":"2024-09-14T04:30:08.766787+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:52:51.954808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.17","version":"422'9571","reported_seq":9647,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.328913+0000","last_change":"2024-09-14T04:28:55.173973+0000","last_active":"2024-09-14T04:30:14.328913+0000","last_peered":"2024-09-14T04:30:14.328913+0000","last_clean":"2024-09-14T04:30:14.328913+0000","last_became_active":"2024-09-14T04:28:02.446942+0000","last_became_peered":"2024-09-14T04:28:02.446942+0000","last_unstale":"2024-09-14T04:30:14.328913+0000","last_undegraded":"2024-09-14T04:30:14.328913+0000","last_fullsized":"2024-09-14T04:30:14.328913+0000","mapping_epoch":395,"log_start":"405'5546","ondisk_log_start":"405'5546","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:27:14.687823+0000","objects_scrubbed":0,"log_size":4025,"log_dups_size":0,"ondisk_log_size":4025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91619328,"num_objects":1398,"num_object_clones":0,"num_object_copies":4194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7492,"num_write_kb":284480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":486,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.771045+0000","last_change":"2024-09-14T04:24:47.613822+0000","last_active":"2024-09-14T04:30:08.771045+0000","last_peered":"2024-09-14T04:30:08.771045+0000","last_clean":"2024-09-14T04:30:08.771045+0000","last_became_active":"2024-09-14T04:24:47.613315+0000","last_became_peered":"2024-09-14T04:24:47.613315+0000","last_unstale":"2024-09-14T04:30:08.771045+0000","last_undegraded":"2024-09-14T04:30:08.771045+0000","last_fullsized":"2024-09-14T04:30:08.771045+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:22.682884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":285,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.758535+0000","last_change":"2024-09-14T04:26:15.727713+0000","last_active":"2024-09-14T04:30:08.758535+0000","last_peered":"2024-09-14T04:30:08.758535+0000","last_clean":"2024-09-14T04:30:08.758535+0000","last_became_active":"2024-09-14T04:26:15.727020+0000","last_became_peered":"2024-09-14T04:26:15.727020+0000","last_unstale":"2024-09-14T04:30:08.758535+0000","last_undegraded":"2024-09-14T04:30:08.758535+0000","last_fullsized":"2024-09-14T04:30:08.758535+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:35:46.915353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.16","version":"422'9284","reported_seq":9413,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.809276+0000","last_change":"2024-09-14T04:28:04.126036+0000","last_active":"2024-09-14T04:30:14.809276+0000","last_peered":"2024-09-14T04:30:14.809276+0000","last_clean":"2024-09-14T04:30:14.809276+0000","last_became_active":"2024-09-14T04:28:02.403141+0000","last_became_peered":"2024-09-14T04:28:02.403141+0000","last_unstale":"2024-09-14T04:30:14.809276+0000","last_undegraded":"2024-09-14T04:30:14.809276+0000","last_fullsized":"2024-09-14T04:30:14.809276+0000","mapping_epoch":395,"log_start":"401'4574","ondisk_log_start":"401'4574","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"396'4255","last_scrub_stamp":"2024-09-14T04:28:04.125956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:04.125956+0000","objects_scrubbed":2150,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:53:19.294212+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82903040,"num_objects":1265,"num_object_clones":0,"num_object_copies":3795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5045,"num_write_kb":133120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":142,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.759864+0000","last_change":"2024-09-14T04:26:15.721332+0000","last_active":"2024-09-14T04:30:08.759864+0000","last_peered":"2024-09-14T04:30:08.759864+0000","last_clean":"2024-09-14T04:30:08.759864+0000","last_became_active":"2024-09-14T04:26:15.721017+0000","last_became_peered":"2024-09-14T04:26:15.721017+0000","last_unstale":"2024-09-14T04:30:08.759864+0000","last_undegraded":"2024-09-14T04:30:08.759864+0000","last_fullsized":"2024-09-14T04:30:08.759864+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:44:50.434373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":161,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.770561+0000","last_change":"2024-09-14T04:26:15.731906+0000","last_active":"2024-09-14T04:30:08.770561+0000","last_peered":"2024-09-14T04:30:08.770561+0000","last_clean":"2024-09-14T04:30:08.770561+0000","last_became_active":"2024-09-14T04:26:15.730836+0000","last_became_peered":"2024-09-14T04:26:15.730836+0000","last_unstale":"2024-09-14T04:30:08.770561+0000","last_undegraded":"2024-09-14T04:30:08.770561+0000","last_fullsized":"2024-09-14T04:30:08.770561+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:03:41.330132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.11","version":"422'9207","reported_seq":13775,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.072085+0000","last_change":"2024-09-14T04:28:56.396050+0000","last_active":"2024-09-14T04:30:14.072085+0000","last_peered":"2024-09-14T04:30:14.072085+0000","last_clean":"2024-09-14T04:30:14.072085+0000","last_became_active":"2024-09-14T04:28:56.371145+0000","last_became_peered":"2024-09-14T04:28:56.371145+0000","last_unstale":"2024-09-14T04:30:14.072085+0000","last_undegraded":"2024-09-14T04:30:14.072085+0000","last_fullsized":"2024-09-14T04:30:14.072085+0000","mapping_epoch":408,"log_start":"401'5044","ondisk_log_start":"401'5044","created":395,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":5,"last_scrub":"396'4149","last_scrub_stamp":"2024-09-14T04:28:04.000577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:04.000577+0000","objects_scrubbed":2134,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:14:10.893795+0000","scrub_duration":458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5071,"num_write_kb":135296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2225,"num_bytes_recovered":145817600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":584,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.770742+0000","last_change":"2024-09-14T04:25:42.236692+0000","last_active":"2024-09-14T04:30:08.770742+0000","last_peered":"2024-09-14T04:30:08.770742+0000","last_clean":"2024-09-14T04:30:08.770742+0000","last_became_active":"2024-09-14T04:25:42.236449+0000","last_became_peered":"2024-09-14T04:25:42.236449+0000","last_unstale":"2024-09-14T04:30:08.770742+0000","last_undegraded":"2024-09-14T04:30:08.770742+0000","last_fullsized":"2024-09-14T04:30:08.770742+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:19:34.323210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":125,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.386853+0000","last_change":"2024-09-14T04:24:47.706532+0000","last_active":"2024-09-14T04:29:51.386853+0000","last_peered":"2024-09-14T04:29:51.386853+0000","last_clean":"2024-09-14T04:29:51.386853+0000","last_became_active":"2024-09-14T04:24:47.628189+0000","last_became_peered":"2024-09-14T04:24:47.628189+0000","last_unstale":"2024-09-14T04:29:51.386853+0000","last_undegraded":"2024-09-14T04:29:51.386853+0000","last_fullsized":"2024-09-14T04:29:51.386853+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:29:46.206803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"422'9125","reported_seq":13498,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.810560+0000","last_change":"2024-09-14T04:29:01.816508+0000","last_active":"2024-09-14T04:30:14.810560+0000","last_peered":"2024-09-14T04:30:14.810560+0000","last_clean":"2024-09-14T04:30:14.810560+0000","last_became_active":"2024-09-14T04:28:29.712822+0000","last_became_peered":"2024-09-14T04:28:29.712822+0000","last_unstale":"2024-09-14T04:30:14.810560+0000","last_undegraded":"2024-09-14T04:30:14.810560+0000","last_fullsized":"2024-09-14T04:30:14.810560+0000","mapping_epoch":402,"log_start":"401'4377","ondisk_log_start":"401'4377","created":395,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":5,"last_scrub":"410'6322","last_scrub_stamp":"2024-09-14T04:29:01.816417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:29:01.816417+0000","objects_scrubbed":4121,"log_size":4748,"log_dups_size":0,"ondisk_log_size":4748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:11:15.126666+0000","scrub_duration":1132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85524480,"num_objects":1305,"num_object_clones":0,"num_object_copies":3915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2816,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":570,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.770029+0000","last_change":"2024-09-14T04:26:15.722033+0000","last_active":"2024-09-14T04:30:08.770029+0000","last_peered":"2024-09-14T04:30:08.770029+0000","last_clean":"2024-09-14T04:30:08.770029+0000","last_became_active":"2024-09-14T04:26:15.721545+0000","last_became_peered":"2024-09-14T04:26:15.721545+0000","last_unstale":"2024-09-14T04:30:08.770029+0000","last_undegraded":"2024-09-14T04:30:08.770029+0000","last_fullsized":"2024-09-14T04:30:08.770029+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:48:21.769211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":74,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.761447+0000","last_change":"2024-09-14T04:27:55.405108+0000","last_active":"2024-09-14T04:30:08.761447+0000","last_peered":"2024-09-14T04:30:08.761447+0000","last_clean":"2024-09-14T04:30:08.761447+0000","last_became_active":"2024-09-14T04:27:55.394475+0000","last_became_peered":"2024-09-14T04:27:55.394475+0000","last_unstale":"2024-09-14T04:30:08.761447+0000","last_undegraded":"2024-09-14T04:30:08.761447+0000","last_fullsized":"2024-09-14T04:30:08.761447+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:44:06.833702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.13","version":"422'9304","reported_seq":9432,"reported_epoch":422,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-14T04:30:14.800488+0000","last_change":"2024-09-14T04:29:03.506519+0000","last_active":"2024-09-14T04:30:14.800488+0000","last_peered":"2024-09-14T04:30:14.800488+0000","last_clean":"2024-09-14T04:29:01.393795+0000","last_became_active":"2024-09-14T04:29:03.501157+0000","last_became_peered":"2024-09-14T04:29:03.501157+0000","last_unstale":"2024-09-14T04:30:14.800488+0000","last_undegraded":"2024-09-14T04:30:14.800488+0000","last_fullsized":"2024-09-14T04:30:14.800488+0000","mapping_epoch":411,"log_start":"405'5376","ondisk_log_start":"405'5376","created":395,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'4303","last_scrub_stamp":"2024-09-14T04:28:07.869714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:07.869714+0000","objects_scrubbed":2150,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86966272,"num_objects":1327,"num_object_clones":0,"num_object_copies":3981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1327,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5009,"num_write_kb":133952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[3,4,5],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":1327}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":494,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.766332+0000","last_change":"2024-09-14T04:26:15.727588+0000","last_active":"2024-09-14T04:30:08.766332+0000","last_peered":"2024-09-14T04:30:08.766332+0000","last_clean":"2024-09-14T04:30:08.766332+0000","last_became_active":"2024-09-14T04:26:15.727153+0000","last_became_peered":"2024-09-14T04:26:15.727153+0000","last_unstale":"2024-09-14T04:30:08.766332+0000","last_undegraded":"2024-09-14T04:30:08.766332+0000","last_fullsized":"2024-09-14T04:30:08.766332+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:15:27.230932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":503,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:08.771033+0000","last_change":"2024-09-14T04:25:42.235102+0000","last_active":"2024-09-14T04:30:08.771033+0000","last_peered":"2024-09-14T04:30:08.771033+0000","last_clean":"2024-09-14T04:30:08.771033+0000","last_became_active":"2024-09-14T04:25:42.234776+0000","last_became_peered":"2024-09-14T04:25:42.234776+0000","last_unstale":"2024-09-14T04:30:08.771033+0000","last_undegraded":"2024-09-14T04:30:08.771033+0000","last_fullsized":"2024-09-14T04:30:08.771033+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:59:52.026651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.12","version":"422'9370","reported_seq":19034,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-14T04:30:14.782802+0000","last_change":"2024-09-14T04:29:51.408294+0000","last_active":"2024-09-14T04:30:14.782802+0000","last_peered":"2024-09-14T04:30:14.782802+0000","last_clean":"2024-09-14T04:30:14.782802+0000","last_became_active":"2024-09-14T04:29:51.390386+0000","last_became_peered":"2024-09-14T04:29:51.390386+0000","last_unstale":"2024-09-14T04:30:14.782802+0000","last_undegraded":"2024-09-14T04:30:14.782802+0000","last_fullsized":"2024-09-14T04:30:14.782802+0000","mapping_epoch":420,"log_start":"401'4596","ondisk_log_start":"401'4596","created":395,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":5,"last_scrub":"399'4281","last_scrub_stamp":"2024-09-14T04:28:07.784588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T04:27:14.687823+0000","last_clean_scrub_stamp":"2024-09-14T04:28:07.784588+0000","objects_scrubbed":2244,"log_size":4774,"log_dups_size":0,"ondisk_log_size":4774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:30:31.749953+0000","scrub_duration":0.338810958,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83296256,"num_objects":1271,"num_object_clones":0,"num_object_copies":3813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5093,"num_write_kb":131840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3157,"num_bytes_recovered":206897152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":235,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387568+0000","last_change":"2024-09-14T04:25:53.300338+0000","last_active":"2024-09-14T04:29:51.387568+0000","last_peered":"2024-09-14T04:29:51.387568+0000","last_clean":"2024-09-14T04:29:51.387568+0000","last_became_active":"2024-09-14T04:25:53.299919+0000","last_became_peered":"2024-09-14T04:25:53.299919+0000","last_unstale":"2024-09-14T04:29:51.387568+0000","last_undegraded":"2024-09-14T04:29:51.387568+0000","last_fullsized":"2024-09-14T04:29:51.387568+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:55:17.341402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"62'4","reported_seq":696,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-14T04:29:51.387257+0000","last_change":"2024-09-14T04:26:15.738521+0000","last_active":"2024-09-14T04:29:51.387257+0000","last_peered":"2024-09-14T04:29:51.387257+0000","last_clean":"2024-09-14T04:29:51.387257+0000","last_became_active":"2024-09-14T04:26:15.736609+0000","last_became_peered":"2024-09-14T04:26:15.736609+0000","last_unstale":"2024-09-14T04:29:51.387257+0000","last_undegraded":"2024-09-14T04:29:51.387257+0000","last_fullsized":"2024-09-14T04:29:51.387257+0000","mapping_epoch":370,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:54:47.371271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":6886,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":21458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":95,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":14,"num_pg":36,"stat_sum":{"num_bytes":2778955800,"num_objects":42405,"num_object_clones":0,"num_object_copies":127215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1826,"num_objects_misplaced":2542,"num_objects_unfound":0,"num_objects_dirty":42405,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145033,"num_write_kb":3628673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6512,"num_bytes_recovered":426704896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9120460800,"data_stored":9120448584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201367,"ondisk_log_size":201367,"up":108,"acting":107,"num_store_stats":5},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2641920,"data_stored":2624160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":0,"seq":206158430798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388627027,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3153572,"kb_used_data":1648604,"kb_used_omap":1,"kb_used_meta":1504894,"kb_avail":90591580,"statfs":{"total":95995035648,"available":92765777920,"internally_reserved":0,"allocated":1688170496,"data_stored":1686143071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1541011800},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823255,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3511124,"kb_used_data":2000652,"kb_used_omap":1,"kb_used_meta":1510462,"kb_avail":90234028,"statfs":{"total":95995035648,"available":92399644672,"internally_reserved":0,"allocated":2048667648,"data_stored":2046651990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1546713432},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917337,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2857180,"kb_used_data":1913940,"kb_used_omap":6,"kb_used_meta":942905,"kb_avail":90887972,"statfs":{"total":95995035648,"available":93069283328,"internally_reserved":0,"allocated":1959874560,"data_stored":1957854690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":965535743},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552636,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3034136,"kb_used_data":1813776,"kb_used_omap":4,"kb_used_meta":1220347,"kb_avail":90711016,"statfs":{"total":95995035648,"available":92888080384,"internally_reserved":0,"allocated":1857306624,"data_stored":1855287016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":1249635773},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435677,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2871316,"kb_used_data":1641868,"kb_used_omap":6,"kb_used_meta":1229433,"kb_avail":90873836,"statfs":{"total":95995035648,"available":93054808064,"internally_reserved":0,"allocated":1681272832,"data_stored":1679261005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":1258939853},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":416611828147,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":343597384136,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1631518720,"data_stored":1631518720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1848049664,"data_stored":1848049664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1948848128,"data_stored":1948844056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2034044928,"data_stored":2034040856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1657999360,"data_stored":1657995288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:30:20.411 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-14T04:30:20.411 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-14T04:30:20.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:20.540 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:20.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:20.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:20 smithi114 ceph-mon[106969]: pgmap v1592: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.45k op/s; 1826/123972 objects degraded (1.473%); 2542/123972 objects misplaced (2.050%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:30:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:20 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1826/123972 objects degraded (1.473%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:20 smithi114 ceph-mon[103029]: pgmap v1592: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.45k op/s; 1826/123972 objects degraded (1.473%); 2542/123972 objects misplaced (2.050%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:30:20.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:20 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1826/123972 objects degraded (1.473%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:20.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:20 smithi160 ceph-mon[88407]: pgmap v1592: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 1.45k op/s; 1826/123972 objects degraded (1.473%); 2542/123972 objects misplaced (2.050%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:30:20.659 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:20 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1826/123972 objects degraded (1.473%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:20.659 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:30:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:30:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:30:20.679 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:20.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:20.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:20.781 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:20.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:20.930 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:20.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:20.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:21.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:21.110 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:21.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:21.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:21.251 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:21.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:21.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:21.392 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:21.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:21.494 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:21.531 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:21.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:21.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:21.565 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:30:21.565 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:30:21.566 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:30:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:30:21] "GET /metrics HTTP/1.1" 200 36571 "" "Prometheus/2.51.0" 2024-09-14T04:30:21.596 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:30:21.596 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:30:21.624 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:30:21.624 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:30:21.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:21.664 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:30:21.665 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:30:21.680 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:21.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:21.695 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:30:21.695 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:30:21.729 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:30:21.729 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:30:21.755 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:30:21.755 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:30:21.781 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:30:21.781 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:30:21.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:21.831 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:21.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:21.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:21.933 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:21.964 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:21.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:21.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:22.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:22.107 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:22.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:22.209 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:22.248 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:22.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:22.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:22.390 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:22.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:22.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:22.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:22.530 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:22.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:22.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:22.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:22 smithi114 ceph-mon[103029]: pgmap v1593: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.68k op/s; 1826/105930 objects degraded (1.724%); 2082/105930 objects misplaced (1.965%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T04:30:22.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:22 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2770117028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:30:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:22 smithi114 ceph-mon[106969]: pgmap v1593: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.68k op/s; 1826/105930 objects degraded (1.724%); 2082/105930 objects misplaced (1.965%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T04:30:22.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:22 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2770117028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:30:22.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:22 smithi160 ceph-mon[88407]: pgmap v1593: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.68k op/s; 1826/105930 objects degraded (1.724%); 2082/105930 objects misplaced (1.965%); 3.5 MiB/s, 56 objects/s recovering 2024-09-14T04:30:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:22 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2770117028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:30:22.663 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:22.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:22.764 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:22.798 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:22.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:22.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:22.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:22.932 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:30:22.932 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:30:22.932 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-14T04:30:22.933 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:30:22.936 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:22.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:23.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:23.070 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:23.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:23.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:23.211 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:23.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:23.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:23.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:23.346 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:23.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:23.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:23.483 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:23.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:23.584 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:23.614 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:23.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:23.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:23.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:23.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:23.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:23.716 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:23.751 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:23.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:23.853 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:23.892 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:23.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:23.994 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:24.033 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:24.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:24.117 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:24.122 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:24.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:24.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:24.174 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:24.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:24.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:24.316 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:24.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:24.417 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:24.453 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:24.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:24.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:24.554 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:24.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:24 smithi114 ceph-mon[106969]: pgmap v1594: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 1826/105930 objects degraded (1.724%); 2082/105930 objects misplaced (1.965%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:30:24.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:24 smithi114 ceph-mon[103029]: pgmap v1594: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 1826/105930 objects degraded (1.724%); 2082/105930 objects misplaced (1.965%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:30:24.604 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:24.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:24.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-14T04:30:24.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:24 smithi160 ceph-mon[88407]: pgmap v1594: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 2.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 1826/105930 objects degraded (1.724%); 2082/105930 objects misplaced (1.965%); 2.4 MiB/s, 37 objects/s recovering 2024-09-14T04:30:24.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:24.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:24.750 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:24.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:24.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:24.893 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:24.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:24.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:25.027 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:25.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:25.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:25.164 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:25.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:25.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:25.310 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:25.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:25.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:25.450 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:25.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:25.466 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:30:25.498 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:25 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1826/105930 objects degraded (1.724%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:25.499 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:25 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1826/105930 objects degraded (1.724%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:25.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:25.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:25.595 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:25.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:25.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:25 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1826/105930 objects degraded (1.724%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:25.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:25.728 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:25.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:25.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:25.861 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:25.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:25.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:26.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:26.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:26.022 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:26.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:26.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:26.162 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:26.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:26.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:26.298 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:26.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:26.352 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:26 smithi114 ceph-mon[106969]: pgmap v1595: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.80k op/s; 1826/83715 objects degraded (2.181%); 1599/83715 objects misplaced (1.910%); 3.7 MiB/s, 58 objects/s recovering 2024-09-14T04:30:26.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:26 smithi114 ceph-mon[103029]: pgmap v1595: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.80k op/s; 1826/83715 objects degraded (2.181%); 1599/83715 objects misplaced (1.910%); 3.7 MiB/s, 58 objects/s recovering 2024-09-14T04:30:26.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:26.440 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:26.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:26.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:26.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:26.581 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:26.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:26 smithi160 ceph-mon[88407]: pgmap v1595: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.80k op/s; 1826/83715 objects degraded (2.181%); 1599/83715 objects misplaced (1.910%); 3.7 MiB/s, 58 objects/s recovering 2024-09-14T04:30:26.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:26.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:26.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:26.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:26.844 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:26.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:26.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:26.985 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:26.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:27.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:27.108 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:27.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:27.210 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:27.244 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:27.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:27.347 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:27.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:27.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:27.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:27.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:27.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:27.618 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:27.655 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:27.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:27.757 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:27.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:27.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:27.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:27.926 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:27.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:27.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:28.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:28.064 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:28.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:28.166 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:28.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:28.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:28.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:28.333 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:28.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:28.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:28.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:28.468 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:28.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:28.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:28.613 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:28.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:28.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:28.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:28 smithi114 ceph-mon[103029]: pgmap v1596: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 1826/83715 objects degraded (2.181%); 1599/83715 objects misplaced (1.910%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:30:28.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:28 smithi114 ceph-mon[106969]: pgmap v1596: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 1826/83715 objects degraded (2.181%); 1599/83715 objects misplaced (1.910%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:30:28.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:28 smithi160 ceph-mon[88407]: pgmap v1596: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 1826/83715 objects degraded (2.181%); 1599/83715 objects misplaced (1.910%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:30:28.715 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:28.743 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:28.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:28.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:28.885 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:28.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:28.986 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:29.026 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:29.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:29.128 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:29.166 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:29.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:29.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:29.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:29.299 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:29.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:29.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:29.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:29.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:29.538 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:29.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:29.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:29.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:29.702 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:29.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:29.804 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:29.837 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:29.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:29.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:29.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:29.974 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:29.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:30.075 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:30.109 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:30.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:30.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:30.251 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:30.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:30.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:30.391 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:30.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:30.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:30.473 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:30:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:30:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:30:30.473 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:30 smithi160 ceph-mon[88407]: pgmap v1597: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 1826/80244 objects degraded (2.276%); 1599/80244 objects misplaced (1.993%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:30:30.474 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:30 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1826/80244 objects degraded (2.276%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:30.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:30.523 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:30.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:30 smithi114 ceph-mon[106969]: pgmap v1597: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 1826/80244 objects degraded (2.276%); 1599/80244 objects misplaced (1.993%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:30:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:30 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1826/80244 objects degraded (2.276%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:30.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:30 smithi114 ceph-mon[103029]: pgmap v1597: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 1826/80244 objects degraded (2.276%); 1599/80244 objects misplaced (1.993%); 2.5 MiB/s, 40 objects/s recovering 2024-09-14T04:30:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:30 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1826/80244 objects degraded (2.276%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:30.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:30.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:30.664 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:30.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:30.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:30.792 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:30.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:30.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:30.922 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:30.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:31.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:31.056 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:31.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:31.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:31.187 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:31.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:31.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:31.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:31.317 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:31.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:31.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:31.446 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:31.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:31.548 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:31.575 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:31.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:31.678 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:31.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:31.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:31.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:31.858 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:31.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:31.872 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:30:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:30:31] "GET /metrics HTTP/1.1" 200 36579 "" "Prometheus/2.51.0" 2024-09-14T04:30:31.959 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:31.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:31.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:31.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:32.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:32.126 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:32.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:32.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:32.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:32.276 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:32.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:32.377 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:32.414 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:32.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:32.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:32.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:32.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:32.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:32.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:32 smithi114 ceph-mon[106969]: pgmap v1598: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.71k op/s; 1826/62496 objects degraded (2.922%); 1127/62496 objects misplaced (1.803%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T04:30:32.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:32 smithi114 ceph-mon[103029]: pgmap v1598: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.71k op/s; 1826/62496 objects degraded (2.922%); 1127/62496 objects misplaced (1.803%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T04:30:32.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:32.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:32 smithi160 ceph-mon[88407]: pgmap v1598: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.71k op/s; 1826/62496 objects degraded (2.922%); 1127/62496 objects misplaced (1.803%); 3.8 MiB/s, 61 objects/s recovering 2024-09-14T04:30:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:32.693 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:32.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:32.794 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:32.834 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:32.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:32.936 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:32.971 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:32.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:32.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:33.073 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:33.104 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:33.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:33.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:33.237 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:33.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:33.339 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:33.360 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:33.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:33.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:33.500 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:33.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:33.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:33.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:33.635 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:33.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:30:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:30:33.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:33 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:33.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:33.760 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:33.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:33.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:33.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:33.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:33.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:33.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:30:33.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:30:33.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:33 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:33.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:33.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:30:33.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:30:33.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:33 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:33.894 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:33.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:33.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:34.041 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:34.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:34.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:34.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:34.171 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:34.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:34.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:34.308 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:34.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:34.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:34.449 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:34.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:34.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:34.587 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:34.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:34.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:34.725 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:34.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:34.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:34.866 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:34.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:34.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:34 smithi114 ceph-mon[103029]: pgmap v1599: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 1826/62496 objects degraded (2.922%); 1127/62496 objects misplaced (1.803%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:30:34.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:34 smithi114 ceph-mon[106969]: pgmap v1599: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 1826/62496 objects degraded (2.922%); 1127/62496 objects misplaced (1.803%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:30:34.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:34 smithi160 ceph-mon[88407]: pgmap v1599: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 1.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 1826/62496 objects degraded (2.922%); 1127/62496 objects misplaced (1.803%); 2.7 MiB/s, 42 objects/s recovering 2024-09-14T04:30:34.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:34.999 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:35.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:35.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:35.137 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:35.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:35.238 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:35.268 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:35.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:35.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:35.409 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:35.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:35.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:35.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:35.550 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:35.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:35.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:35.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:35.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:35.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:35.821 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:35.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:35.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:35.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:35 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1826/62496 objects degraded (2.922%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:35.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:35 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1826/62496 objects degraded (2.922%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:35.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:35 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1826/62496 objects degraded (2.922%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:35.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:35.955 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:35.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:36.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:36.097 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:36.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:36.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:36.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:36.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:36.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:36.348 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:36.382 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:36.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:36.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:36.517 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:36.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:36.618 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:36.638 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:36.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:36.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:36.785 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:36.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:36.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:36 smithi114 ceph-mon[106969]: pgmap v1600: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 901 MiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.74k op/s; 1826/43317 objects degraded (4.215%); 618/43317 objects misplaced (1.427%); 4.2 MiB/s, 67 objects/s recovering 2024-09-14T04:30:36.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:36 smithi114 ceph-mon[103029]: pgmap v1600: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 901 MiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.74k op/s; 1826/43317 objects degraded (4.215%); 618/43317 objects misplaced (1.427%); 4.2 MiB/s, 67 objects/s recovering 2024-09-14T04:30:36.887 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:36.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:36 smithi160 ceph-mon[88407]: pgmap v1600: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 901 MiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.74k op/s; 1826/43317 objects degraded (4.215%); 618/43317 objects misplaced (1.427%); 4.2 MiB/s, 67 objects/s recovering 2024-09-14T04:30:36.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:36.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:37.026 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:37.066 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:37.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:37.168 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:37.205 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:37.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:37.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:37.345 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:37.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:37.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:37.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:37.479 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:37.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:37.581 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:37.613 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:37.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:37.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:37.750 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:37.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:37.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:37.886 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:37.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:37.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:37.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:38.014 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:38.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:38.115 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:38.146 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:38.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:38.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:38.280 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:38.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:38.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:38.413 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:38.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:38.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:38.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:38.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:38.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:38.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:38.689 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:38.690 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:38.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:38.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:38.694 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:38.788 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:38 smithi114 ceph-mon[103029]: pgmap v1601: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 901 MiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 1826/43317 objects degraded (4.215%); 618/43317 objects misplaced (1.427%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:30:38.788 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:38.789 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:38 smithi114 ceph-mon[106969]: pgmap v1601: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 901 MiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 1826/43317 objects degraded (4.215%); 618/43317 objects misplaced (1.427%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:30:38.789 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:38.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:38.832 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:38.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:38 smithi160 ceph-mon[88407]: pgmap v1601: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 901 MiB data, 9.9 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 1.12k op/s; 1826/43317 objects degraded (4.215%); 618/43317 objects misplaced (1.427%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:30:38.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:38.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:38.965 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:38.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:39.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:39.106 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:39.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:39.208 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:39.246 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:39.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:39.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:39.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:39.378 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:39.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:39.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:39.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:39.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:39.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:39.661 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:39.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:39.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:39.800 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:39.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:39.901 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:39.939 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:39.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:40.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:40.076 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:40.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:40.177 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:40.212 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:40.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:40.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:40.363 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:40.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:40.454 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 1 pg_num to 21 2024-09-14T04:30:40.461 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:30:40.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:40.499 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:40.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:40.602 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:40.637 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:40.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:40.644 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:40 smithi114 ceph-mon[106969]: pgmap v1602: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 833 MiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 1826/40050 objects degraded (4.559%); 618/40050 objects misplaced (1.543%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:30:40.644 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:40 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1826/40050 objects degraded (4.559%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:40.644 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1015858420' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:30:40.644 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:40 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:30:40.644 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2099449235' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-14T04:30:40.645 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:40 smithi114 ceph-mon[103029]: pgmap v1602: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 833 MiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 1826/40050 objects degraded (4.559%); 618/40050 objects misplaced (1.543%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:30:40.645 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:40 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1826/40050 objects degraded (4.559%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:40.645 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1015858420' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:30:40.645 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:40 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:30:40.645 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2099449235' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-14T04:30:40.657 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:30:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:30:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:30:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:40 smithi160 ceph-mon[88407]: pgmap v1602: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 833 MiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.21k op/s; 1826/40050 objects degraded (4.559%); 618/40050 objects misplaced (1.543%); 2.9 MiB/s, 46 objects/s recovering 2024-09-14T04:30:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:40 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1826/40050 objects degraded (4.559%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1015858420' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:30:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:40 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:30:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2099449235' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-14T04:30:40.740 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:40.774 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:40.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:40.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:40.876 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:40.910 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:40.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:41.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:41.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:41.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:41.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:41.187 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:41.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:41.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:41.311 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:41.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:41.412 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:41.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:41.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:41.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:41.585 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:41.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:41.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:30:41] "GET /metrics HTTP/1.1" 200 36572 "" "Prometheus/2.51.0" 2024-09-14T04:30:41.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:30:41.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:30:41.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2099449235' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-14T04:30:41.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-mon[103029]: osdmap e423: 8 total, 8 up, 5 in 2024-09-14T04:30:41.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-14T04:30:41.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:30:41.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:30:41.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2099449235' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-14T04:30:41.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-mon[106969]: osdmap e423: 8 total, 8 up, 5 in 2024-09-14T04:30:41.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:41 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-14T04:30:41.686 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:41.714 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:41.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:41.717 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:41 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:30:41.717 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:41 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:30:41.717 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:41 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2099449235' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-14T04:30:41.718 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:41 smithi160 ceph-mon[88407]: osdmap e423: 8 total, 8 up, 5 in 2024-09-14T04:30:41.718 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:41 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-14T04:30:41.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:41.855 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:41.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:41.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:41.995 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:41.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:42.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:42.122 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:42.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:42.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:42.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:42.261 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:42.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:42.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:42.404 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:42.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:42.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:42.546 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:42.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:42.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:42.684 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:42.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:42.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:42.817 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:42.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:42 smithi114 ceph-mon[106969]: pgmap v1604: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 451 MiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.36k op/s; 1718/21693 objects degraded (7.920%); 224/21693 objects misplaced (1.033%); 3.0 MiB/s, 63 objects/s recovering 2024-09-14T04:30:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:42 smithi114 ceph-mon[106969]: 14.17 scrub starts 2024-09-14T04:30:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:42 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-14T04:30:42.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:42 smithi114 ceph-mon[106969]: osdmap e424: 8 total, 8 up, 5 in 2024-09-14T04:30:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:42 smithi114 ceph-mon[103029]: pgmap v1604: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 451 MiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.36k op/s; 1718/21693 objects degraded (7.920%); 224/21693 objects misplaced (1.033%); 3.0 MiB/s, 63 objects/s recovering 2024-09-14T04:30:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:42 smithi114 ceph-mon[103029]: 14.17 scrub starts 2024-09-14T04:30:42.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:42 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-14T04:30:42.872 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:42 smithi114 ceph-mon[103029]: osdmap e424: 8 total, 8 up, 5 in 2024-09-14T04:30:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:42 smithi160 ceph-mon[88407]: pgmap v1604: 119 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 451 MiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 1.36k op/s; 1718/21693 objects degraded (7.920%); 224/21693 objects misplaced (1.033%); 3.0 MiB/s, 63 objects/s recovering 2024-09-14T04:30:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:42 smithi160 ceph-mon[88407]: 14.17 scrub starts 2024-09-14T04:30:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:42 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-14T04:30:42.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:42 smithi160 ceph-mon[88407]: osdmap e424: 8 total, 8 up, 5 in 2024-09-14T04:30:42.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:42.950 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:42.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:43.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:43.083 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:43.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:43.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:43.222 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:43.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:43.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:43.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:43.356 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:43.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:43.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:43.493 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:43.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:43.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:43.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[103029]: 14.a scrub starts 2024-09-14T04:30:43.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[103029]: osdmap e425: 8 total, 8 up, 5 in 2024-09-14T04:30:43.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[103029]: 1.4 scrub starts 2024-09-14T04:30:43.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[103029]: 1.4 scrub ok 2024-09-14T04:30:43.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[103029]: 1.5 scrub starts 2024-09-14T04:30:43.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[103029]: 1.5 scrub ok 2024-09-14T04:30:43.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[106969]: 14.a scrub starts 2024-09-14T04:30:43.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[106969]: osdmap e425: 8 total, 8 up, 5 in 2024-09-14T04:30:43.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[106969]: 1.4 scrub starts 2024-09-14T04:30:43.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[106969]: 1.4 scrub ok 2024-09-14T04:30:43.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[106969]: 1.5 scrub starts 2024-09-14T04:30:43.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:43 smithi114 ceph-mon[106969]: 1.5 scrub ok 2024-09-14T04:30:43.636 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:43.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:43.738 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:43.773 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:43.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:43.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:43 smithi160 ceph-mon[88407]: 14.a scrub starts 2024-09-14T04:30:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:43 smithi160 ceph-mon[88407]: osdmap e425: 8 total, 8 up, 5 in 2024-09-14T04:30:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:43 smithi160 ceph-mon[88407]: 1.4 scrub starts 2024-09-14T04:30:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:43 smithi160 ceph-mon[88407]: 1.4 scrub ok 2024-09-14T04:30:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:43 smithi160 ceph-mon[88407]: 1.5 scrub starts 2024-09-14T04:30:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:43 smithi160 ceph-mon[88407]: 1.5 scrub ok 2024-09-14T04:30:43.908 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:43.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:43.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:44.010 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:44.032 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:44.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:44.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:44.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:44.170 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:44.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:44.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:44.300 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:44.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:44.402 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:44.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:44.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:44.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:44.587 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:44.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:44.688 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:44.725 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:44.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:44.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:44.848 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:44.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:44 smithi114 ceph-mon[106969]: pgmap v1607: 129 pgs: 10 unknown, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 451 MiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 1718/21693 objects degraded (7.920%); 224/21693 objects misplaced (1.033%); 1.8 MiB/s, 54 objects/s recovering 2024-09-14T04:30:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:44 smithi114 ceph-mon[106969]: 1.14 scrub starts 2024-09-14T04:30:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:44 smithi114 ceph-mon[106969]: 1.14 scrub ok 2024-09-14T04:30:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:44 smithi114 ceph-mon[106969]: 1.2 scrub starts 2024-09-14T04:30:44.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:44 smithi114 ceph-mon[106969]: 1.2 scrub ok 2024-09-14T04:30:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:44 smithi114 ceph-mon[103029]: pgmap v1607: 129 pgs: 10 unknown, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 451 MiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 1718/21693 objects degraded (7.920%); 224/21693 objects misplaced (1.033%); 1.8 MiB/s, 54 objects/s recovering 2024-09-14T04:30:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:44 smithi114 ceph-mon[103029]: 1.14 scrub starts 2024-09-14T04:30:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:44 smithi114 ceph-mon[103029]: 1.14 scrub ok 2024-09-14T04:30:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:44 smithi114 ceph-mon[103029]: 1.2 scrub starts 2024-09-14T04:30:44.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:44 smithi114 ceph-mon[103029]: 1.2 scrub ok 2024-09-14T04:30:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:44 smithi160 ceph-mon[88407]: pgmap v1607: 129 pgs: 10 unknown, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 116 active+clean; 451 MiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 1718/21693 objects degraded (7.920%); 224/21693 objects misplaced (1.033%); 1.8 MiB/s, 54 objects/s recovering 2024-09-14T04:30:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:44 smithi160 ceph-mon[88407]: 1.14 scrub starts 2024-09-14T04:30:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:44 smithi160 ceph-mon[88407]: 1.14 scrub ok 2024-09-14T04:30:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:44 smithi160 ceph-mon[88407]: 1.2 scrub starts 2024-09-14T04:30:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:44 smithi160 ceph-mon[88407]: 1.2 scrub ok 2024-09-14T04:30:44.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:44.966 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Removed 137783 objects 2024-09-14T04:30:44.966 INFO:tasks.radosbench.radosbench.0.smithi114.stdout:Clean up completed and total clean up time :104.408 2024-09-14T04:30:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:44.978 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:44.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:45.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:45.100 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:45.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:45.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:45.224 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:45.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:45.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:45.347 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:45.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:45.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:45.470 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:45.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:45.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:45.592 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:45.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:45.694 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:45.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:45.717 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:45.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:45.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:45.839 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:45.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:45.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:45.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:45 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1718/21693 objects degraded (7.920%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:45.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:45 smithi114 ceph-mon[103029]: 1.c scrub starts 2024-09-14T04:30:45.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:45 smithi114 ceph-mon[103029]: 1.c scrub ok 2024-09-14T04:30:45.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:45 smithi114 ceph-mon[103029]: 1.d scrub starts 2024-09-14T04:30:45.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:45 smithi114 ceph-mon[103029]: 1.d scrub ok 2024-09-14T04:30:45.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:45 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1718/21693 objects degraded (7.920%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:45.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:45 smithi114 ceph-mon[106969]: 1.c scrub starts 2024-09-14T04:30:45.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:45 smithi114 ceph-mon[106969]: 1.c scrub ok 2024-09-14T04:30:45.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:45 smithi114 ceph-mon[106969]: 1.d scrub starts 2024-09-14T04:30:45.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:45 smithi114 ceph-mon[106969]: 1.d scrub ok 2024-09-14T04:30:45.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:45 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1718/21693 objects degraded (7.920%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:45 smithi160 ceph-mon[88407]: 1.c scrub starts 2024-09-14T04:30:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:45 smithi160 ceph-mon[88407]: 1.c scrub ok 2024-09-14T04:30:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:45 smithi160 ceph-mon[88407]: 1.d scrub starts 2024-09-14T04:30:45.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:45 smithi160 ceph-mon[88407]: 1.d scrub ok 2024-09-14T04:30:45.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:45.962 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:45.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:46.065 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:46.086 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:46.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:46.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:46.209 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:46.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:46.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:46.332 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:46.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:46.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:46.457 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:46.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:46.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:46.579 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:46.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:46.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:46.681 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:46.703 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:46.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:46.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:46.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:46.826 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:46.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:46.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:46 smithi114 ceph-mon[103029]: pgmap v1608: 129 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 126 active+clean; 24 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 2.16k op/s; 1321/1218 objects degraded (108.456%); 1.8 MiB/s, 120 objects/s recovering 2024-09-14T04:30:46.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:46 smithi114 ceph-mon[103029]: 1.12 scrub starts 2024-09-14T04:30:46.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:46 smithi114 ceph-mon[103029]: 1.12 scrub ok 2024-09-14T04:30:46.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:46 smithi114 ceph-mon[106969]: pgmap v1608: 129 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 126 active+clean; 24 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 2.16k op/s; 1321/1218 objects degraded (108.456%); 1.8 MiB/s, 120 objects/s recovering 2024-09-14T04:30:46.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:46 smithi114 ceph-mon[106969]: 1.12 scrub starts 2024-09-14T04:30:46.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:46 smithi114 ceph-mon[106969]: 1.12 scrub ok 2024-09-14T04:30:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:46 smithi160 ceph-mon[88407]: pgmap v1608: 129 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 126 active+clean; 24 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 2.16k op/s; 1321/1218 objects degraded (108.456%); 1.8 MiB/s, 120 objects/s recovering 2024-09-14T04:30:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:46 smithi160 ceph-mon[88407]: 1.12 scrub starts 2024-09-14T04:30:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:46 smithi160 ceph-mon[88407]: 1.12 scrub ok 2024-09-14T04:30:46.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:46.951 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:46.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:47.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:47.074 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:47.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:47.176 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:47.199 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:47.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:47.215 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_9 2024-09-14T04:30:47.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_9 unique_pool_9 --yes-i-really-really-mean-it 2024-09-14T04:30:47.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:47.321 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:47.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:47.423 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:47.445 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:47.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:47.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:47.568 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:47.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:47.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:47.691 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:47.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:47.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:47.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:47.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:47.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:47.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:47.938 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:47.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:48.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:48.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:48.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:48.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:48.184 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:48.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:48.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:48.306 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:48.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:48.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:48.430 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:48.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:48.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:48.552 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:48.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:48.655 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:48.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:48.682 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:48.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:48.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:48.805 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:48.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:48.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:48 smithi114 ceph-mon[106969]: pgmap v1609: 129 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 126 active+clean; 24 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.01k op/s; 1321/1218 objects degraded (108.456%); 0 B/s, 58 objects/s recovering 2024-09-14T04:30:48.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:48 smithi114 ceph-mon[106969]: 14.1a scrub starts 2024-09-14T04:30:48.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:48 smithi114 ceph-mon[103029]: pgmap v1609: 129 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 126 active+clean; 24 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.01k op/s; 1321/1218 objects degraded (108.456%); 0 B/s, 58 objects/s recovering 2024-09-14T04:30:48.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:48 smithi114 ceph-mon[103029]: 14.1a scrub starts 2024-09-14T04:30:48.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:48 smithi160 ceph-mon[88407]: pgmap v1609: 129 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 126 active+clean; 24 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.01k op/s; 1321/1218 objects degraded (108.456%); 0 B/s, 58 objects/s recovering 2024-09-14T04:30:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:48 smithi160 ceph-mon[88407]: 14.1a scrub starts 2024-09-14T04:30:48.928 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:48.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:49.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:49.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:49.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:49.178 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:49.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:49.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:49.302 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:49.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:49.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:49.427 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:49.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:49.528 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:49.551 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:49.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:49.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:49.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:49.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:49.675 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:49.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:49.776 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:49.799 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:49.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:49.850 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:49 smithi114 ceph-mon[106969]: osdmap e426: 8 total, 8 up, 5 in 2024-09-14T04:30:49.850 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:49 smithi114 ceph-mon[106969]: 14.23 scrub starts 2024-09-14T04:30:49.850 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:49 smithi114 ceph-mon[106969]: 14.23 scrub ok 2024-09-14T04:30:49.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:49 smithi114 ceph-mon[103029]: osdmap e426: 8 total, 8 up, 5 in 2024-09-14T04:30:49.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:49 smithi114 ceph-mon[103029]: 14.23 scrub starts 2024-09-14T04:30:49.851 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:49 smithi114 ceph-mon[103029]: 14.23 scrub ok 2024-09-14T04:30:49.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:49.907 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:49 smithi160 ceph-mon[88407]: osdmap e426: 8 total, 8 up, 5 in 2024-09-14T04:30:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:49 smithi160 ceph-mon[88407]: 14.23 scrub starts 2024-09-14T04:30:49.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:49 smithi160 ceph-mon[88407]: 14.23 scrub ok 2024-09-14T04:30:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:49.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:49.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:49.940 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:50.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:50.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:50.048 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:50.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:50.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:50.173 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:50.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:50.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:50.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:50.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:50.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:50.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:50.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:50.454 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:30:50.454 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:30:50.454 INFO:teuthology.orchestra.run.smithi114.stdout: "complaint_time": 30, 2024-09-14T04:30:50.455 INFO:teuthology.orchestra.run.smithi114.stdout: "num_blocked_ops": 0 2024-09-14T04:30:50.455 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:30:50.510 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:30:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:30:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:30:50.521 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:50.543 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:50.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:50.644 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:50.666 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:50.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:50.768 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:50.791 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:50.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:50.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[103029]: pgmap v1611: 129 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 126 active+clean; 16 MiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 899 op/s; 1321/849 objects degraded (155.595%); 0 B/s, 51 objects/s recovering 2024-09-14T04:30:50.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[103029]: 1.1 scrub starts 2024-09-14T04:30:50.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[103029]: 1.1 scrub ok 2024-09-14T04:30:50.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[103029]: osdmap e427: 8 total, 8 up, 5 in 2024-09-14T04:30:50.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[103029]: Health check update: Degraded data redundancy: 1321/849 objects degraded (155.595%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:50.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[103029]: 1.6 scrub starts 2024-09-14T04:30:50.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[103029]: 1.6 scrub ok 2024-09-14T04:30:50.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[103029]: 14.14 scrub starts 2024-09-14T04:30:50.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[103029]: 14.14 scrub ok 2024-09-14T04:30:50.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[106969]: pgmap v1611: 129 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 126 active+clean; 16 MiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 899 op/s; 1321/849 objects degraded (155.595%); 0 B/s, 51 objects/s recovering 2024-09-14T04:30:50.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[106969]: 1.1 scrub starts 2024-09-14T04:30:50.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[106969]: 1.1 scrub ok 2024-09-14T04:30:50.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[106969]: osdmap e427: 8 total, 8 up, 5 in 2024-09-14T04:30:50.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[106969]: Health check update: Degraded data redundancy: 1321/849 objects degraded (155.595%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:50.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[106969]: 1.6 scrub starts 2024-09-14T04:30:50.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[106969]: 1.6 scrub ok 2024-09-14T04:30:50.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[106969]: 14.14 scrub starts 2024-09-14T04:30:50.833 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:50 smithi114 ceph-mon[106969]: 14.14 scrub ok 2024-09-14T04:30:50.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:50 smithi160 ceph-mon[88407]: pgmap v1611: 129 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 126 active+clean; 16 MiB data, 7.2 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 899 op/s; 1321/849 objects degraded (155.595%); 0 B/s, 51 objects/s recovering 2024-09-14T04:30:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:50 smithi160 ceph-mon[88407]: 1.1 scrub starts 2024-09-14T04:30:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:50 smithi160 ceph-mon[88407]: 1.1 scrub ok 2024-09-14T04:30:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:50 smithi160 ceph-mon[88407]: osdmap e427: 8 total, 8 up, 5 in 2024-09-14T04:30:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:50 smithi160 ceph-mon[88407]: Health check update: Degraded data redundancy: 1321/849 objects degraded (155.595%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-14T04:30:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:50 smithi160 ceph-mon[88407]: 1.6 scrub starts 2024-09-14T04:30:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:50 smithi160 ceph-mon[88407]: 1.6 scrub ok 2024-09-14T04:30:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:50 smithi160 ceph-mon[88407]: 14.14 scrub starts 2024-09-14T04:30:50.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:50 smithi160 ceph-mon[88407]: 14.14 scrub ok 2024-09-14T04:30:50.916 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:50.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:51.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:51.040 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:51.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:51.142 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:51.163 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:51.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:51.175 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:30:51.175 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:30:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:30:51.176 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:30:51.189 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:30:51.189 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:30:51.207 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:30:51.207 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:30:51.222 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:30:51.222 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:30:51.238 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:30:51.238 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:30:51.253 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:30:51.254 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:30:51.265 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:51.269 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:30:51.269 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:30:51.288 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:51.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:51.389 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:51.411 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:51.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:51.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:51.541 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:51.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:51.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:51.664 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:51.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:51.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:51.787 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:51.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:51.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:51.789 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:30:51] "GET /metrics HTTP/1.1" 200 36572 "" "Prometheus/2.51.0" 2024-09-14T04:30:51.789 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-mon[106969]: 1.0 scrub starts 2024-09-14T04:30:51.789 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-mon[106969]: 1.0 scrub ok 2024-09-14T04:30:51.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4218530195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:30:51.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-14T04:30:51.790 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-14T04:30:51.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-mon[103029]: 1.0 scrub starts 2024-09-14T04:30:51.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-mon[103029]: 1.0 scrub ok 2024-09-14T04:30:51.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4218530195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:30:51.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-14T04:30:51.790 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:51 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-14T04:30:51.813 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-14T04:30:51.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:51 smithi160 ceph-mon[88407]: 1.0 scrub starts 2024-09-14T04:30:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:51 smithi160 ceph-mon[88407]: 1.0 scrub ok 2024-09-14T04:30:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:51 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4218530195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:30:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-14T04:30:51.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:51 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-14T04:30:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:51.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:51.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:30:52.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:52.035 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:52.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:52.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:52.159 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:52.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:52.216 INFO:tasks.thrashosds.thrasher:in_osds: [3, 6, 4, 5, 2] out_osds: [1, 7, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-14T04:30:52.216 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-14T04:30:52.216 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-14T04:30:52.216 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-14T04:30:52.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:52.282 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:52.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:52.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:52.384 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:52.410 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:52.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:52.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:52.518 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[103029]: pgmap v1613: 129 pgs: 129 active+clean; 5.2 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 495 op/s; 0 B/s, 214 objects/s recovering 2024-09-14T04:30:52.518 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[103029]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1321/849 objects degraded (155.595%), 1 pg degraded, 1 pg undersized) 2024-09-14T04:30:52.518 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-14T04:30:52.544 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:52.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:52.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:52.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:52.669 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:52.669 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:52.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:52.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:52.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-14T04:30:52.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[103029]: osdmap e428: 8 total, 8 up, 5 in 2024-09-14T04:30:52.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[103029]: 1.3 scrub starts 2024-09-14T04:30:52.792 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[103029]: 1.3 scrub ok 2024-09-14T04:30:52.792 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[106969]: pgmap v1613: 129 pgs: 129 active+clean; 5.2 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 495 op/s; 0 B/s, 214 objects/s recovering 2024-09-14T04:30:52.793 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[106969]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1321/849 objects degraded (155.595%), 1 pg degraded, 1 pg undersized) 2024-09-14T04:30:52.793 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-14T04:30:52.793 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-14T04:30:52.793 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[106969]: osdmap e428: 8 total, 8 up, 5 in 2024-09-14T04:30:52.793 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[106969]: 1.3 scrub starts 2024-09-14T04:30:52.793 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:52 smithi114 ceph-mon[106969]: 1.3 scrub ok 2024-09-14T04:30:52.793 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:52.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:52.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:52 smithi160 ceph-mon[88407]: pgmap v1613: 129 pgs: 129 active+clean; 5.2 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 495 op/s; 0 B/s, 214 objects/s recovering 2024-09-14T04:30:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:52 smithi160 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1321/849 objects degraded (155.595%), 1 pg degraded, 1 pg undersized) 2024-09-14T04:30:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:52 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-14T04:30:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:52 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-14T04:30:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:52 smithi160 ceph-mon[88407]: osdmap e428: 8 total, 8 up, 5 in 2024-09-14T04:30:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:52 smithi160 ceph-mon[88407]: 1.3 scrub starts 2024-09-14T04:30:52.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:52 smithi160 ceph-mon[88407]: 1.3 scrub ok 2024-09-14T04:30:52.917 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:52.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:52.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:53.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:53.041 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:53.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:53.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:53.164 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:53.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:53.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:53.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:53.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:53.394 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:53.419 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:53.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:53.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:53.542 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:53.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:53.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[106969]: 14.17 scrub starts 2024-09-14T04:30:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[106969]: 14.17 scrub ok 2024-09-14T04:30:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[106969]: osdmap e429: 8 total, 8 up, 5 in 2024-09-14T04:30:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[106969]: 1.7 scrub starts 2024-09-14T04:30:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[106969]: 1.7 scrub ok 2024-09-14T04:30:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[106969]: 1.13 scrub starts 2024-09-14T04:30:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[106969]: 1.13 scrub ok 2024-09-14T04:30:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-14T04:30:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[103029]: 14.17 scrub starts 2024-09-14T04:30:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[103029]: 14.17 scrub ok 2024-09-14T04:30:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[103029]: osdmap e429: 8 total, 8 up, 5 in 2024-09-14T04:30:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[103029]: 1.7 scrub starts 2024-09-14T04:30:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[103029]: 1.7 scrub ok 2024-09-14T04:30:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[103029]: 1.13 scrub starts 2024-09-14T04:30:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[103029]: 1.13 scrub ok 2024-09-14T04:30:53.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-14T04:30:53.644 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:53.665 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:53.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:53.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:53.782 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:53.792 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:53.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:53.897 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:53 smithi160 ceph-mon[88407]: 14.17 scrub starts 2024-09-14T04:30:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:53 smithi160 ceph-mon[88407]: 14.17 scrub ok 2024-09-14T04:30:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:53 smithi160 ceph-mon[88407]: osdmap e429: 8 total, 8 up, 5 in 2024-09-14T04:30:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:30:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:53 smithi160 ceph-mon[88407]: 1.7 scrub starts 2024-09-14T04:30:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:53 smithi160 ceph-mon[88407]: 1.7 scrub ok 2024-09-14T04:30:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:53 smithi160 ceph-mon[88407]: 1.13 scrub starts 2024-09-14T04:30:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:53 smithi160 ceph-mon[88407]: 1.13 scrub ok 2024-09-14T04:30:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-14T04:30:53.923 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:53.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:54.025 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:54.053 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:54.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:54.154 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:54.178 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:54.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:54.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:54.302 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:54.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:54.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:54.435 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:54.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:54.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:54.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:54.558 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:54.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:54.659 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:54.686 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:54.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:54.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:54.809 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:54.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:54.911 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:54.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:54.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:54.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[106969]: pgmap v1616: 129 pgs: 129 active+clean; 5.2 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s, 278 objects/s recovering 2024-09-14T04:30:54.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[106969]: 14.a scrub starts 2024-09-14T04:30:54.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[106969]: 14.a scrub ok 2024-09-14T04:30:54.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-14T04:30:54.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[106969]: osdmap e430: 8 total, 8 up, 5 in 2024-09-14T04:30:54.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[106969]: 1.e scrub starts 2024-09-14T04:30:54.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[106969]: 1.e scrub ok 2024-09-14T04:30:54.939 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[106969]: osdmap e431: 8 total, 8 up, 5 in 2024-09-14T04:30:54.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[103029]: pgmap v1616: 129 pgs: 129 active+clean; 5.2 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s, 278 objects/s recovering 2024-09-14T04:30:54.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[103029]: 14.a scrub starts 2024-09-14T04:30:54.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[103029]: 14.a scrub ok 2024-09-14T04:30:54.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-14T04:30:54.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[103029]: osdmap e430: 8 total, 8 up, 5 in 2024-09-14T04:30:54.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[103029]: 1.e scrub starts 2024-09-14T04:30:54.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[103029]: 1.e scrub ok 2024-09-14T04:30:54.940 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:54 smithi114 ceph-mon[103029]: osdmap e431: 8 total, 8 up, 5 in 2024-09-14T04:30:55.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:55.061 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:55.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:54 smithi160 ceph-mon[88407]: pgmap v1616: 129 pgs: 129 active+clean; 5.2 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s, 278 objects/s recovering 2024-09-14T04:30:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:54 smithi160 ceph-mon[88407]: 14.a scrub starts 2024-09-14T04:30:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:54 smithi160 ceph-mon[88407]: 14.a scrub ok 2024-09-14T04:30:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_9", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-14T04:30:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:54 smithi160 ceph-mon[88407]: osdmap e430: 8 total, 8 up, 5 in 2024-09-14T04:30:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:54 smithi160 ceph-mon[88407]: 1.e scrub starts 2024-09-14T04:30:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:54 smithi160 ceph-mon[88407]: 1.e scrub ok 2024-09-14T04:30:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:54 smithi160 ceph-mon[88407]: osdmap e431: 8 total, 8 up, 5 in 2024-09-14T04:30:55.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:55.183 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:55.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:55.285 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:55.306 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:55.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:55.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:55.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:55.434 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:55.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:55.536 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:55.559 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:55.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:55.660 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:30:55.683 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:55.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:30:55.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:55.810 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:55.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:55.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:55 smithi114 ceph-mon[103029]: 1.f scrub starts 2024-09-14T04:30:55.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:55 smithi114 ceph-mon[103029]: 1.f scrub ok 2024-09-14T04:30:55.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:55 smithi114 ceph-mon[103029]: osdmap e432: 8 total, 8 up, 5 in 2024-09-14T04:30:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:55 smithi114 ceph-mon[106969]: 1.f scrub starts 2024-09-14T04:30:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:55 smithi114 ceph-mon[106969]: 1.f scrub ok 2024-09-14T04:30:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:55 smithi114 ceph-mon[106969]: osdmap e432: 8 total, 8 up, 5 in 2024-09-14T04:30:55.912 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:55.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:55.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:56.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:56.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:56.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:55 smithi160 ceph-mon[88407]: 1.f scrub starts 2024-09-14T04:30:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:55 smithi160 ceph-mon[88407]: 1.f scrub ok 2024-09-14T04:30:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:55 smithi160 ceph-mon[88407]: osdmap e432: 8 total, 8 up, 5 in 2024-09-14T04:30:56.165 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:56.189 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:56.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:56.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:56.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:56.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:56.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:56.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:56.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:56.560 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:56.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:56.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:56.685 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:56.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:56.695 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'unique_pool_9' removed 2024-09-14T04:30:56.787 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:56.790 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-14T04:30:56.821 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:56.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:56.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:56.944 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:56.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:56.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:56 smithi114 ceph-mon[103029]: pgmap v1619: 129 pgs: 1 remapped+peering, 128 active+clean; 3.0 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 23 KiB/s rd, 8 op/s 2024-09-14T04:30:56.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:56 smithi114 ceph-mon[103029]: 14.16 starting backfill to osd.5 from (0'0,0'0] MAX to 425'10549 2024-09-14T04:30:56.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:56 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/98678454' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:30:56.951 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:56 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:30:56.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:56 smithi114 ceph-mon[106969]: pgmap v1619: 129 pgs: 1 remapped+peering, 128 active+clean; 3.0 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 23 KiB/s rd, 8 op/s 2024-09-14T04:30:56.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:56 smithi114 ceph-mon[106969]: 14.16 starting backfill to osd.5 from (0'0,0'0] MAX to 425'10549 2024-09-14T04:30:56.951 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:56 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/98678454' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:30:56.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:56 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:30:57.046 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:57.068 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:57.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:57.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:56 smithi160 ceph-mon[88407]: pgmap v1619: 129 pgs: 1 remapped+peering, 128 active+clean; 3.0 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 23 KiB/s rd, 8 op/s 2024-09-14T04:30:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:56 smithi160 ceph-mon[88407]: 14.16 starting backfill to osd.5 from (0'0,0'0] MAX to 425'10549 2024-09-14T04:30:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:56 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/98678454' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:30:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:56 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-14T04:30:57.170 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:30:57.192 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:57.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:30:57.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:57.314 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:57.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:57.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:57.439 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:57.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:57.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:57.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:30:57.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:57.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:57.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:30:57.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:57.688 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:57.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:57.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:57.813 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:57.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:57.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:57.915 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:57.937 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:57.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:57.952 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:57 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:30:57.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:57 smithi114 ceph-mon[103029]: osdmap e433: 8 total, 8 up, 5 in 2024-09-14T04:30:57.953 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:57 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:30:57.953 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:57 smithi114 ceph-mon[106969]: osdmap e433: 8 total, 8 up, 5 in 2024-09-14T04:30:57.954 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-14T04:30:57.954 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-14T04:30:57.954 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-14T04:30:57.961 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-09-14T04:30:57.961 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-14T04:30:57.962 DEBUG:teuthology.orchestra.run.smithi114:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-14T04:30:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:57.988 INFO:teuthology.orchestra.run.smithi114.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-14T04:30:57.988 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-14T04:30:58.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:58.062 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:58.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:58.069 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-14T04:30:58.069 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-14T04:30:58.124 INFO:tasks.workunit:timeout=3h 2024-09-14T04:30:58.124 INFO:tasks.workunit:cleanup=True 2024-09-14T04:30:58.125 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-14T04:30:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:57 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-09-14T04:30:58.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:57 smithi160 ceph-mon[88407]: osdmap e433: 8 total, 8 up, 5 in 2024-09-14T04:30:58.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:58.188 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:58.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:58.201 INFO:tasks.workunit.client.0.smithi114.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-14T04:30:58.237 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:58.289 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:58.312 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:58.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:58.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:58.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:58.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:58.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:58.562 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:58.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:58.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:58.687 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:30:58.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:58.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:58.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:30:58.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:58.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:30:58.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:58.936 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:58.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:58.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:58.949 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[106969]: 1.b scrub starts 2024-09-14T04:30:58.949 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[106969]: 1.b scrub ok 2024-09-14T04:30:58.949 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[106969]: pgmap v1622: 93 pgs: 93 active+clean; 652 KiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 22 KiB/s rd, 8 op/s 2024-09-14T04:30:58.949 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:58.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/4036275954' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:30:58.950 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:30:58.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[103029]: 1.b scrub starts 2024-09-14T04:30:58.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[103029]: 1.b scrub ok 2024-09-14T04:30:58.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[103029]: pgmap v1622: 93 pgs: 93 active+clean; 652 KiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 22 KiB/s rd, 8 op/s 2024-09-14T04:30:58.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:58.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/4036275954' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:30:58.950 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:58 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:30:59.038 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:59.060 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:59.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:58 smithi160 ceph-mon[88407]: 1.b scrub starts 2024-09-14T04:30:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:58 smithi160 ceph-mon[88407]: 1.b scrub ok 2024-09-14T04:30:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:58 smithi160 ceph-mon[88407]: pgmap v1622: 93 pgs: 93 active+clean; 652 KiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 22 KiB/s rd, 8 op/s 2024-09-14T04:30:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:58 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:30:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:58 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/4036275954' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:30:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:58 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:30:59.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:59.171 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:30:59.186 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:59.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:30:59.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:59.311 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:59.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:59.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:59.413 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:30:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:59.436 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:30:59.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:30:59.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:59.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:59.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:59.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:30:59.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:59.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:30:59.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:30:59.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:59.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:30:59.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[103029]: 1.11 scrub starts 2024-09-14T04:30:59.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[103029]: 1.11 scrub ok 2024-09-14T04:30:59.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:30:59.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:30:59.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:30:59.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[103029]: osdmap e434: 8 total, 8 up, 5 in 2024-09-14T04:30:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[106969]: 1.11 scrub starts 2024-09-14T04:30:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[106969]: 1.11 scrub ok 2024-09-14T04:30:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:30:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:30:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:30:59.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:30:59 smithi114 ceph-mon[106969]: osdmap e434: 8 total, 8 up, 5 in 2024-09-14T04:30:59.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:30:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:30:59.939 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:30:59.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:00.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:59 smithi160 ceph-mon[88407]: 1.11 scrub starts 2024-09-14T04:31:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:59 smithi160 ceph-mon[88407]: 1.11 scrub ok 2024-09-14T04:31:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:59 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:31:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:59 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:31:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:59 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:31:00.060 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:30:59 smithi160 ceph-mon[88407]: osdmap e434: 8 total, 8 up, 5 in 2024-09-14T04:31:00.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:00.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:00.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:00.186 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:00.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:00.288 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:00.309 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:00.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:00.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:00.334 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:31:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:31:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:31:00.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:00.433 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:00.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:00.534 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:00.556 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:00.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:00.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:00.678 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:00.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:00.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:00.802 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:00.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:00.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:00.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:00.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:01.027 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:01.028 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:00 smithi114 ceph-mon[106969]: pgmap v1624: 93 pgs: 93 active+clean; 652 KiB data, 6.8 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:01.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:00 smithi114 ceph-mon[103029]: pgmap v1624: 93 pgs: 93 active+clean; 652 KiB data, 6.8 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:01.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:01.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:01.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:01.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:01.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:00 smithi160 ceph-mon[88407]: pgmap v1624: 93 pgs: 93 active+clean; 652 KiB data, 6.8 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:01.173 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:01.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:01.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:01.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:01.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:01.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:01.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:01.421 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:01.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:01.488 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:31:01.488 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:31:01.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:31:01.545 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:01.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:31:01.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:01.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:01.669 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:01.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:01.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:31:01.791 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:01.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:31:01.827 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:31:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:31:01] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-09-14T04:31:01.893 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:01.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:02.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:31:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:02.038 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:02.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:31:02.139 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:31:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:02.162 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:02.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:31:02.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:31:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:02.285 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:02.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:31:02.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:02.407 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:02.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:02.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:02.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:02.532 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:02.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:02.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:02.654 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:02.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:02.756 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:02.777 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:02.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:02.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:02.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:02.901 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:02.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:02.924 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:31:02.925 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:31:03.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:03.026 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:03.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:02 smithi114 ceph-mon[103029]: pgmap v1625: 93 pgs: 93 active+clean; 634 KiB data, 7.1 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:02 smithi114 ceph-mon[103029]: 1.12 scrub starts 2024-09-14T04:31:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:02 smithi114 ceph-mon[103029]: 1.12 scrub ok 2024-09-14T04:31:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:02 smithi114 ceph-mon[106969]: pgmap v1625: 93 pgs: 93 active+clean; 634 KiB data, 7.1 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:02 smithi114 ceph-mon[106969]: 1.12 scrub starts 2024-09-14T04:31:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:02 smithi114 ceph-mon[106969]: 1.12 scrub ok 2024-09-14T04:31:03.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:03.150 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:03.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:03.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:02 smithi160 ceph-mon[88407]: pgmap v1625: 93 pgs: 93 active+clean; 634 KiB data, 7.1 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:02 smithi160 ceph-mon[88407]: 1.12 scrub starts 2024-09-14T04:31:03.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:02 smithi160 ceph-mon[88407]: 1.12 scrub ok 2024-09-14T04:31:03.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:03.272 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:03.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:03.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:03.395 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:03.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:03.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:03.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:03.521 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:03.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:03.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:03.647 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:03.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:03.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:03.772 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:03.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:03.874 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:03.897 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:03.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:03.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:03 smithi114 ceph-mon[103029]: 1.10 scrub starts 2024-09-14T04:31:03.977 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:03 smithi114 ceph-mon[103029]: 1.10 scrub ok 2024-09-14T04:31:03.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:03 smithi114 ceph-mon[106969]: 1.10 scrub starts 2024-09-14T04:31:03.977 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:03 smithi114 ceph-mon[106969]: 1.10 scrub ok 2024-09-14T04:31:03.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:31:04.001 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:31:04.001 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-14T04:31:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:04.019 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:04.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:04.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:04.144 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:04.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:03 smithi160 ceph-mon[88407]: 1.10 scrub starts 2024-09-14T04:31:04.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:03 smithi160 ceph-mon[88407]: 1.10 scrub ok 2024-09-14T04:31:04.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-14T04:31:04.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:04.267 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:04.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:04.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:04.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:04.391 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:04.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:04.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:04.514 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:04.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:04.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:04.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:31:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:04.643 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:04.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:31:04.694 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-14T04:31:04.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-14T04:31:04.732 INFO:tasks.workunit.client.0.smithi114.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-14T04:31:04.734 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-14T04:31:04.734 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-14T04:31:04.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:04.796 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:04.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:04.898 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:04.919 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:04.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:04.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:04.999 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:04 smithi114 ceph-mon[106969]: pgmap v1626: 93 pgs: 93 active+clean; 634 KiB data, 7.1 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:04.999 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:04 smithi114 ceph-mon[103029]: pgmap v1626: 93 pgs: 93 active+clean; 634 KiB data, 7.1 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:05.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:05.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:05.049 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:05.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:05.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:05.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:04 smithi160 ceph-mon[88407]: pgmap v1626: 93 pgs: 93 active+clean; 634 KiB data, 7.1 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:05.173 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:05.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:05.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:05.296 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:05.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:05.357 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:31:05.358 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:31:05.375 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:31:05.375 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:31:05.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:05.400 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:31:05.400 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:31:05.406 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:05.407 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:05.412 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:05.424 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:05.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:05.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:05.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:05.651 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:05.673 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:05.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:05.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:05.800 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:05.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:05.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:05.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:05.925 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:05.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:06.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:06.037 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:05 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1784462375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:31:06.037 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:05 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1784462375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:31:06.050 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:06.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:06.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:06.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:05 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1784462375' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:31:06.178 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:06.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:06.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:06.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:06.301 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:06.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:06.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:06.424 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:06.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:06.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:06.552 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:06.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:06.653 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:06.676 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:06.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:06.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:06.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:06.804 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:06.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:06.873 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-14T04:31:06.907 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:06.932 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:06.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:07.010 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-14T04:31:07.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:07.053 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-14T04:31:07.054 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:07.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:07.055 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-14T04:31:07.055 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-14T04:31:07.105 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-14T04:31:07.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:06 smithi114 ceph-mon[106969]: pgmap v1627: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:07.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:06 smithi114 ceph-mon[103029]: pgmap v1627: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:07.133 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-14T04:31:07.133 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-14T04:31:07.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:07.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:06 smithi160 ceph-mon[88407]: pgmap v1627: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:07.194 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-09-14T04:31:07.195 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-09-14T04:31:07.196 DEBUG:teuthology.orchestra.run.smithi114:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-09-14T04:31:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:07.197 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:07.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:07.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:07.316 INFO:tasks.workunit.client.0.smithi114.stdout:Running main() from gmock_main.cc 2024-09-14T04:31:07.317 INFO:tasks.workunit.client.0.smithi114.stdout:[==========] Running 70 tests from 1 test suite. 2024-09-14T04:31:07.317 INFO:tasks.workunit.client.0.smithi114.stdout:[----------] Global test environment set-up. 2024-09-14T04:31:07.317 INFO:tasks.workunit.client.0.smithi114.stdout:[----------] 70 tests from TestClsRbd 2024-09-14T04:31:07.322 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:07.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:07.424 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:07.447 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:07.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:07.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:07.569 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:07.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:07.672 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:07.692 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:07.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:07.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:07.815 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:07.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:07.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:07.944 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:07.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:07.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:08.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:08.071 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:08.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:08.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:08.198 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:08.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:08.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:08.325 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:08.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:08.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:08.448 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:08.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:08.550 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:08.572 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:08.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:08.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:08.695 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:08.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:08.737 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.get_all_features 2024-09-14T04:31:08.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:08.822 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:08.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:08.903 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.get_all_features (166 ms) 2024-09-14T04:31:08.903 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.copyup 2024-09-14T04:31:08.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:31:08.949 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:08.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:31:09.000 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[106969]: pgmap v1628: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:09.000 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:09.000 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:09.000 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[106969]: osdmap e435: 8 total, 8 up, 5 in 2024-09-14T04:31:09.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/579555709' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-355559-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-14T04:31:09.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-355559-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-14T04:31:09.001 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[103029]: pgmap v1628: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:09.001 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:09.001 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:09.001 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[103029]: osdmap e435: 8 total, 8 up, 5 in 2024-09-14T04:31:09.001 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/579555709' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-355559-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-14T04:31:09.001 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:08 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-355559-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-14T04:31:09.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:09.072 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:09.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:09.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:09.091 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.copyup (188 ms) 2024-09-14T04:31:09.091 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-09-14T04:31:09.135 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.sparse_copyup (43 ms) 2024-09-14T04:31:09.135 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-09-14T04:31:09.141 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.get_and_set_id (7 ms) 2024-09-14T04:31:09.142 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-09-14T04:31:09.151 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.add_remove_child (10 ms) 2024-09-14T04:31:09.151 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.directory_methods 2024-09-14T04:31:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:08 smithi160 ceph-mon[88407]: pgmap v1628: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:08 smithi160 ceph-mon[88407]: osdmap e435: 8 total, 8 up, 5 in 2024-09-14T04:31:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/579555709' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-355559-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-14T04:31:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:08 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-355559-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-14T04:31:09.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:09.175 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.directory_methods (24 ms) 2024-09-14T04:31:09.175 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.create 2024-09-14T04:31:09.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-14T04:31:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:09.196 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:09.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:09.212 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.create (37 ms) 2024-09-14T04:31:09.212 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.get_features 2024-09-14T04:31:09.214 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.get_features (2 ms) 2024-09-14T04:31:09.214 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-09-14T04:31:09.216 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.get_object_prefix (1 ms) 2024-09-14T04:31:09.216 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-09-14T04:31:09.217 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.get_create_timestamp (1 ms) 2024-09-14T04:31:09.217 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-09-14T04:31:09.219 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.get_access_timestamp (2 ms) 2024-09-14T04:31:09.219 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-09-14T04:31:09.220 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.get_modify_timestamp (1 ms) 2024-09-14T04:31:09.220 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-09-14T04:31:09.224 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.get_data_pool (4 ms) 2024-09-14T04:31:09.224 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.get_size 2024-09-14T04:31:09.228 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.get_size (3 ms) 2024-09-14T04:31:09.228 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.set_size 2024-09-14T04:31:09.233 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.set_size (5 ms) 2024-09-14T04:31:09.233 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.protection_status 2024-09-14T04:31:09.248 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.protection_status (16 ms) 2024-09-14T04:31:09.248 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-09-14T04:31:09.254 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.snapshot_limits (6 ms) 2024-09-14T04:31:09.254 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.parents_v1 2024-09-14T04:31:09.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:09.298 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.parents_v1 (43 ms) 2024-09-14T04:31:09.298 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.parents_v2 2024-09-14T04:31:09.319 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:09.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:09.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:09.326 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.parents_v2 (29 ms) 2024-09-14T04:31:09.327 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.snapshots 2024-09-14T04:31:09.349 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.snapshots (22 ms) 2024-09-14T04:31:09.349 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.snapid_race 2024-09-14T04:31:09.353 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.snapid_race (4 ms) 2024-09-14T04:31:09.353 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.stripingv2 2024-09-14T04:31:09.361 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.stripingv2 (8 ms) 2024-09-14T04:31:09.361 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.object_map_save 2024-09-14T04:31:09.364 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.object_map_save (3 ms) 2024-09-14T04:31:09.364 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-09-14T04:31:09.371 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.object_map_resize (7 ms) 2024-09-14T04:31:09.371 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.object_map_update 2024-09-14T04:31:09.379 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.object_map_update (8 ms) 2024-09-14T04:31:09.379 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-09-14T04:31:09.379 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.object_map_load_enoent (0 ms) 2024-09-14T04:31:09.380 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-09-14T04:31:09.385 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.object_map_snap_add (6 ms) 2024-09-14T04:31:09.385 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-09-14T04:31:09.393 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.object_map_snap_remove (8 ms) 2024-09-14T04:31:09.393 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.flags 2024-09-14T04:31:09.399 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.flags (6 ms) 2024-09-14T04:31:09.399 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.metadata 2024-09-14T04:31:09.420 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:09.422 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.metadata (23 ms) 2024-09-14T04:31:09.422 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.set_features 2024-09-14T04:31:09.428 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.set_features (6 ms) 2024-09-14T04:31:09.429 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.mirror 2024-09-14T04:31:09.443 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:09.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:09.460 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.mirror (32 ms) 2024-09-14T04:31:09.460 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.mirror_image 2024-09-14T04:31:09.473 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.mirror_image (13 ms) 2024-09-14T04:31:09.473 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-09-14T04:31:09.545 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:09.566 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:09.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:09.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:09.691 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:09.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:09.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:09.817 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:09.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:09.918 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:09.940 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:09.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:09.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:10.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:10.065 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:10.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:10.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-355559-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-14T04:31:10.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:09 smithi114 ceph-mon[103029]: osdmap e436: 8 total, 8 up, 5 in 2024-09-14T04:31:10.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-355559-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-14T04:31:10.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:09 smithi114 ceph-mon[106969]: osdmap e436: 8 total, 8 up, 5 in 2024-09-14T04:31:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi114-355559-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-14T04:31:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:09 smithi160 ceph-mon[88407]: osdmap e436: 8 total, 8 up, 5 in 2024-09-14T04:31:10.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:10.189 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:10.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:10.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:10.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:10.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:10.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:10.439 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:10.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:10.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:10.563 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:10.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:10.657 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:31:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:31:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:31:10.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:10.687 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:10.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:10.739 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is set 2024-09-14T04:31:10.789 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:10.811 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:10.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:10.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:10.936 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:10.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:11.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:11.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:10 smithi114 ceph-mon[106969]: pgmap v1631: 125 pgs: 23 unknown, 102 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:11.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:10 smithi114 ceph-mon[106969]: osdmap e437: 8 total, 8 up, 5 in 2024-09-14T04:31:11.056 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:10 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/211823037' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:31:11.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:10 smithi114 ceph-mon[103029]: pgmap v1631: 125 pgs: 23 unknown, 102 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:11.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:10 smithi114 ceph-mon[103029]: osdmap e437: 8 total, 8 up, 5 in 2024-09-14T04:31:11.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:10 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/211823037' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:31:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:11.062 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:11.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:10 smithi160 ceph-mon[88407]: pgmap v1631: 125 pgs: 23 unknown, 102 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:10 smithi160 ceph-mon[88407]: osdmap e437: 8 total, 8 up, 5 in 2024-09-14T04:31:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:10 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/211823037' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-14T04:31:11.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:11.188 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:11.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:11.291 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:11.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:11.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:11.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:11.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:11.439 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:11.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:11.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:11.564 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:11.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:11.611 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:31:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:31:11] "GET /metrics HTTP/1.1" 200 34137 "" "Prometheus/2.51.0" 2024-09-14T04:31:11.665 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:11.687 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:11.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:11.741 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.mirror_image_status (2267 ms) 2024-09-14T04:31:11.741 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-09-14T04:31:11.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:11.812 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:11.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:11.825 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.mirror_image_map (84 ms) 2024-09-14T04:31:11.825 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-09-14T04:31:11.832 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.mirror_instances (7 ms) 2024-09-14T04:31:11.832 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-09-14T04:31:11.847 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.mirror_snapshot (15 ms) 2024-09-14T04:31:11.847 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-09-14T04:31:11.849 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_dir_list (2 ms) 2024-09-14T04:31:11.849 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-09-14T04:31:11.851 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_dir_add (2 ms) 2024-09-14T04:31:11.851 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-09-14T04:31:11.854 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.dir_add_already_existing (3 ms) 2024-09-14T04:31:11.854 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-09-14T04:31:11.859 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_dir_rename (5 ms) 2024-09-14T04:31:11.859 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-09-14T04:31:11.862 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_dir_remove (3 ms) 2024-09-14T04:31:11.862 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-09-14T04:31:11.865 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (3 ms) 2024-09-14T04:31:11.865 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_image_add 2024-09-14T04:31:11.867 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_image_add (2 ms) 2024-09-14T04:31:11.867 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-09-14T04:31:11.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:11 smithi114 ceph-mon[103029]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:31:11.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:11 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/211823037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:31:11.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:11 smithi114 ceph-mon[103029]: osdmap e438: 8 total, 8 up, 5 in 2024-09-14T04:31:11.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:11 smithi114 ceph-mon[106969]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:31:11.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:11 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/211823037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:31:11.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:11 smithi114 ceph-mon[106969]: osdmap e438: 8 total, 8 up, 5 in 2024-09-14T04:31:11.872 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_image_remove (4 ms) 2024-09-14T04:31:11.872 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_image_list 2024-09-14T04:31:11.873 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_image_list (2 ms) 2024-09-14T04:31:11.873 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-09-14T04:31:11.877 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_image_clean (4 ms) 2024-09-14T04:31:11.877 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.image_group_add 2024-09-14T04:31:11.879 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-09-14T04:31:11.879 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-09-14T04:31:11.881 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.image_group_remove (2 ms) 2024-09-14T04:31:11.882 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.image_group_get 2024-09-14T04:31:11.884 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.image_group_get (3 ms) 2024-09-14T04:31:11.884 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-09-14T04:31:11.885 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (1 ms) 2024-09-14T04:31:11.885 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-09-14T04:31:11.887 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (2 ms) 2024-09-14T04:31:11.887 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-09-14T04:31:11.890 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (3 ms) 2024-09-14T04:31:11.890 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-09-14T04:31:11.892 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (2 ms) 2024-09-14T04:31:11.892 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-09-14T04:31:11.894 INFO:tasks.workunit.client.0.smithi114.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.7-1629-geb16b97b/rpm/el9/BUILD/ceph-17.2.7-1629-geb16b97b/src/test/cls_rbd/test_cls_rbd.cc:2681: Failure 2024-09-14T04:31:11.895 INFO:tasks.workunit.client.0.smithi114.stdout:Expected equality of these values: 2024-09-14T04:31:11.895 INFO:tasks.workunit.client.0.smithi114.stdout: 1U 2024-09-14T04:31:11.895 INFO:tasks.workunit.client.0.smithi114.stdout: Which is: 1 2024-09-14T04:31:11.895 INFO:tasks.workunit.client.0.smithi114.stdout: keys.size() 2024-09-14T04:31:11.895 INFO:tasks.workunit.client.0.smithi114.stdout: Which is: 3 2024-09-14T04:31:11.895 INFO:tasks.workunit.client.0.smithi114.stdout:[ FAILED ] TestClsRbd.group_snap_set (2 ms) 2024-09-14T04:31:11.895 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-09-14T04:31:11.897 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_snap_list (3 ms) 2024-09-14T04:31:11.897 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-09-14T04:31:11.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:11.915 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (17 ms) 2024-09-14T04:31:11.915 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-09-14T04:31:11.938 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:11.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:12.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:12.063 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:12.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:12.100 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (186 ms) 2024-09-14T04:31:12.100 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-09-14T04:31:12.102 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_snap_remove (2 ms) 2024-09-14T04:31:12.103 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-09-14T04:31:12.104 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (2 ms) 2024-09-14T04:31:12.105 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.trash_methods 2024-09-14T04:31:12.111 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.trash_methods (7 ms) 2024-09-14T04:31:12.111 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.op_features 2024-09-14T04:31:12.118 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.op_features (7 ms) 2024-09-14T04:31:12.118 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.clone_parent 2024-09-14T04:31:12.138 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.clone_parent (20 ms) 2024-09-14T04:31:12.138 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-09-14T04:31:12.146 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.clone_parent_ns (8 ms) 2024-09-14T04:31:12.146 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.clone_child 2024-09-14T04:31:12.157 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.clone_child (11 ms) 2024-09-14T04:31:12.157 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-09-14T04:31:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:11 smithi160 ceph-mon[88407]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:31:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:11 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/211823037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-14T04:31:12.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:11 smithi160 ceph-mon[88407]: osdmap e438: 8 total, 8 up, 5 in 2024-09-14T04:31:12.163 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.namespace_methods (6 ms) 2024-09-14T04:31:12.164 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.migration 2024-09-14T04:31:12.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:12.173 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.migration (10 ms) 2024-09-14T04:31:12.174 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.migration_v1 2024-09-14T04:31:12.186 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.migration_v1 (12 ms) 2024-09-14T04:31:12.186 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-09-14T04:31:12.190 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:12.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:12.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:12.314 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:12.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:12.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:12.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:12.437 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:12.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:12.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:12.539 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:12.561 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:12.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:12.613 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:31:12.613 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:31:12.613 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:31:12.613 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:31:12.613 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:31:12.613 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:31:12.614 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:31:12.614 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:31:12.614 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:31:12.614 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:31:12.614 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:31:12.614 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:31:12.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:12.684 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:12.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:12.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:12.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4 2024-09-14T04:31:12.807 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:12.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.4' 2024-09-14T04:31:12.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:12.930 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:12.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:13.031 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:13.052 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:13.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:13.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:13.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:12 smithi114 ceph-mon[103029]: pgmap v1634: 125 pgs: 125 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 213 op/s 2024-09-14T04:31:13.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:12 smithi114 ceph-mon[106969]: pgmap v1634: 125 pgs: 125 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 213 op/s 2024-09-14T04:31:13.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:13.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:12 smithi160 ceph-mon[88407]: pgmap v1634: 125 pgs: 125 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 213 op/s 2024-09-14T04:31:13.176 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:13.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:13.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:31:13.283 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:31:13.283 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-14T04:31:13.299 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:13.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:13.401 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:13.423 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:13.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:13.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:13.547 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:13.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:13.550 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:13.551 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-14T04:31:13.551 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-09-14T04:31:13.551 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-09-14T04:31:13.552 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-14T04:31:13.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:13.670 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:13.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:13.766 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1581 ms) 2024-09-14T04:31:13.767 INFO:tasks.workunit.client.0.smithi114.stdout:[ RUN ] TestClsRbd.sparsify 2024-09-14T04:31:13.772 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:13.795 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:13.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:13.807 INFO:tasks.workunit.client.0.smithi114.stdout:[ OK ] TestClsRbd.sparsify (41 ms) 2024-09-14T04:31:13.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:13 smithi114 ceph-mon[103029]: osdmap e439: 8 total, 8 up, 5 in 2024-09-14T04:31:13.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:13 smithi114 ceph-mon[106969]: osdmap e439: 8 total, 8 up, 5 in 2024-09-14T04:31:13.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3 2024-09-14T04:31:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:13.920 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:13.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.3' 2024-09-14T04:31:14.021 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:14.042 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:14.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:14.143 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:14.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:13 smithi160 ceph-mon[88407]: osdmap e439: 8 total, 8 up, 5 in 2024-09-14T04:31:14.164 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:14.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:14.265 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5 2024-09-14T04:31:14.286 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-14T04:31:14.292 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:14.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.5' 2024-09-14T04:31:14.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2 2024-09-14T04:31:14.415 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:14.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.2' 2024-09-14T04:31:14.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-14T04:31:14.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1 2024-09-14T04:31:14.545 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:14.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.1' 2024-09-14T04:31:14.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6 2024-09-14T04:31:14.673 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:14.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.6' 2024-09-14T04:31:14.766 INFO:tasks.workunit.client.0.smithi114.stdout:[----------] 70 tests from TestClsRbd (5070 ms total) 2024-09-14T04:31:14.766 INFO:tasks.workunit.client.0.smithi114.stdout: 2024-09-14T04:31:14.766 INFO:tasks.workunit.client.0.smithi114.stdout:[----------] Global test environment tear-down 2024-09-14T04:31:14.766 INFO:tasks.workunit.client.0.smithi114.stdout:[==========] 70 tests from 1 test suite ran. (7449 ms total) 2024-09-14T04:31:14.766 INFO:tasks.workunit.client.0.smithi114.stdout:[ PASSED ] 69 tests. 2024-09-14T04:31:14.766 INFO:tasks.workunit.client.0.smithi114.stdout:[ FAILED ] 1 test, listed below: 2024-09-14T04:31:14.766 INFO:tasks.workunit.client.0.smithi114.stdout:[ FAILED ] TestClsRbd.group_snap_set 2024-09-14T04:31:14.766 INFO:tasks.workunit.client.0.smithi114.stdout: 2024-09-14T04:31:14.766 INFO:tasks.workunit.client.0.smithi114.stdout: 1 FAILED TEST 2024-09-14T04:31:14.775 DEBUG:teuthology.orchestra.run.smithi114:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0 2024-09-14T04:31:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:14.784 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-09-14T04:31:14.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-14T04:31:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:14.798 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: docker: command not found 2024-09-14T04:31:14.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi114 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.0' 2024-09-14T04:31:14.817 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:14 smithi114 ceph-mon[103029]: pgmap v1636: 125 pgs: 125 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.9 MiB/s rd, 945 KiB/s wr, 189 op/s 2024-09-14T04:31:14.817 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:14 smithi114 ceph-mon[103029]: osdmap e440: 8 total, 8 up, 5 in 2024-09-14T04:31:14.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:14 smithi114 ceph-mon[106969]: pgmap v1636: 125 pgs: 125 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.9 MiB/s rd, 945 KiB/s wr, 189 op/s 2024-09-14T04:31:14.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:14 smithi114 ceph-mon[106969]: osdmap e440: 8 total, 8 up, 5 in 2024-09-14T04:31:14.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7 2024-09-14T04:31:14.924 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: docker: command not found 2024-09-14T04:31:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T04:31:14.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi160 with status 1: 'sudo docker kill -s 1 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd.7' 2024-09-14T04:31:15.000 INFO:tasks.thrashosds:joining thrashosds 2024-09-14T04:31:15.114 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:15.150 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:14 smithi160 ceph-mon[88407]: pgmap v1636: 125 pgs: 125 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 1.9 MiB/s rd, 945 KiB/s wr, 189 op/s 2024-09-14T04:31:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:14 smithi160 ceph-mon[88407]: osdmap e440: 8 total, 8 up, 5 in 2024-09-14T04:31:15.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:15 smithi114 ceph-mon[103029]: osdmap e441: 8 total, 8 up, 5 in 2024-09-14T04:31:15.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:15 smithi114 ceph-mon[103029]: pgmap v1639: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:15 smithi114 ceph-mon[106969]: osdmap e441: 8 total, 8 up, 5 in 2024-09-14T04:31:15.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:15 smithi114 ceph-mon[106969]: pgmap v1639: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:15 smithi160 ceph-mon[88407]: osdmap e441: 8 total, 8 up, 5 in 2024-09-14T04:31:16.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:15 smithi160 ceph-mon[88407]: pgmap v1639: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:17.413 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:18.345 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0: {} 2024-09-14T04:31:18.345 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1: {} 2024-09-14T04:31:18.345 INFO:teuthology.orchestra.run.smithi114.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-14T04:31:18.346 INFO:teuthology.orchestra.run.smithi114.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-14T04:31:18.352 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2: {} 2024-09-14T04:31:18.353 INFO:teuthology.orchestra.run.smithi114.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-14T04:31:18.370 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3: {} 2024-09-14T04:31:18.370 INFO:teuthology.orchestra.run.smithi114.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-14T04:31:18.386 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4: {} 2024-09-14T04:31:18.387 INFO:teuthology.orchestra.run.smithi114.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-14T04:31:18.402 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5: {} 2024-09-14T04:31:18.403 INFO:teuthology.orchestra.run.smithi114.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-14T04:31:18.418 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6: {} 2024-09-14T04:31:18.419 INFO:teuthology.orchestra.run.smithi114.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-14T04:31:18.434 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7: {} 2024-09-14T04:31:18.434 INFO:teuthology.orchestra.run.smithi114.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-14T04:31:18.473 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:18 smithi114 ceph-mon[106969]: pgmap v1640: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:18.473 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:18.473 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:18 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2760767684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:31:18.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:18 smithi114 ceph-mon[103029]: pgmap v1640: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:18.474 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:18.474 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:18 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2760767684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:31:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:18 smithi160 ceph-mon[88407]: pgmap v1640: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:18.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:18 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2760767684' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-14T04:31:20.392 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:31:20.393 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:31:20.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:20 smithi114 ceph-mon[103029]: pgmap v1641: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:20 smithi114 ceph-mon[106969]: pgmap v1641: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:20.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:20 smithi160 ceph-mon[88407]: pgmap v1641: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:31:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:31:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:31:20.828 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:31:20.829 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:31:21.860 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:31:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:31:21] "GET /metrics HTTP/1.1" 200 34137 "" "Prometheus/2.51.0" 2024-09-14T04:31:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:31:22.309 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:31:22.310 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-14T04:31:22.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:22 smithi114 ceph-mon[103029]: pgmap v1642: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:22.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:22 smithi114 ceph-mon[106969]: pgmap v1642: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:22.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:22 smithi160 ceph-mon[88407]: pgmap v1642: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:22.797 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-14T04:31:23.317 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:23.353 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:23.672 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is set 2024-09-14T04:31:23.689 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:23.689 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:23.689 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3261581519' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:31:23.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:23.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:23.689 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:23 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3261581519' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:31:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:24.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:23 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3261581519' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:31:25.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:24 smithi114 ceph-mon[103029]: pgmap v1643: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:25.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:24 smithi114 ceph-mon[103029]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:31:25.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:24 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3261581519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:31:25.081 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:24 smithi114 ceph-mon[103029]: osdmap e442: 8 total, 8 up, 5 in 2024-09-14T04:31:25.082 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:24 smithi114 ceph-mon[106969]: pgmap v1643: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:25.082 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:24 smithi114 ceph-mon[106969]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:31:25.082 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:24 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3261581519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:31:25.082 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:24 smithi114 ceph-mon[106969]: osdmap e442: 8 total, 8 up, 5 in 2024-09-14T04:31:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:24 smithi160 ceph-mon[88407]: pgmap v1643: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:24 smithi160 ceph-mon[88407]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:31:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:24 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3261581519' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-14T04:31:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:24 smithi160 ceph-mon[88407]: osdmap e442: 8 total, 8 up, 5 in 2024-09-14T04:31:26.463 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:31:26.464 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:31:26.552 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-14T04:31:26.793 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:26 smithi114 ceph-mon[106969]: pgmap v1645: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:26.793 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:26 smithi114 ceph-mon[103029]: pgmap v1645: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:31:27.117 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:31:27.117 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-14T04:31:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:26 smithi160 ceph-mon[88407]: pgmap v1645: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:28.574 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:31:28.574 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:31:28.867 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:28.902 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:28.996 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:28 smithi114 ceph-mon[103029]: pgmap v1646: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:28.997 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:28 smithi114 ceph-mon[106969]: pgmap v1646: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:28 smithi160 ceph-mon[88407]: pgmap v1646: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:31:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:31:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:31:31.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:30 smithi114 ceph-mon[106969]: pgmap v1647: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:31.055 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:30 smithi114 ceph-mon[103029]: pgmap v1647: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:31.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-14T04:31:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:30 smithi160 ceph-mon[88407]: pgmap v1647: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:31.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:31:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:31:31] "GET /metrics HTTP/1.1" 200 34142 "" "Prometheus/2.51.0" 2024-09-14T04:31:32.014 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:32.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:32 smithi114 ceph-mon[103029]: pgmap v1648: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:32.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:32 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:31:32.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:32 smithi114 ceph-mon[106969]: pgmap v1648: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:32.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:32 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:31:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:32 smithi160 ceph-mon[88407]: pgmap v1648: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:32 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:31:33.699 INFO:teuthology.orchestra.run.smithi114.stderr:noscrub is unset 2024-09-14T04:31:33.821 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:33 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/739792595' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:31:33.821 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:33 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:31:33.821 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:33 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/739792595' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:31:33.821 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:33 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:31:34.041 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:33 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/739792595' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:31:34.041 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:33 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-14T04:31:34.336 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:31:34.336 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:31:34.644 INFO:teuthology.orchestra.run.smithi114.stderr:no valid command found; 10 closest matches: 2024-09-14T04:31:34.644 INFO:teuthology.orchestra.run.smithi114.stderr:0 2024-09-14T04:31:34.644 INFO:teuthology.orchestra.run.smithi114.stderr:1 2024-09-14T04:31:34.645 INFO:teuthology.orchestra.run.smithi114.stderr:2 2024-09-14T04:31:34.645 INFO:teuthology.orchestra.run.smithi114.stderr:abort 2024-09-14T04:31:34.645 INFO:teuthology.orchestra.run.smithi114.stderr:assert 2024-09-14T04:31:34.645 INFO:teuthology.orchestra.run.smithi114.stderr:bench [] [] [] [] 2024-09-14T04:31:34.645 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs debug_inject_read_zeros 2024-09-14T04:31:34.645 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs files list 2024-09-14T04:31:34.645 INFO:teuthology.orchestra.run.smithi114.stderr:bluefs stats 2024-09-14T04:31:34.645 INFO:teuthology.orchestra.run.smithi114.stderr:bluestore allocator dump block 2024-09-14T04:31:34.645 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINVAL: invalid command 2024-09-14T04:31:35.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:34 smithi114 ceph-mon[103029]: pgmap v1649: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:35.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:34 smithi114 ceph-mon[103029]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:31:35.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:34 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:31:35.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:34 smithi114 ceph-mon[103029]: osdmap e443: 8 total, 8 up, 5 in 2024-09-14T04:31:35.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:34 smithi114 ceph-mon[106969]: pgmap v1649: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:35.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:34 smithi114 ceph-mon[106969]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:31:35.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:34 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:31:35.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:34 smithi114 ceph-mon[106969]: osdmap e443: 8 total, 8 up, 5 in 2024-09-14T04:31:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:34 smithi160 ceph-mon[88407]: pgmap v1649: 93 pgs: 93 active+clean; 582 KiB data, 7.0 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:34 smithi160 ceph-mon[88407]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-14T04:31:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:34 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-14T04:31:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:34 smithi160 ceph-mon[88407]: osdmap e443: 8 total, 8 up, 5 in 2024-09-14T04:31:35.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-14T04:31:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-14T04:31:36.159 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-14T04:31:36.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-14T04:31:36.777 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:36 smithi114 ceph-mon[103029]: pgmap v1651: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:36.777 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:36 smithi114 ceph-mon[106969]: pgmap v1651: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:36 smithi160 ceph-mon[88407]: pgmap v1651: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:37.184 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:37.185 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:37.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:37.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:37.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:37 smithi114 ceph-mon[103029]: pgmap v1652: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:37.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:37 smithi114 ceph-mon[106969]: pgmap v1652: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:37.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:37 smithi160 ceph-mon[88407]: pgmap v1652: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:40.374 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:31:40.375 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:31:40.584 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:40 smithi114 ceph-mon[106969]: pgmap v1653: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:40.584 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:40 smithi114 ceph-mon[103029]: pgmap v1653: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:40 smithi160 ceph-mon[88407]: pgmap v1653: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:31:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:31:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:31:41.160 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-14T04:31:41.160 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-14T04:31:41.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd in 7 2024-09-14T04:31:41.232 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-14T04:31:41.319 INFO:teuthology.orchestra.run.smithi114.stderr:nodeep-scrub is unset 2024-09-14T04:31:41.613 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:41 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1704315157' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:31:41.613 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:41 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1704315157' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:31:41.613 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:31:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:31:41] "GET /metrics HTTP/1.1" 200 34139 "" "Prometheus/2.51.0" 2024-09-14T04:31:41.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:41 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1704315157' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-14T04:31:42.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:42 smithi114 ceph-mon[103029]: pgmap v1654: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:42.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:42 smithi114 ceph-mon[103029]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:31:42.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:42 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:31:42.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1704315157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:31:42.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:42 smithi114 ceph-mon[103029]: osdmap e444: 8 total, 8 up, 5 in 2024-09-14T04:31:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:42 smithi114 ceph-mon[106969]: pgmap v1654: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:42 smithi114 ceph-mon[106969]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:31:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:42 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:31:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1704315157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:31:42.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:42 smithi114 ceph-mon[106969]: osdmap e444: 8 total, 8 up, 5 in 2024-09-14T04:31:42.634 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:42 smithi160 ceph-mon[88407]: pgmap v1654: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:42 smithi160 ceph-mon[88407]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-14T04:31:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:42 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:31:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1704315157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-14T04:31:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:42 smithi160 ceph-mon[88407]: osdmap e444: 8 total, 8 up, 5 in 2024-09-14T04:31:43.368 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:44.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:44 smithi114 ceph-mon[103029]: pgmap v1656: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:44.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:44 smithi114 ceph-mon[106969]: pgmap v1656: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:44.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:44 smithi160 ceph-mon[88407]: pgmap v1656: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:45.750 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:45 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:31:45.374+0000 7f3162f23640 -1 mon.a@0(leader).osd e444 definitely_dead 0 2024-09-14T04:31:46.333 INFO:teuthology.orchestra.run.smithi114.stderr:marked in osd.7. 2024-09-14T04:31:46.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:46 smithi114 ceph-mon[103029]: pgmap v1657: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:46.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1329787006' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-14T04:31:46.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:46 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-14T04:31:46.430 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:46 smithi114 ceph-mon[106969]: pgmap v1657: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:46.430 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1329787006' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-14T04:31:46.430 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:46 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-14T04:31:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:46 smithi160 ceph-mon[88407]: pgmap v1657: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1329787006' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-14T04:31:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:46 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-14T04:31:47.076 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-14T04:31:47.076 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-14T04:31:47.076 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-14T04:31:47.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd in 1 2024-09-14T04:31:47.535 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:31:47.535 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:31:47.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:47 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-14T04:31:47.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:47 smithi114 ceph-mon[103029]: osdmap e445: 8 total, 8 up, 6 in 2024-09-14T04:31:47.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:47 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-14T04:31:47.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:47 smithi114 ceph-mon[106969]: osdmap e445: 8 total, 8 up, 6 in 2024-09-14T04:31:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:47 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-14T04:31:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:47 smithi160 ceph-mon[88407]: osdmap e445: 8 total, 8 up, 6 in 2024-09-14T04:31:48.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:48 smithi114 ceph-mon[103029]: pgmap v1659: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:48.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:48 smithi114 ceph-mon[103029]: osdmap e446: 8 total, 8 up, 6 in 2024-09-14T04:31:48.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:48 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:48.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:48 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:48.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:48 smithi114 ceph-mon[106969]: pgmap v1659: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:48.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:48 smithi114 ceph-mon[106969]: osdmap e446: 8 total, 8 up, 6 in 2024-09-14T04:31:48.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:48 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:48.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:48 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:48 smithi160 ceph-mon[88407]: pgmap v1659: 93 pgs: 93 active+clean; 582 KiB data, 6.9 GiB used, 440 GiB / 447 GiB avail 2024-09-14T04:31:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:48 smithi160 ceph-mon[88407]: osdmap e446: 8 total, 8 up, 6 in 2024-09-14T04:31:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:48 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:48.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:48 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:48.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-14T04:31:48.816 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:49.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:49 smithi114 ceph-mon[103029]: 1.e scrub starts 2024-09-14T04:31:49.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:49 smithi114 ceph-mon[103029]: 1.e scrub ok 2024-09-14T04:31:49.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:49 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:31:49.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:49 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:31:49.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:49 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:49.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:49 smithi114 ceph-mon[106969]: 1.e scrub starts 2024-09-14T04:31:49.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:49 smithi114 ceph-mon[106969]: 1.e scrub ok 2024-09-14T04:31:49.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:49 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:31:49.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:49 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:31:49.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:49 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:49 smithi160 ceph-mon[88407]: 1.e scrub starts 2024-09-14T04:31:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:49 smithi160 ceph-mon[88407]: 1.e scrub ok 2024-09-14T04:31:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:49 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:31:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:49 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:31:49.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:49 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:50.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:50 smithi114 ceph-mon[103029]: pgmap v1661: 93 pgs: 93 active+clean; 582 KiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-14T04:31:50.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:50 smithi114 ceph-mon[106969]: pgmap v1661: 93 pgs: 93 active+clean; 582 KiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-14T04:31:50.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:50 smithi160 ceph-mon[88407]: pgmap v1661: 93 pgs: 93 active+clean; 582 KiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-14T04:31:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:31:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:31:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:31:50.974 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:50 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:31:50.691+0000 7f3162f23640 -1 mon.a@0(leader).osd e446 definitely_dead 0 2024-09-14T04:31:51.357 INFO:teuthology.orchestra.run.smithi114.stderr:marked in osd.1. 2024-09-14T04:31:51.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:51 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3342499589' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-14T04:31:51.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:51 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-14T04:31:51.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:51 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3342499589' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-14T04:31:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:51 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-14T04:31:51.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:31:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:31:51] "GET /metrics HTTP/1.1" 200 34139 "" "Prometheus/2.51.0" 2024-09-14T04:31:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:51 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3342499589' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-14T04:31:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:51 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-14T04:31:52.458 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:52 smithi114 ceph-mon[106969]: pgmap v1662: 93 pgs: 93 active+clean; 582 KiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 26 B/s, 1 keys/s, 1 objects/s recovering 2024-09-14T04:31:52.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:52 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-14T04:31:52.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:52 smithi114 ceph-mon[106969]: osdmap e447: 8 total, 8 up, 7 in 2024-09-14T04:31:52.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:52 smithi114 ceph-mon[103029]: pgmap v1662: 93 pgs: 93 active+clean; 582 KiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 26 B/s, 1 keys/s, 1 objects/s recovering 2024-09-14T04:31:52.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:52 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-14T04:31:52.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:52 smithi114 ceph-mon[103029]: osdmap e447: 8 total, 8 up, 7 in 2024-09-14T04:31:52.460 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:31:52.460 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:31:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:52 smithi160 ceph-mon[88407]: pgmap v1662: 93 pgs: 93 active+clean; 582 KiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 26 B/s, 1 keys/s, 1 objects/s recovering 2024-09-14T04:31:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:52 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-14T04:31:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:52 smithi160 ceph-mon[88407]: osdmap e447: 8 total, 8 up, 7 in 2024-09-14T04:31:53.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-14T04:31:53.582 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-14T04:31:53.582 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-14T04:31:53.582 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-14T04:31:53.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd in 0 2024-09-14T04:31:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:53 smithi114 ceph-mon[103029]: osdmap e448: 8 total, 8 up, 7 in 2024-09-14T04:31:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:53 smithi114 ceph-mon[103029]: 1.b scrub starts 2024-09-14T04:31:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:53 smithi114 ceph-mon[103029]: 1.b scrub ok 2024-09-14T04:31:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:53 smithi114 ceph-mon[106969]: osdmap e448: 8 total, 8 up, 7 in 2024-09-14T04:31:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:53 smithi114 ceph-mon[106969]: 1.b scrub starts 2024-09-14T04:31:53.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:53 smithi114 ceph-mon[106969]: 1.b scrub ok 2024-09-14T04:31:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:53 smithi160 ceph-mon[88407]: osdmap e448: 8 total, 8 up, 7 in 2024-09-14T04:31:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:31:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:31:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:53 smithi160 ceph-mon[88407]: 1.b scrub starts 2024-09-14T04:31:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:53 smithi160 ceph-mon[88407]: 1.b scrub ok 2024-09-14T04:31:53.715 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:54.510 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:54.907 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[106969]: pgmap v1665: 93 pgs: 93 active+clean; 617 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 4 op/s; 5.9 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-14T04:31:54.907 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[106969]: 1.3 scrub starts 2024-09-14T04:31:54.908 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[106969]: 1.3 scrub ok 2024-09-14T04:31:54.908 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 7]} : dispatch 2024-09-14T04:31:54.908 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 7]} : dispatch 2024-09-14T04:31:54.908 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]} : dispatch 2024-09-14T04:31:54.908 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-14T04:31:54.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[103029]: pgmap v1665: 93 pgs: 93 active+clean; 617 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 4 op/s; 5.9 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-14T04:31:54.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[103029]: 1.3 scrub starts 2024-09-14T04:31:54.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[103029]: 1.3 scrub ok 2024-09-14T04:31:54.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 7]} : dispatch 2024-09-14T04:31:54.908 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 7]} : dispatch 2024-09-14T04:31:54.909 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]} : dispatch 2024-09-14T04:31:54.909 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:54 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-14T04:31:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:54 smithi160 ceph-mon[88407]: pgmap v1665: 93 pgs: 93 active+clean; 617 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 4 op/s; 5.9 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-14T04:31:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:54 smithi160 ceph-mon[88407]: 1.3 scrub starts 2024-09-14T04:31:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:54 smithi160 ceph-mon[88407]: 1.3 scrub ok 2024-09-14T04:31:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 7]} : dispatch 2024-09-14T04:31:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 7]} : dispatch 2024-09-14T04:31:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]} : dispatch 2024-09-14T04:31:55.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:54 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-14T04:31:55.696 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 7]}]': finished 2024-09-14T04:31:55.696 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 7]}]': finished 2024-09-14T04:31:55.696 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-09-14T04:31:55.696 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-14T04:31:55.697 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[103029]: osdmap e449: 8 total, 8 up, 7 in 2024-09-14T04:31:55.697 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[103029]: osdmap e450: 8 total, 8 up, 7 in 2024-09-14T04:31:55.697 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 7]}]': finished 2024-09-14T04:31:55.697 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 7]}]': finished 2024-09-14T04:31:55.697 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-09-14T04:31:55.697 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-14T04:31:55.697 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[106969]: osdmap e449: 8 total, 8 up, 7 in 2024-09-14T04:31:55.697 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:55 smithi114 ceph-mon[106969]: osdmap e450: 8 total, 8 up, 7 in 2024-09-14T04:31:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 7]}]': finished 2024-09-14T04:31:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 7]}]': finished 2024-09-14T04:31:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-09-14T04:31:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-14T04:31:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:55 smithi160 ceph-mon[88407]: osdmap e449: 8 total, 8 up, 7 in 2024-09-14T04:31:56.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:55 smithi160 ceph-mon[88407]: osdmap e450: 8 total, 8 up, 7 in 2024-09-14T04:31:56.174 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:31:56.174 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:31:56.761 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:56 smithi114 ceph-mon[103029]: pgmap v1667: 93 pgs: 3 peering, 90 active+clean; 617 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 4 op/s; 5.9 KiB/s, 1 objects/s recovering 2024-09-14T04:31:56.762 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:56 smithi114 ceph-mon[106969]: pgmap v1667: 93 pgs: 3 peering, 90 active+clean; 617 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 4 op/s; 5.9 KiB/s, 1 objects/s recovering 2024-09-14T04:31:56.785 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-14T04:31:57.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:56 smithi160 ceph-mon[88407]: pgmap v1667: 93 pgs: 3 peering, 90 active+clean; 617 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 4 op/s; 5.9 KiB/s, 1 objects/s recovering 2024-09-14T04:31:57.311 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:31:58.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:58 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:31:58.110+0000 7f3162f23640 -1 mon.a@0(leader).osd e450 definitely_dead 0 2024-09-14T04:31:58.701 INFO:teuthology.orchestra.run.smithi114.stderr:marked in osd.0. 2024-09-14T04:31:58.816 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:58 smithi114 ceph-mon[103029]: pgmap v1669: 93 pgs: 3 peering, 90 active+clean; 617 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 7.6 KiB/s rd, 17 KiB/s wr, 4 op/s; 5.9 KiB/s, 0 objects/s recovering 2024-09-14T04:31:58.816 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:58 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3716299534' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-14T04:31:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:58 smithi114 ceph-mon[106969]: pgmap v1669: 93 pgs: 3 peering, 90 active+clean; 617 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 7.6 KiB/s rd, 17 KiB/s wr, 4 op/s; 5.9 KiB/s, 0 objects/s recovering 2024-09-14T04:31:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:58 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3716299534' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-14T04:31:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:58 smithi160 ceph-mon[88407]: pgmap v1669: 93 pgs: 3 peering, 90 active+clean; 617 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 7.6 KiB/s rd, 17 KiB/s wr, 4 op/s; 5.9 KiB/s, 0 objects/s recovering 2024-09-14T04:31:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:58 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3716299534' entity='client.admin' cmd={"prefix": "osd in", "ids": ["0"]} : dispatch 2024-09-14T04:31:59.754 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-14T04:31:59.755 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-14T04:31:59.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 21 2024-09-14T04:32:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:59 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3716299534' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-14T04:32:00.059 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:31:59 smithi160 ceph-mon[88407]: osdmap e451: 8 total, 8 up, 8 in 2024-09-14T04:32:00.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:59 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3716299534' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-14T04:32:00.077 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:31:59 smithi114 ceph-mon[103029]: osdmap e451: 8 total, 8 up, 8 in 2024-09-14T04:32:00.077 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:59 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3716299534' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-14T04:32:00.077 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:31:59 smithi114 ceph-mon[106969]: osdmap e451: 8 total, 8 up, 8 in 2024-09-14T04:32:00.336 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:32:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:32:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:32:00.345 INFO:teuthology.orchestra.run.smithi114.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:00.346 INFO:teuthology.orchestra.run.smithi114.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:00.607 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:01.111 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:00 smithi114 ceph-mon[106969]: pgmap v1671: 93 pgs: 8 peering, 85 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-14T04:32:01.111 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:00 smithi114 ceph-mon[106969]: osdmap e452: 8 total, 8 up, 8 in 2024-09-14T04:32:01.112 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:00 smithi114 ceph-mon[103029]: pgmap v1671: 93 pgs: 8 peering, 85 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-14T04:32:01.112 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:00 smithi114 ceph-mon[103029]: osdmap e452: 8 total, 8 up, 8 in 2024-09-14T04:32:01.137 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-14T04:32:01.157 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:00 smithi160 ceph-mon[88407]: pgmap v1671: 93 pgs: 8 peering, 85 active+clean; 582 KiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-14T04:32:01.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:00 smithi160 ceph-mon[88407]: osdmap e452: 8 total, 8 up, 8 in 2024-09-14T04:32:01.404 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:01.665 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:32:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:32:01] "GET /metrics HTTP/1.1" 200 34155 "" "Prometheus/2.51.0" 2024-09-14T04:32:02.951 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:02 smithi160 ceph-mon[88407]: pgmap v1673: 93 pgs: 11 peering, 82 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-09-14T04:32:03.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:02 smithi114 ceph-mon[103029]: pgmap v1673: 93 pgs: 11 peering, 82 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-09-14T04:32:03.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:02 smithi114 ceph-mon[106969]: pgmap v1673: 93 pgs: 11 peering, 82 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-09-14T04:32:03.186 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:03.186 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 1 pgp_num to 21 2024-09-14T04:32:03.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:03 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3777303218' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-14T04:32:03.800 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:03 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-14T04:32:03.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-14T04:32:04.109 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:03 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3777303218' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-14T04:32:04.109 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:03 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-14T04:32:04.110 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:03 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3777303218' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-14T04:32:04.110 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:03 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-14T04:32:04.119 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:04.523 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-14T04:32:04.774 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:04.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:04 smithi114 ceph-mon[106969]: pgmap v1674: 93 pgs: 11 peering, 82 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 38 KiB/s, 0 objects/s recovering 2024-09-14T04:32:04.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:04 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-14T04:32:04.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:04 smithi114 ceph-mon[106969]: osdmap e453: 8 total, 8 up, 8 in 2024-09-14T04:32:04.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:04 smithi114 ceph-mon[103029]: pgmap v1674: 93 pgs: 11 peering, 82 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 38 KiB/s, 0 objects/s recovering 2024-09-14T04:32:04.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:04 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-14T04:32:04.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:04 smithi114 ceph-mon[103029]: osdmap e453: 8 total, 8 up, 8 in 2024-09-14T04:32:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:04 smithi160 ceph-mon[88407]: pgmap v1674: 93 pgs: 11 peering, 82 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 38 KiB/s, 0 objects/s recovering 2024-09-14T04:32:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:04 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-14T04:32:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:04 smithi160 ceph-mon[88407]: osdmap e453: 8 total, 8 up, 8 in 2024-09-14T04:32:05.978 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:05.978 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:06.454 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a: {} 2024-09-14T04:32:06.455 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b: {} 2024-09-14T04:32:06.455 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c: {} 2024-09-14T04:32:06.455 INFO:teuthology.orchestra.run.smithi114.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-14T04:32:06.455 INFO:teuthology.orchestra.run.smithi114.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-14T04:32:06.455 INFO:teuthology.orchestra.run.smithi114.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-14T04:32:06.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-14T04:32:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:06 smithi160 ceph-mon[88407]: pgmap v1676: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 49 KiB/s, 1 objects/s recovering 2024-09-14T04:32:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/882338993' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-14T04:32:06.925 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:07.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:06 smithi114 ceph-mon[103029]: pgmap v1676: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 49 KiB/s, 1 objects/s recovering 2024-09-14T04:32:07.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/882338993' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-14T04:32:07.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:06 smithi114 ceph-mon[106969]: pgmap v1676: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 49 KiB/s, 1 objects/s recovering 2024-09-14T04:32:07.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/882338993' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-14T04:32:07.352 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-14T04:32:07.596 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:08.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:07 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:08.005 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:07 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:08.130 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:07 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:08.726 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:08.727 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:08.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:08 smithi114 ceph-mon[106969]: pgmap v1677: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 40 KiB/s, 1 objects/s recovering 2024-09-14T04:32:08.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:08 smithi114 ceph-mon[103029]: pgmap v1677: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 40 KiB/s, 1 objects/s recovering 2024-09-14T04:32:09.096 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:08 smithi160 ceph-mon[88407]: pgmap v1677: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 40 KiB/s, 1 objects/s recovering 2024-09-14T04:32:09.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-14T04:32:09.650 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:09.733 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.0 to 1 (10000) 2024-09-14T04:32:10.041 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:09 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3346118777' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:32:10.041 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:09 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:32:10.099 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:09 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3346118777' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:32:10.100 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:09 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:32:10.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:09 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3346118777' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:32:10.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:09 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:32:10.407 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:32:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:32:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:32:10.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-14T04:32:10.732 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:10.786 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:10 smithi160 ceph-mon[88407]: pgmap v1678: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 34 KiB/s, 1 objects/s recovering 2024-09-14T04:32:10.786 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:10 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:32:10.786 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:10 smithi160 ceph-mon[88407]: osdmap e454: 8 total, 8 up, 8 in 2024-09-14T04:32:11.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:10 smithi114 ceph-mon[103029]: pgmap v1678: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 34 KiB/s, 1 objects/s recovering 2024-09-14T04:32:11.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:10 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:32:11.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:10 smithi114 ceph-mon[103029]: osdmap e454: 8 total, 8 up, 8 in 2024-09-14T04:32:11.003 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:10 smithi114 ceph-mon[106969]: pgmap v1678: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 34 KiB/s, 1 objects/s recovering 2024-09-14T04:32:11.004 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:10 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:32:11.004 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:10 smithi114 ceph-mon[106969]: osdmap e454: 8 total, 8 up, 8 in 2024-09-14T04:32:11.481 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:11.481 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:11.591 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:32:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:32:11] "GET /metrics HTTP/1.1" 200 34170 "" "Prometheus/2.51.0" 2024-09-14T04:32:12.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-14T04:32:12.353 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:13.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:12 smithi114 ceph-mon[103029]: pgmap v1680: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 73 B/s, 0 objects/s recovering 2024-09-14T04:32:13.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:12 smithi114 ceph-mon[106969]: pgmap v1680: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 73 B/s, 0 objects/s recovering 2024-09-14T04:32:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:12 smithi160 ceph-mon[88407]: pgmap v1680: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 73 B/s, 0 objects/s recovering 2024-09-14T04:32:13.746 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-14T04:32:13.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:13 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3859077630' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:32:13.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:13 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:14.041 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:14.041 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:14.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:13 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3859077630' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:32:14.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:13 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:14.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:13 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3859077630' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-14T04:32:14.121 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:13 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:14.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-14T04:32:14.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-14T04:32:14.804 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:14.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:14 smithi114 ceph-mon[103029]: pgmap v1681: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 61 B/s, 0 objects/s recovering 2024-09-14T04:32:14.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:14 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3859077630' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:32:14.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:14 smithi114 ceph-mon[103029]: osdmap e455: 8 total, 8 up, 8 in 2024-09-14T04:32:14.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:14 smithi114 ceph-mon[106969]: pgmap v1681: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 61 B/s, 0 objects/s recovering 2024-09-14T04:32:14.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:14 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3859077630' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:32:14.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:14 smithi114 ceph-mon[106969]: osdmap e455: 8 total, 8 up, 8 in 2024-09-14T04:32:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:14 smithi160 ceph-mon[88407]: pgmap v1681: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 61 B/s, 0 objects/s recovering 2024-09-14T04:32:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:14 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3859077630' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-14T04:32:14.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:14 smithi160 ceph-mon[88407]: osdmap e455: 8 total, 8 up, 8 in 2024-09-14T04:32:14.922 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:16.030 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:15 smithi160 ceph-mon[88407]: osdmap e456: 8 total, 8 up, 8 in 2024-09-14T04:32:16.040 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:15 smithi114 ceph-mon[103029]: osdmap e456: 8 total, 8 up, 8 in 2024-09-14T04:32:16.040 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:15 smithi114 ceph-mon[106969]: osdmap e456: 8 total, 8 up, 8 in 2024-09-14T04:32:16.717 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:16.717 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:16.769 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.1 to 1 (10000) 2024-09-14T04:32:16.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:16 smithi114 ceph-mon[106969]: pgmap v1684: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:16.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:16 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/478827109' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:32:16.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:16 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:32:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:16 smithi114 ceph-mon[103029]: pgmap v1684: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:16 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/478827109' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:32:16.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:16 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:32:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:16 smithi160 ceph-mon[88407]: pgmap v1684: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:16 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/478827109' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:32:16.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:16 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:32:17.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-14T04:32:17.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-14T04:32:17.725 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:17.765 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:18.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:17 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:32:18.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:17 smithi114 ceph-mon[103029]: osdmap e457: 8 total, 8 up, 8 in 2024-09-14T04:32:18.027 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:17 smithi114 ceph-mon[103029]: pgmap v1686: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:18.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:17 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:32:18.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:17 smithi114 ceph-mon[106969]: osdmap e457: 8 total, 8 up, 8 in 2024-09-14T04:32:18.027 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:17 smithi114 ceph-mon[106969]: pgmap v1686: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:18.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:17 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:32:18.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:17 smithi160 ceph-mon[88407]: osdmap e457: 8 total, 8 up, 8 in 2024-09-14T04:32:18.035 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:17 smithi160 ceph-mon[88407]: pgmap v1686: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:19.579 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:19.579 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:20.262 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-14T04:32:20.321 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-14T04:32:20.517 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:20.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:20 smithi114 ceph-mon[103029]: pgmap v1687: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:20.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:20 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3324622673' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:32:20.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:20 smithi114 ceph-mon[106969]: pgmap v1687: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:20.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:20 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3324622673' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:32:20.640 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:20 smithi160 ceph-mon[88407]: pgmap v1687: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:20.640 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:20 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3324622673' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-14T04:32:20.640 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:32:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:32:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:32:21.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-14T04:32:21.285 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:21.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:21 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3324622673' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:32:21.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:21 smithi114 ceph-mon[103029]: osdmap e458: 8 total, 8 up, 8 in 2024-09-14T04:32:21.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:32:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:32:21] "GET /metrics HTTP/1.1" 200 34170 "" "Prometheus/2.51.0" 2024-09-14T04:32:21.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:21 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3324622673' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:32:21.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:21 smithi114 ceph-mon[106969]: osdmap e458: 8 total, 8 up, 8 in 2024-09-14T04:32:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:21 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3324622673' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-14T04:32:21.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:21 smithi160 ceph-mon[88407]: osdmap e458: 8 total, 8 up, 8 in 2024-09-14T04:32:22.264 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:22.265 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:22.538 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:22 smithi114 ceph-mon[103029]: pgmap v1689: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:22.539 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:22 smithi114 ceph-mon[106969]: pgmap v1689: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:22.591 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:22 smithi160 ceph-mon[88407]: pgmap v1689: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:22.915 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-14T04:32:23.165 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.2 to 1 (10000) 2024-09-14T04:32:23.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:23 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:23.535 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:23 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3222582055' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:32:23.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:23 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:23.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:23 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3222582055' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:32:23.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:23 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:23.658 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:23 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3222582055' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:32:23.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-14T04:32:24.215 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:24.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:24 smithi160 ceph-mon[88407]: pgmap v1690: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:24.601 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:24 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3222582055' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:32:24.602 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:24 smithi160 ceph-mon[88407]: osdmap e459: 8 total, 8 up, 8 in 2024-09-14T04:32:24.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:24 smithi114 ceph-mon[103029]: pgmap v1690: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:24.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:24 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3222582055' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:32:24.616 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:24 smithi114 ceph-mon[103029]: osdmap e459: 8 total, 8 up, 8 in 2024-09-14T04:32:24.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:24 smithi114 ceph-mon[106969]: pgmap v1690: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:24.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:24 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3222582055' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:32:24.616 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:24 smithi114 ceph-mon[106969]: osdmap e459: 8 total, 8 up, 8 in 2024-09-14T04:32:24.834 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:24.834 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:25.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-14T04:32:25.665 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:26.341 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-14T04:32:26.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:26 smithi114 ceph-mon[106969]: pgmap v1692: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:26.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:26 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3725885614' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:32:26.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:26 smithi114 ceph-mon[103029]: pgmap v1692: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:26.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:26 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3725885614' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:32:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:26 smithi160 ceph-mon[88407]: pgmap v1692: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:26.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:26 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3725885614' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-14T04:32:27.071 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-14T04:32:27.319 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:27.352 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:27 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3725885614' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:32:27.353 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:27 smithi114 ceph-mon[106969]: osdmap e460: 8 total, 8 up, 8 in 2024-09-14T04:32:27.451 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:27.451 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:27.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:27 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3725885614' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:32:27.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:27 smithi114 ceph-mon[103029]: osdmap e460: 8 total, 8 up, 8 in 2024-09-14T04:32:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:27 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3725885614' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-14T04:32:27.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:27 smithi160 ceph-mon[88407]: osdmap e460: 8 total, 8 up, 8 in 2024-09-14T04:32:28.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-14T04:32:28.384 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:28.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:28 smithi160 ceph-mon[88407]: pgmap v1694: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:28.603 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:28 smithi114 ceph-mon[103029]: pgmap v1694: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:28.603 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:28 smithi114 ceph-mon[106969]: pgmap v1694: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:29.356 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.3 to 1 (10000) 2024-09-14T04:32:29.593 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:29 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/588824454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:32:29.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:29 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/588824454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:32:29.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:29 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/588824454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:32:29.978 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-14T04:32:30.113 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:30.113 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:30.222 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:30.392 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:30 smithi160 ceph-mon[88407]: pgmap v1695: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:30.393 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:30 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/588824454' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:32:30.393 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:30 smithi160 ceph-mon[88407]: osdmap e461: 8 total, 8 up, 8 in 2024-09-14T04:32:30.393 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:32:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:32:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:32:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:30 smithi114 ceph-mon[106969]: pgmap v1695: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:30 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/588824454' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:32:30.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:30 smithi114 ceph-mon[106969]: osdmap e461: 8 total, 8 up, 8 in 2024-09-14T04:32:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:30 smithi114 ceph-mon[103029]: pgmap v1695: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:30 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/588824454' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:32:30.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:30 smithi114 ceph-mon[103029]: osdmap e461: 8 total, 8 up, 8 in 2024-09-14T04:32:30.769 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-14T04:32:31.034 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.b/config 2024-09-14T04:32:31.552 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:32:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:32:31] "GET /metrics HTTP/1.1" 200 34156 "" "Prometheus/2.51.0" 2024-09-14T04:32:32.366 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-14T04:32:32.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:32 smithi114 ceph-mon[103029]: pgmap v1697: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:32.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:32 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/770983046' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:32:32.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:32 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:32:32.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:32 smithi114 ceph-mon[106969]: pgmap v1697: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:32.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:32 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/770983046' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:32:32.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:32 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:32:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:32 smithi160 ceph-mon[88407]: pgmap v1697: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:32 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/770983046' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:32:32.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:32 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-14T04:32:32.953 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-14T04:32:32.953 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-14T04:32:33.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-14T04:32:33.395 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:33.506 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:33 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:32:33.506 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:33 smithi160 ceph-mon[88407]: osdmap e462: 8 total, 8 up, 8 in 2024-09-14T04:32:33.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:33 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:32:33.516 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:33 smithi114 ceph-mon[103029]: osdmap e462: 8 total, 8 up, 8 in 2024-09-14T04:32:33.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:33 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-14T04:32:33.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:33 smithi114 ceph-mon[106969]: osdmap e462: 8 total, 8 up, 8 in 2024-09-14T04:32:34.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:34 smithi114 ceph-mon[103029]: pgmap v1699: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:34.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:34 smithi114 ceph-mon[106969]: pgmap v1699: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:34.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:34 smithi160 ceph-mon[88407]: pgmap v1699: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:36.386 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.5 to 1 (10000) 2024-09-14T04:32:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:36 smithi160 ceph-mon[88407]: pgmap v1700: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:36 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2809745603' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:32:36.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:36 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:32:36.767 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:36 smithi114 ceph-mon[106969]: pgmap v1700: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:36.768 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:36 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2809745603' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:32:36.768 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:36 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:32:36.768 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:36 smithi114 ceph-mon[103029]: pgmap v1700: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:36.768 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:36 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2809745603' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:32:36.768 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:36 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:32:37.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-14T04:32:37.360 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:37.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:37 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:32:37.640 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:37 smithi114 ceph-mon[106969]: osdmap e463: 8 total, 8 up, 8 in 2024-09-14T04:32:37.640 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:37 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:32:37.641 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:37 smithi114 ceph-mon[103029]: osdmap e463: 8 total, 8 up, 8 in 2024-09-14T04:32:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:37 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:32:37.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:37 smithi160 ceph-mon[88407]: osdmap e463: 8 total, 8 up, 8 in 2024-09-14T04:32:38.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:38 smithi114 ceph-mon[103029]: pgmap v1702: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:38.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:38 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:38.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:38 smithi114 ceph-mon[106969]: pgmap v1702: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:38.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:38 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:38 smithi160 ceph-mon[88407]: pgmap v1702: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:38.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:38 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:39.396 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-14T04:32:39.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:39 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1115520261' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:32:39.748 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:39 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1115520261' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:32:39.749 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:39 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1115520261' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-14T04:32:40.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-14T04:32:40.350 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:40.627 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:40 smithi114 ceph-mon[106969]: pgmap v1703: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:40.628 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1115520261' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:32:40.628 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:40 smithi114 ceph-mon[106969]: osdmap e464: 8 total, 8 up, 8 in 2024-09-14T04:32:40.628 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:40 smithi114 ceph-mon[103029]: pgmap v1703: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:40.628 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1115520261' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:32:40.628 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:40 smithi114 ceph-mon[103029]: osdmap e464: 8 total, 8 up, 8 in 2024-09-14T04:32:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:32:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:32:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:32:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:40 smithi160 ceph-mon[88407]: pgmap v1703: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1115520261' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-14T04:32:40.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:40 smithi160 ceph-mon[88407]: osdmap e464: 8 total, 8 up, 8 in 2024-09-14T04:32:41.558 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:32:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:32:41] "GET /metrics HTTP/1.1" 200 34158 "" "Prometheus/2.51.0" 2024-09-14T04:32:42.404 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.6 to 1 (10000) 2024-09-14T04:32:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:42 smithi160 ceph-mon[88407]: pgmap v1705: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2350538574' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:42.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:42 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:42.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:42 smithi114 ceph-mon[103029]: pgmap v1705: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:42.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2350538574' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:42.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:42 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:42.800 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:42 smithi114 ceph-mon[106969]: pgmap v1705: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:42.800 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2350538574' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:42.800 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:42 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:43.258 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-14T04:32:43.511 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:43.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:43 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:32:43.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:43 smithi114 ceph-mon[103029]: osdmap e465: 8 total, 8 up, 8 in 2024-09-14T04:32:43.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:43 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:32:43.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:43 smithi114 ceph-mon[106969]: osdmap e465: 8 total, 8 up, 8 in 2024-09-14T04:32:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:43 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:32:43.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:43 smithi160 ceph-mon[88407]: osdmap e465: 8 total, 8 up, 8 in 2024-09-14T04:32:44.716 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:44 smithi114 ceph-mon[106969]: pgmap v1707: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:44.716 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:44 smithi114 ceph-mon[103029]: pgmap v1707: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:44.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:44 smithi160 ceph-mon[88407]: pgmap v1707: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:46.427 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-14T04:32:46.807 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:46 smithi114 ceph-mon[106969]: pgmap v1708: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:46.808 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1873871453' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:46.808 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:46 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:46.808 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:46 smithi114 ceph-mon[103029]: pgmap v1708: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:46.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1873871453' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:46.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:46 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:46 smithi160 ceph-mon[88407]: pgmap v1708: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1873871453' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:46.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:46 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-14T04:32:47.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-14T04:32:47.422 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:47.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:47 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:32:47.699 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:47 smithi114 ceph-mon[106969]: osdmap e466: 8 total, 8 up, 8 in 2024-09-14T04:32:47.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:47 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:32:47.700 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:47 smithi114 ceph-mon[103029]: osdmap e466: 8 total, 8 up, 8 in 2024-09-14T04:32:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:47 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-14T04:32:47.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:47 smithi160 ceph-mon[88407]: osdmap e466: 8 total, 8 up, 8 in 2024-09-14T04:32:48.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:48 smithi114 ceph-mon[103029]: pgmap v1710: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:48.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:48 smithi114 ceph-mon[106969]: pgmap v1710: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:48.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:48 smithi160 ceph-mon[88407]: pgmap v1710: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:49.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:49 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:32:49.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:49 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:32:49.738 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:49 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-14T04:32:50.448 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.7 to 1 (10000) 2024-09-14T04:32:50.500 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:32:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:32:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:32:50.501 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:50 smithi160 ceph-mon[88407]: pgmap v1711: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:50.501 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:50 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/871867862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:50.501 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:50 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:50.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:50 smithi114 ceph-mon[103029]: pgmap v1711: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:50.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:50 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/871867862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:50.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:50 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:50.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:50 smithi114 ceph-mon[106969]: pgmap v1711: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:50.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:50 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/871867862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:50.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:50 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:51.318 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-14T04:32:51.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:51 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:32:51.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:51 smithi114 ceph-mon[103029]: osdmap e467: 8 total, 8 up, 8 in 2024-09-14T04:32:51.621 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:32:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:32:51] "GET /metrics HTTP/1.1" 200 34158 "" "Prometheus/2.51.0" 2024-09-14T04:32:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:51 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:32:51.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:51 smithi114 ceph-mon[106969]: osdmap e467: 8 total, 8 up, 8 in 2024-09-14T04:32:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:51 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:32:51.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:51 smithi160 ceph-mon[88407]: osdmap e467: 8 total, 8 up, 8 in 2024-09-14T04:32:52.087 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:52.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:52 smithi114 ceph-mon[106969]: pgmap v1713: 93 pgs: 3 peering, 90 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:52.492 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:52 smithi114 ceph-mon[106969]: osdmap e468: 8 total, 8 up, 8 in 2024-09-14T04:32:52.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:52 smithi114 ceph-mon[103029]: pgmap v1713: 93 pgs: 3 peering, 90 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:52.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:52 smithi114 ceph-mon[103029]: osdmap e468: 8 total, 8 up, 8 in 2024-09-14T04:32:52.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:52 smithi160 ceph-mon[88407]: pgmap v1713: 93 pgs: 3 peering, 90 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:52.797 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:52 smithi160 ceph-mon[88407]: osdmap e468: 8 total, 8 up, 8 in 2024-09-14T04:32:53.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:53.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:53.731 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:53.731 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:53.731 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:53.732 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:32:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:53.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:54.800 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:54 smithi114 ceph-mon[106969]: pgmap v1715: 93 pgs: 3 peering, 90 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:54.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:54 smithi114 ceph-mon[103029]: pgmap v1715: 93 pgs: 3 peering, 90 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:54.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:54 smithi160 ceph-mon[88407]: pgmap v1715: 93 pgs: 3 peering, 90 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:55.472 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-14T04:32:55.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:55 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2209961784' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:55.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:55 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:55.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:55.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:55 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:55 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2209961784' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:55 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:55.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:55 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:55 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2209961784' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:55 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-14T04:32:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:55.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:55 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:56.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-14T04:32:56.438 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:56.723 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[106969]: pgmap v1716: 93 pgs: 3 peering, 90 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:56.723 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:32:56.723 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[106969]: osdmap e469: 8 total, 8 up, 8 in 2024-09-14T04:32:56.723 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:32:56.723 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:32:56.724 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:56.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[103029]: pgmap v1716: 93 pgs: 3 peering, 90 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:56.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:32:56.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[103029]: osdmap e469: 8 total, 8 up, 8 in 2024-09-14T04:32:56.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:32:56.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:32:56.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:56 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:56 smithi160 ceph-mon[88407]: pgmap v1716: 93 pgs: 3 peering, 90 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:56 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-14T04:32:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:56 smithi160 ceph-mon[88407]: osdmap e469: 8 total, 8 up, 8 in 2024-09-14T04:32:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:56 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-14T04:32:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:56 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-14T04:32:56.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:56 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:32:57.711 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:57 smithi114 ceph-mon[106969]: osdmap e470: 8 total, 8 up, 8 in 2024-09-14T04:32:57.712 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:57 smithi114 ceph-mon[103029]: osdmap e470: 8 total, 8 up, 8 in 2024-09-14T04:32:57.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:57 smithi160 ceph-mon[88407]: osdmap e470: 8 total, 8 up, 8 in 2024-09-14T04:32:58.489 INFO:teuthology.orchestra.run.smithi114.stderr:reweighted osd.4 to 1 (10000) 2024-09-14T04:32:58.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:58 smithi114 ceph-mon[103029]: pgmap v1719: 93 pgs: 1 peering, 92 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:58.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:58 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1232753926' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:32:58.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:58 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:32:58.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:58 smithi114 ceph-mon[106969]: pgmap v1719: 93 pgs: 1 peering, 92 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:58.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:58 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1232753926' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:32:58.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:58 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:32:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:58 smithi160 ceph-mon[88407]: pgmap v1719: 93 pgs: 1 peering, 92 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:32:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:58 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1232753926' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:32:58.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:58 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:32:59.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-14T04:32:59.495 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:59 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:32:59.496 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:32:59 smithi114 ceph-mon[106969]: osdmap e471: 8 total, 8 up, 8 in 2024-09-14T04:32:59.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:59 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:32:59.496 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:32:59 smithi114 ceph-mon[103029]: osdmap e471: 8 total, 8 up, 8 in 2024-09-14T04:32:59.510 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:32:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:59 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:32:59.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:32:59 smithi160 ceph-mon[88407]: osdmap e471: 8 total, 8 up, 8 in 2024-09-14T04:33:00.497 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:33:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:33:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:33:00.671 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:00 smithi114 ceph-mon[106969]: pgmap v1721: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:00.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:00 smithi114 ceph-mon[103029]: pgmap v1721: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:00.766 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:00 smithi160 ceph-mon[88407]: pgmap v1721: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:01.449 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:01 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:33:01] "GET /metrics HTTP/1.1" 200 34142 "" "Prometheus/2.51.0" 2024-09-14T04:33:02.508 INFO:teuthology.orchestra.run.smithi114.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-14T04:33:02.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:02 smithi114 ceph-mon[103029]: pgmap v1722: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:02 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1142676977' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:33:02.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:02 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:33:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:02 smithi114 ceph-mon[106969]: pgmap v1722: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:02.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:02 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1142676977' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:33:02.872 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:02 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:33:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:02 smithi160 ceph-mon[88407]: pgmap v1722: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:02 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1142676977' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:33:02.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:02 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-14T04:33:03.342 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-14T04:33:03.343 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-14T04:33:03.343 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-14T04:33:03.343 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-14T04:33:03.343 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-14T04:33:03.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-14T04:33:03.598 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:03.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:03 smithi114 ceph-mon[106969]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:33:03.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:03 smithi114 ceph-mon[106969]: osdmap e472: 8 total, 8 up, 8 in 2024-09-14T04:33:03.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:03 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:03.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:03 smithi114 ceph-mon[103029]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:33:03.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:03 smithi114 ceph-mon[103029]: osdmap e472: 8 total, 8 up, 8 in 2024-09-14T04:33:03.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:03 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:03.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:03 smithi160 ceph-mon[88407]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-14T04:33:03.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:03 smithi160 ceph-mon[88407]: osdmap e472: 8 total, 8 up, 8 in 2024-09-14T04:33:03.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:03 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:04.837 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:04 smithi114 ceph-mon[106969]: pgmap v1724: 93 pgs: 15 peering, 78 active+clean; 634 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 6 op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-14T04:33:04.837 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:04 smithi114 ceph-mon[106969]: osdmap e473: 8 total, 8 up, 8 in 2024-09-14T04:33:04.837 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:04 smithi114 ceph-mon[106969]: Health check failed: Reduced data availability: 13 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-14T04:33:04.837 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:04 smithi114 ceph-mon[103029]: pgmap v1724: 93 pgs: 15 peering, 78 active+clean; 634 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 6 op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-14T04:33:04.837 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:04 smithi114 ceph-mon[103029]: osdmap e473: 8 total, 8 up, 8 in 2024-09-14T04:33:04.838 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:04 smithi114 ceph-mon[103029]: Health check failed: Reduced data availability: 13 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-14T04:33:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:04 smithi160 ceph-mon[88407]: pgmap v1724: 93 pgs: 15 peering, 78 active+clean; 634 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 6 op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-14T04:33:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:04 smithi160 ceph-mon[88407]: osdmap e473: 8 total, 8 up, 8 in 2024-09-14T04:33:04.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:04 smithi160 ceph-mon[88407]: Health check failed: Reduced data availability: 13 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-14T04:33:05.559 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:33:05.559 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":473,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","created":"2024-09-14T03:36:30.445371+0000","modified":"2024-09-14T04:33:03.505206+0000","last_up_change":"2024-09-14T04:06:53.058416+0000","last_in_change":"2024-09-14T04:31:58.693181+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T03:39:29.348427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"453","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"424","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T03:41:54.688727+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-14T03:43:09.784984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-14T03:43:10.496413+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0a021a3a-45d2-4825-a701-d7c3793ef6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":76,"up_from":80,"up_thru":467,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6803","nonce":2022879658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6805","nonce":2022879658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6809","nonce":2022879658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6807","nonce":2022879658}]},"public_addr":"172.21.15.114:6803/2022879658","cluster_addr":"172.21.15.114:6805/2022879658","heartbeat_back_addr":"172.21.15.114:6809/2022879658","heartbeat_front_addr":"172.21.15.114:6807/2022879658","state":["exists","up"]},{"osd":1,"uuid":"076291c4-5927-456d-bf74-2ab79be4c756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":93,"up_from":97,"up_thru":467,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6811","nonce":3980338706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6813","nonce":3980338706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6817","nonce":3980338706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6815","nonce":3980338706}]},"public_addr":"172.21.15.114:6811/3980338706","cluster_addr":"172.21.15.114:6813/3980338706","heartbeat_back_addr":"172.21.15.114:6817/3980338706","heartbeat_front_addr":"172.21.15.114:6815/3980338706","state":["exists","up"]},{"osd":2,"uuid":"d6787400-73ea-4edf-bb17-440ac36bd4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":103,"up_from":109,"up_thru":451,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6819","nonce":3555775440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6821","nonce":3555775440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6825","nonce":3555775440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6823","nonce":3555775440}]},"public_addr":"172.21.15.114:6819/3555775440","cluster_addr":"172.21.15.114:6821/3555775440","heartbeat_back_addr":"172.21.15.114:6825/3555775440","heartbeat_front_addr":"172.21.15.114:6823/3555775440","state":["exists","up"]},{"osd":3,"uuid":"ead755f3-173a-4465-a28f-af7288b0a42b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":132,"up_from":136,"up_thru":467,"down_at":133,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6827","nonce":288809847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6829","nonce":288809847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6833","nonce":288809847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6831","nonce":288809847}]},"public_addr":"172.21.15.114:6827/288809847","cluster_addr":"172.21.15.114:6829/288809847","heartbeat_back_addr":"172.21.15.114:6833/288809847","heartbeat_front_addr":"172.21.15.114:6831/288809847","state":["exists","up"]},{"osd":4,"uuid":"60a0aac2-e235-484f-b25f-f112c83da7f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":30,"last_clean_end":178,"up_from":186,"up_thru":472,"down_at":179,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":1799506225},{"type":"v1","addr":"172.21.15.160:6801","nonce":1799506225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":1799506225},{"type":"v1","addr":"172.21.15.160:6803","nonce":1799506225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":1799506225},{"type":"v1","addr":"172.21.15.160:6807","nonce":1799506225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":1799506225},{"type":"v1","addr":"172.21.15.160:6805","nonce":1799506225}]},"public_addr":"172.21.15.160:6801/1799506225","cluster_addr":"172.21.15.160:6803/1799506225","heartbeat_back_addr":"172.21.15.160:6807/1799506225","heartbeat_front_addr":"172.21.15.160:6805/1799506225","state":["exists","up"]},{"osd":5,"uuid":"8f49c7de-c810-4075-b731-022bb5031d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":467,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6809","nonce":3752160003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6811","nonce":3752160003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6815","nonce":3752160003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6813","nonce":3752160003}]},"public_addr":"172.21.15.160:6809/3752160003","cluster_addr":"172.21.15.160:6811/3752160003","heartbeat_back_addr":"172.21.15.160:6815/3752160003","heartbeat_front_addr":"172.21.15.160:6813/3752160003","state":["exists","up"]},{"osd":6,"uuid":"455de335-7290-4242-b743-6b0c446683ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":467,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716932982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716932982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716932982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716932982}]},"public_addr":"172.21.15.160:6817/3716932982","cluster_addr":"172.21.15.160:6819/3716932982","heartbeat_back_addr":"172.21.15.160:6823/3716932982","heartbeat_front_addr":"172.21.15.160:6821/3716932982","state":["exists","up"]},{"osd":7,"uuid":"7d8b5297-b1db-41b1-be35-073e7780128e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":469,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6825","nonce":1323253841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6827","nonce":1323253841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6831","nonce":1323253841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6829","nonce":1323253841}]},"public_addr":"172.21.15.160:6825/1323253841","cluster_addr":"172.21.15.160:6827/1323253841","heartbeat_back_addr":"172.21.15.160:6831/1323253841","heartbeat_front_addr":"172.21.15.160:6829/1323253841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:38:38.363862+0000","dead_epoch":76},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:02.363463+0000","dead_epoch":93},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:26.353194+0000","dead_epoch":103},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:50.132880+0000","dead_epoch":132},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:11.402862+0000","dead_epoch":178},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:34.539123+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:58.191287+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:41:22.159510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":6,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"4.10","mappings":[{"from":3,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":7}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.16","mappings":[{"from":3,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.160:6835/3913895673":"2024-09-15T03:47:52.734325+0000","172.21.15.114:6801/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:6800/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1089490408":"2024-09-15T03:36:58.525006+0000","172.21.15.160:0/1361660388":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/767494479":"2024-09-15T03:41:51.162793+0000","172.21.15.160:0/1393128007":"2024-09-15T03:47:52.734325+0000","172.21.15.160:0/710559079":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/2155247703":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/1727106713":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/4089637122":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/2536880143":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/3344792350":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6800/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/426398557":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/1634520325":"2024-09-15T03:48:52.256052+0000","172.21.15.160:6834/3913895673":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/2694023561":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/210127168":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/2067882043":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/3853151361":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/1570092639":"2024-09-15T03:48:52.256052+0000","172.21.15.114:6801/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.160:0/1259038325":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/1481425303":"2024-09-15T03:48:52.256052+0000","172.21.15.114:6800/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6801/1727106713":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/1784685209":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6801/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2995702914":"2024-09-15T03:46:19.354331+0000","172.21.15.114:6800/3774959769":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/386045858":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/546762471":"2024-09-15T03:46:19.354331+0000","172.21.15.114:6801/3774959769":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/1795949490":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/2095546075":"2024-09-15T03:37:21.054968+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T04:33:05.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:05 smithi114 ceph-mon[106969]: 1.5 scrub starts 2024-09-14T04:33:05.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:05 smithi114 ceph-mon[106969]: 1.5 scrub ok 2024-09-14T04:33:05.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:05 smithi114 ceph-mon[103029]: 1.5 scrub starts 2024-09-14T04:33:05.871 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:05 smithi114 ceph-mon[103029]: 1.5 scrub ok 2024-09-14T04:33:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:05 smithi160 ceph-mon[88407]: 1.5 scrub starts 2024-09-14T04:33:05.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:05 smithi160 ceph-mon[88407]: 1.5 scrub ok 2024-09-14T04:33:06.296 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-14T04:33:06.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-14T04:33:06.542 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:06.797 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:06 smithi114 ceph-mon[106969]: pgmap v1726: 93 pgs: 15 peering, 78 active+clean; 634 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 6 op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-14T04:33:06.797 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:06 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2609872442' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:33:06.797 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:06 smithi114 ceph-mon[106969]: 1.e scrub starts 2024-09-14T04:33:06.797 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:06 smithi114 ceph-mon[106969]: 1.e scrub ok 2024-09-14T04:33:06.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:06 smithi114 ceph-mon[103029]: pgmap v1726: 93 pgs: 15 peering, 78 active+clean; 634 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 6 op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-14T04:33:06.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:06 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2609872442' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:33:06.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:06 smithi114 ceph-mon[103029]: 1.e scrub starts 2024-09-14T04:33:06.798 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:06 smithi114 ceph-mon[103029]: 1.e scrub ok 2024-09-14T04:33:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:06 smithi160 ceph-mon[88407]: pgmap v1726: 93 pgs: 15 peering, 78 active+clean; 634 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 10 KiB/s rd, 22 KiB/s wr, 6 op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-14T04:33:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:06 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2609872442' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:33:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:06 smithi160 ceph-mon[88407]: 1.e scrub starts 2024-09-14T04:33:06.908 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:06 smithi160 ceph-mon[88407]: 1.e scrub ok 2024-09-14T04:33:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:33:08.570 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":473,"fsid":"63f4e762-724a-11ef-bceb-c7b262605968","created":"2024-09-14T03:36:30.445371+0000","modified":"2024-09-14T04:33:03.505206+0000","last_up_change":"2024-09-14T04:06:53.058416+0000","last_in_change":"2024-09-14T04:31:58.693181+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T03:39:29.348427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"453","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"424","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T03:41:54.688727+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-14T03:43:09.784984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-14T03:43:10.496413+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0a021a3a-45d2-4825-a701-d7c3793ef6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":76,"up_from":80,"up_thru":467,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6803","nonce":2022879658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6805","nonce":2022879658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6809","nonce":2022879658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2022879658},{"type":"v1","addr":"172.21.15.114:6807","nonce":2022879658}]},"public_addr":"172.21.15.114:6803/2022879658","cluster_addr":"172.21.15.114:6805/2022879658","heartbeat_back_addr":"172.21.15.114:6809/2022879658","heartbeat_front_addr":"172.21.15.114:6807/2022879658","state":["exists","up"]},{"osd":1,"uuid":"076291c4-5927-456d-bf74-2ab79be4c756","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":93,"up_from":97,"up_thru":467,"down_at":94,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6811","nonce":3980338706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6813","nonce":3980338706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6817","nonce":3980338706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3980338706},{"type":"v1","addr":"172.21.15.114:6815","nonce":3980338706}]},"public_addr":"172.21.15.114:6811/3980338706","cluster_addr":"172.21.15.114:6813/3980338706","heartbeat_back_addr":"172.21.15.114:6817/3980338706","heartbeat_front_addr":"172.21.15.114:6815/3980338706","state":["exists","up"]},{"osd":2,"uuid":"d6787400-73ea-4edf-bb17-440ac36bd4ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":103,"up_from":109,"up_thru":451,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6819","nonce":3555775440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6821","nonce":3555775440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6825","nonce":3555775440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":3555775440},{"type":"v1","addr":"172.21.15.114:6823","nonce":3555775440}]},"public_addr":"172.21.15.114:6819/3555775440","cluster_addr":"172.21.15.114:6821/3555775440","heartbeat_back_addr":"172.21.15.114:6825/3555775440","heartbeat_front_addr":"172.21.15.114:6823/3555775440","state":["exists","up"]},{"osd":3,"uuid":"ead755f3-173a-4465-a28f-af7288b0a42b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":132,"up_from":136,"up_thru":467,"down_at":133,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6827","nonce":288809847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6829","nonce":288809847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6833","nonce":288809847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":288809847},{"type":"v1","addr":"172.21.15.114:6831","nonce":288809847}]},"public_addr":"172.21.15.114:6827/288809847","cluster_addr":"172.21.15.114:6829/288809847","heartbeat_back_addr":"172.21.15.114:6833/288809847","heartbeat_front_addr":"172.21.15.114:6831/288809847","state":["exists","up"]},{"osd":4,"uuid":"60a0aac2-e235-484f-b25f-f112c83da7f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":30,"last_clean_end":178,"up_from":186,"up_thru":472,"down_at":179,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":1799506225},{"type":"v1","addr":"172.21.15.160:6801","nonce":1799506225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":1799506225},{"type":"v1","addr":"172.21.15.160:6803","nonce":1799506225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":1799506225},{"type":"v1","addr":"172.21.15.160:6807","nonce":1799506225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":1799506225},{"type":"v1","addr":"172.21.15.160:6805","nonce":1799506225}]},"public_addr":"172.21.15.160:6801/1799506225","cluster_addr":"172.21.15.160:6803/1799506225","heartbeat_back_addr":"172.21.15.160:6807/1799506225","heartbeat_front_addr":"172.21.15.160:6805/1799506225","state":["exists","up"]},{"osd":5,"uuid":"8f49c7de-c810-4075-b731-022bb5031d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":467,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6809","nonce":3752160003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6811","nonce":3752160003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6815","nonce":3752160003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3752160003},{"type":"v1","addr":"172.21.15.160:6813","nonce":3752160003}]},"public_addr":"172.21.15.160:6809/3752160003","cluster_addr":"172.21.15.160:6811/3752160003","heartbeat_back_addr":"172.21.15.160:6815/3752160003","heartbeat_front_addr":"172.21.15.160:6813/3752160003","state":["exists","up"]},{"osd":6,"uuid":"455de335-7290-4242-b743-6b0c446683ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":467,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6817","nonce":3716932982}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6819","nonce":3716932982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6823","nonce":3716932982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":3716932982},{"type":"v1","addr":"172.21.15.160:6821","nonce":3716932982}]},"public_addr":"172.21.15.160:6817/3716932982","cluster_addr":"172.21.15.160:6819/3716932982","heartbeat_back_addr":"172.21.15.160:6823/3716932982","heartbeat_front_addr":"172.21.15.160:6821/3716932982","state":["exists","up"]},{"osd":7,"uuid":"7d8b5297-b1db-41b1-be35-073e7780128e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":469,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6824","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6825","nonce":1323253841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6826","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6827","nonce":1323253841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6830","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6831","nonce":1323253841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6828","nonce":1323253841},{"type":"v1","addr":"172.21.15.160:6829","nonce":1323253841}]},"public_addr":"172.21.15.160:6825/1323253841","cluster_addr":"172.21.15.160:6827/1323253841","heartbeat_back_addr":"172.21.15.160:6831/1323253841","heartbeat_front_addr":"172.21.15.160:6829/1323253841","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:38:38.363862+0000","dead_epoch":76},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:02.363463+0000","dead_epoch":93},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:26.353194+0000","dead_epoch":103},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:39:50.132880+0000","dead_epoch":132},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:11.402862+0000","dead_epoch":178},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:34.539123+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:40:58.191287+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T03:41:22.159510+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":6,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"4.10","mappings":[{"from":3,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":7}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.16","mappings":[{"from":3,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/1795949490":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/386045858":"2024-09-15T03:48:52.256052+0000","172.21.15.114:6800/3774959769":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/1481425303":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/210127168":"2024-09-15T03:48:52.256052+0000","172.21.15.160:0/1259038325":"2024-09-15T03:47:52.734325+0000","172.21.15.160:0/710559079":"2024-09-15T03:47:52.734325+0000","172.21.15.160:0/1393128007":"2024-09-15T03:47:52.734325+0000","172.21.15.114:0/426398557":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/1634520325":"2024-09-15T03:48:52.256052+0000","172.21.15.114:6800/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6801/1727106713":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/767494479":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/2155247703":"2024-09-15T03:37:21.054968+0000","172.21.15.114:6800/1727106713":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/4089637122":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/2536880143":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/3344792350":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6800/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/1570092639":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/2067882043":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/3853151361":"2024-09-15T03:46:19.354331+0000","172.21.15.114:6801/441101312":"2024-09-15T03:41:51.162793+0000","172.21.15.114:0/1784685209":"2024-09-15T03:41:51.162793+0000","172.21.15.114:6801/2959862646":"2024-09-15T03:37:21.054968+0000","172.21.15.114:0/2995702914":"2024-09-15T03:46:19.354331+0000","172.21.15.114:0/2095546075":"2024-09-15T03:37:21.054968+0000","172.21.15.160:0/1361660388":"2024-09-15T03:47:52.734325+0000","172.21.15.160:6835/3913895673":"2024-09-15T03:47:52.734325+0000","172.21.15.114:6801/3774959769":"2024-09-15T03:48:52.256052+0000","172.21.15.114:0/546762471":"2024-09-15T03:46:19.354331+0000","172.21.15.114:6801/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/1089490408":"2024-09-15T03:36:58.525006+0000","172.21.15.114:6800/1434563052":"2024-09-15T03:36:58.525006+0000","172.21.15.114:0/2694023561":"2024-09-15T03:36:58.525006+0000","172.21.15.160:6834/3913895673":"2024-09-15T03:47:52.734325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T04:33:08.928 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:08 smithi114 ceph-mon[106969]: pgmap v1727: 93 pgs: 15 peering, 78 active+clean; 634 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 5 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-14T04:33:08.928 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:08.928 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:08 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:08.928 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:08 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1939723051' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:33:08.929 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:08 smithi114 ceph-mon[103029]: pgmap v1727: 93 pgs: 15 peering, 78 active+clean; 634 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 5 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-14T04:33:08.929 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:08.929 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:08 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:08.929 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:08 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1939723051' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:33:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:08 smithi160 ceph-mon[88407]: pgmap v1727: 93 pgs: 15 peering, 78 active+clean; 634 KiB data, 10 GiB used, 705 GiB / 715 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 5 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-14T04:33:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:08 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:09.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:08 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1939723051' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-14T04:33:09.357 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-14T04:33:09.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-14T04:33:09.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-14T04:33:09.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-14T04:33:09.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-14T04:33:09.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-14T04:33:09.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-14T04:33:09.360 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-14T04:33:09.820 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:09.896 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:09.897 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:09.898 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:09.898 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:09.899 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:09.903 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:09.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:09 smithi114 ceph-mon[106969]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 15 pgs peering) 2024-09-14T04:33:09.986 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:09 smithi114 ceph-mon[106969]: Cluster is now healthy 2024-09-14T04:33:09.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:09 smithi114 ceph-mon[103029]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 15 pgs peering) 2024-09-14T04:33:09.987 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:09 smithi114 ceph-mon[103029]: Cluster is now healthy 2024-09-14T04:33:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:09 smithi160 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 15 pgs peering) 2024-09-14T04:33:10.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:09 smithi160 ceph-mon[88407]: Cluster is now healthy 2024-09-14T04:33:10.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:33:10 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:33:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:33:10.678 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:10.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:10 smithi114 ceph-mon[103029]: pgmap v1728: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:10.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:10 smithi114 ceph-mon[106969]: pgmap v1728: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:11.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:10 smithi160 ceph-mon[88407]: pgmap v1728: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:11.730 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:33:11] "GET /metrics HTTP/1.1" 200 34162 "" "Prometheus/2.51.0" 2024-09-14T04:33:12.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:12 smithi114 ceph-mon[103029]: pgmap v1729: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:12.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:12 smithi114 ceph-mon[106969]: pgmap v1729: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:13.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:12 smithi160 ceph-mon[88407]: pgmap v1729: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:15.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:14 smithi114 ceph-mon[103029]: pgmap v1730: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:15.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:14 smithi114 ceph-mon[106969]: pgmap v1730: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:15.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:14 smithi160 ceph-mon[88407]: pgmap v1730: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:16.847 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:16 smithi114 ceph-mon[106969]: pgmap v1731: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:16.847 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:16 smithi114 ceph-mon[103029]: pgmap v1731: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:17.160 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:16 smithi160 ceph-mon[88407]: pgmap v1731: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:19.058 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:18 smithi114 ceph-mon[106969]: pgmap v1732: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:19.058 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:18 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:19.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:18 smithi114 ceph-mon[103029]: pgmap v1732: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:19.058 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:18 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:19.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:18 smithi160 ceph-mon[88407]: pgmap v1732: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:19.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:18 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:20.065 INFO:teuthology.orchestra.run.smithi114.stdout:180388627064 2024-09-14T04:33:20.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-14T04:33:20.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:33:20 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:33:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:33:20.855 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:20 smithi114 ceph-mon[103029]: pgmap v1733: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:20.855 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:20 smithi114 ceph-mon[106969]: pgmap v1733: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:21.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:20 smithi160 ceph-mon[88407]: pgmap v1733: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:21.492 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:21 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:33:21] "GET /metrics HTTP/1.1" 200 34162 "" "Prometheus/2.51.0" 2024-09-14T04:33:21.835 INFO:teuthology.orchestra.run.smithi114.stdout:343597384174 2024-09-14T04:33:21.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-14T04:33:22.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:22 smithi114 ceph-mon[103029]: pgmap v1734: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:22.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:22.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:22 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:22 smithi114 ceph-mon[106969]: pgmap v1734: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:22.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:22 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:22 smithi160 ceph-mon[88407]: pgmap v1734: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' 2024-09-14T04:33:23.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:22 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:24.011 INFO:teuthology.orchestra.run.smithi114.stdout:584115552673 2024-09-14T04:33:24.011 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-14T04:33:24.329 INFO:teuthology.orchestra.run.smithi114.stdout:468151435715 2024-09-14T04:33:24.329 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-14T04:33:24.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:24 smithi114 ceph-mon[103029]: pgmap v1735: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:24.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:24 smithi114 ceph-mon[106969]: pgmap v1735: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:25.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:24 smithi160 ceph-mon[88407]: pgmap v1735: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:25.598 INFO:teuthology.orchestra.run.smithi114.stdout:416611828184 2024-09-14T04:33:25.599 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-14T04:33:26.916 INFO:teuthology.orchestra.run.smithi114.stdout:154618823294 2024-09-14T04:33:26.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-14T04:33:27.042 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:26 smithi114 ceph-mon[106969]: pgmap v1736: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:27.042 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:26 smithi114 ceph-mon[103029]: pgmap v1736: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:27.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:26 smithi160 ceph-mon[88407]: pgmap v1736: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:27.356 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:27.388 INFO:teuthology.orchestra.run.smithi114.stdout:206158430837 2024-09-14T04:33:27.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-14T04:33:27.397 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:28.632 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:29.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:28 smithi114 ceph-mon[103029]: pgmap v1737: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:29.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:28 smithi114 ceph-mon[106969]: pgmap v1737: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:29.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:28 smithi160 ceph-mon[88407]: pgmap v1737: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:29.189 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:30.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:33:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:33:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:33:31.001 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:30 smithi114 ceph-mon[106969]: pgmap v1738: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:31.001 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:30 smithi114 ceph-mon[103029]: pgmap v1738: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:31.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:30 smithi160 ceph-mon[88407]: pgmap v1738: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:31.620 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:33:31] "GET /metrics HTTP/1.1" 200 34143 "" "Prometheus/2.51.0" 2024-09-14T04:33:31.937 INFO:teuthology.orchestra.run.smithi114.stdout:798863917377 2024-09-14T04:33:31.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-14T04:33:33.105 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:32 smithi114 ceph-mon[106969]: pgmap v1739: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:33.105 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:32 smithi114 ceph-mon[103029]: pgmap v1739: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:33.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:32 smithi160 ceph-mon[88407]: pgmap v1739: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:33.566 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:33.720 INFO:teuthology.orchestra.run.smithi114.stdout:343597384176 2024-09-14T04:33:34.029 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:33 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1741977892' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-14T04:33:34.029 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:33 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1741977892' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-14T04:33:34.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:33 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1741977892' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-14T04:33:34.596 INFO:teuthology.orchestra.run.smithi114.stdout:468151435717 2024-09-14T04:33:34.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:34 smithi114 ceph-mon[103029]: pgmap v1740: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:34.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:34 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3364837411' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-14T04:33:34.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:34 smithi114 ceph-mon[106969]: pgmap v1740: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:34.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:34 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3364837411' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-14T04:33:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:34 smithi160 ceph-mon[88407]: pgmap v1740: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:35.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:34 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3364837411' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-14T04:33:35.412 INFO:tasks.cephadm.ceph_manager.ceph:need seq 468151435715 got 468151435717 for osd.2 2024-09-14T04:33:35.412 DEBUG:teuthology.parallel:result is None 2024-09-14T04:33:35.867 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597384174 got 343597384176 for osd.0 2024-09-14T04:33:35.867 DEBUG:teuthology.parallel:result is None 2024-09-14T04:33:36.218 INFO:teuthology.orchestra.run.smithi114.stdout:180388627068 2024-09-14T04:33:36.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:36 smithi114 ceph-mon[106969]: pgmap v1741: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:36.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:36 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2711211409' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-14T04:33:36.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:36 smithi114 ceph-mon[103029]: pgmap v1741: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:36.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:36 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2711211409' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-14T04:33:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:36 smithi160 ceph-mon[88407]: pgmap v1741: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:37.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:36 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2711211409' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-14T04:33:37.700 INFO:teuthology.orchestra.run.smithi114.stdout:416611828188 2024-09-14T04:33:38.010 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:37 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:38.011 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:37 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/3426663336' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-14T04:33:38.011 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:37 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:38.011 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:37 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/3426663336' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-14T04:33:38.017 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:38.017 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:38.045 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388627064 got 180388627068 for osd.6 2024-09-14T04:33:38.045 DEBUG:teuthology.parallel:result is None 2024-09-14T04:33:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:37 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:38.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:37 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/3426663336' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-14T04:33:39.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:38 smithi114 ceph-mon[103029]: pgmap v1742: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:39.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:38 smithi114 ceph-mon[106969]: pgmap v1742: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:39.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:38 smithi160 ceph-mon[88407]: pgmap v1742: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:39.325 INFO:tasks.cephadm.ceph_manager.ceph:need seq 416611828184 got 416611828188 for osd.1 2024-09-14T04:33:39.326 DEBUG:teuthology.parallel:result is None 2024-09-14T04:33:40.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:39 smithi114 ceph-mon[103029]: pgmap v1743: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:40.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:39 smithi114 ceph-mon[106969]: pgmap v1743: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:40.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:39 smithi160 ceph-mon[88407]: pgmap v1743: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:40.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:33:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:33:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:33:41.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:40 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/406322664' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-14T04:33:41.120 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:40 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/406322664' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-14T04:33:41.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:40 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/406322664' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-14T04:33:41.787 INFO:teuthology.orchestra.run.smithi114.stdout:584115552677 2024-09-14T04:33:41.787 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:41 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:33:41] "GET /metrics HTTP/1.1" 200 34140 "" "Prometheus/2.51.0" 2024-09-14T04:33:42.092 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:41 smithi114 ceph-mon[106969]: pgmap v1744: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:42.093 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:41 smithi114 ceph-mon[103029]: pgmap v1744: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:42.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:41 smithi160 ceph-mon[88407]: pgmap v1744: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:42.514 INFO:tasks.cephadm.ceph_manager.ceph:need seq 584115552673 got 584115552677 for osd.3 2024-09-14T04:33:42.514 DEBUG:teuthology.parallel:result is None 2024-09-14T04:33:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:42.892 INFO:teuthology.orchestra.run.smithi114.stdout:154618823297 2024-09-14T04:33:43.120 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:42 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/69307699' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-14T04:33:43.121 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:42 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/69307699' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-14T04:33:43.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:42 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/69307699' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-14T04:33:44.025 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618823294 got 154618823297 for osd.5 2024-09-14T04:33:44.026 DEBUG:teuthology.parallel:result is None 2024-09-14T04:33:44.284 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:43 smithi114 ceph-mon[106969]: pgmap v1745: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:44.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:43 smithi114 ceph-mon[103029]: pgmap v1745: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:44.408 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:43 smithi160 ceph-mon[88407]: pgmap v1745: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:45.954 INFO:teuthology.orchestra.run.smithi114.stdout:798863917380 2024-09-14T04:33:46.527 INFO:teuthology.orchestra.run.smithi114.stdout:206158430841 2024-09-14T04:33:46.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:46 smithi114 ceph-mon[103029]: pgmap v1746: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:46.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:46 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/1435977381' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-14T04:33:46.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:46 smithi114 ceph-mon[106969]: pgmap v1746: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:46.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:46 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/1435977381' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-14T04:33:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:46 smithi160 ceph-mon[88407]: pgmap v1746: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:46.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:46 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/1435977381' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-14T04:33:47.257 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430837 got 206158430841 for osd.7 2024-09-14T04:33:47.257 DEBUG:teuthology.parallel:result is None 2024-09-14T04:33:47.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:47 smithi114 ceph-mon[103029]: from='client.? 172.21.15.114:0/2295695443' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-14T04:33:47.621 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:47 smithi114 ceph-mon[106969]: from='client.? 172.21.15.114:0/2295695443' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-14T04:33:47.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:47 smithi160 ceph-mon[88407]: from='client.? 172.21.15.114:0/2295695443' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-14T04:33:47.746 INFO:tasks.cephadm.ceph_manager.ceph:need seq 798863917377 got 798863917380 for osd.4 2024-09-14T04:33:47.747 DEBUG:teuthology.parallel:result is None 2024-09-14T04:33:47.747 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-14T04:33:47.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:33:48.000 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:48.453 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:48 smithi114 ceph-mon[103029]: pgmap v1747: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:48.453 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:48 smithi114 ceph-mon[106969]: pgmap v1747: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:48.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:48 smithi160 ceph-mon[88407]: pgmap v1747: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:50.195 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:33:50.197 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:33:50.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:50 smithi114 ceph-mon[103029]: pgmap v1748: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:50.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:50 smithi114 ceph-mon[106969]: pgmap v1748: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:50.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:50 smithi160 ceph-mon[88407]: pgmap v1748: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:50.658 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:33:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:33:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:33:50.940 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1748,"stamp":"2024-09-14T04:33:49.290601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":662378,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2158,"ondisk_log_size":2158,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10875148,"kb_used_data":48220,"kb_used_omap":33,"kb_used_meta":10826782,"kb_avail":739086068,"statfs":{"total":767960285184,"available":756824133632,"internally_reserved":0,"allocated":49377280,"data_stored":31605570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34628,"internal_metadata":11086624956},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003515"},"pg_stats":[{"pgid":"3.1f","version":"62'4","reported_seq":774,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513865+0000","last_change":"2024-09-14T04:32:14.758999+0000","last_active":"2024-09-14T04:33:03.513865+0000","last_peered":"2024-09-14T04:33:03.513865+0000","last_clean":"2024-09-14T04:33:03.513865+0000","last_became_active":"2024-09-14T04:32:14.758383+0000","last_became_peered":"2024-09-14T04:32:14.758383+0000","last_unstale":"2024-09-14T04:33:03.513865+0000","last_undegraded":"2024-09-14T04:33:03.513865+0000","last_fullsized":"2024-09-14T04:33:03.513865+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:21:59.358588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":275,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.520443+0000","last_change":"2024-09-14T04:33:03.520443+0000","last_active":"2024-09-14T04:33:03.520443+0000","last_peered":"2024-09-14T04:33:03.520443+0000","last_clean":"2024-09-14T04:33:03.520443+0000","last_became_active":"2024-09-14T04:33:03.520131+0000","last_became_peered":"2024-09-14T04:33:03.520131+0000","last_unstale":"2024-09-14T04:33:03.520443+0000","last_undegraded":"2024-09-14T04:33:03.520443+0000","last_fullsized":"2024-09-14T04:33:03.520443+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:27:25.194028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":606,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517661+0000","last_change":"2024-09-14T04:25:42.235102+0000","last_active":"2024-09-14T04:33:03.517661+0000","last_peered":"2024-09-14T04:33:03.517661+0000","last_clean":"2024-09-14T04:33:03.517661+0000","last_became_active":"2024-09-14T04:25:42.234776+0000","last_became_peered":"2024-09-14T04:25:42.234776+0000","last_unstale":"2024-09-14T04:33:03.517661+0000","last_undegraded":"2024-09-14T04:33:03.517661+0000","last_fullsized":"2024-09-14T04:33:03.517661+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:59:52.026651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":604,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516734+0000","last_change":"2024-09-14T04:31:59.732915+0000","last_active":"2024-09-14T04:33:03.516734+0000","last_peered":"2024-09-14T04:33:03.516734+0000","last_clean":"2024-09-14T04:33:03.516734+0000","last_became_active":"2024-09-14T04:31:59.732583+0000","last_became_peered":"2024-09-14T04:31:59.732583+0000","last_unstale":"2024-09-14T04:33:03.516734+0000","last_undegraded":"2024-09-14T04:33:03.516734+0000","last_fullsized":"2024-09-14T04:33:03.516734+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:38:28.751668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":178,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517493+0000","last_change":"2024-09-14T04:27:55.405108+0000","last_active":"2024-09-14T04:33:03.517493+0000","last_peered":"2024-09-14T04:33:03.517493+0000","last_clean":"2024-09-14T04:33:03.517493+0000","last_became_active":"2024-09-14T04:27:55.394475+0000","last_became_peered":"2024-09-14T04:27:55.394475+0000","last_unstale":"2024-09-14T04:33:03.517493+0000","last_undegraded":"2024-09-14T04:33:03.517493+0000","last_fullsized":"2024-09-14T04:33:03.517493+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:44:06.833702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":641,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.521856+0000","last_change":"2024-09-14T04:33:03.521856+0000","last_active":"2024-09-14T04:33:03.521856+0000","last_peered":"2024-09-14T04:33:03.521856+0000","last_clean":"2024-09-14T04:33:03.521856+0000","last_became_active":"2024-09-14T04:33:03.521485+0000","last_became_peered":"2024-09-14T04:33:03.521485+0000","last_unstale":"2024-09-14T04:33:03.521856+0000","last_undegraded":"2024-09-14T04:33:03.521856+0000","last_fullsized":"2024-09-14T04:33:03.521856+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:54:29.353652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":190,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.507006+0000","last_change":"2024-09-14T04:31:52.391477+0000","last_active":"2024-09-14T04:33:02.507006+0000","last_peered":"2024-09-14T04:33:02.507006+0000","last_clean":"2024-09-14T04:33:02.507006+0000","last_became_active":"2024-09-14T04:31:52.374373+0000","last_became_peered":"2024-09-14T04:31:52.374373+0000","last_unstale":"2024-09-14T04:33:02.507006+0000","last_undegraded":"2024-09-14T04:33:02.507006+0000","last_fullsized":"2024-09-14T04:33:02.507006+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:42:35.354303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":658,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.522497+0000","last_change":"2024-09-14T04:33:03.522497+0000","last_active":"2024-09-14T04:33:03.522497+0000","last_peered":"2024-09-14T04:33:03.522497+0000","last_clean":"2024-09-14T04:33:03.522497+0000","last_became_active":"2024-09-14T04:33:03.522200+0000","last_became_peered":"2024-09-14T04:33:03.522200+0000","last_unstale":"2024-09-14T04:33:03.522497+0000","last_undegraded":"2024-09-14T04:33:03.522497+0000","last_fullsized":"2024-09-14T04:33:03.522497+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:47:02.685520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":54,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513937+0000","last_change":"2024-09-14T04:31:59.727688+0000","last_active":"2024-09-14T04:33:03.513937+0000","last_peered":"2024-09-14T04:33:03.513937+0000","last_clean":"2024-09-14T04:33:03.513937+0000","last_became_active":"2024-09-14T04:31:59.727253+0000","last_became_peered":"2024-09-14T04:31:59.727253+0000","last_unstale":"2024-09-14T04:33:03.513937+0000","last_undegraded":"2024-09-14T04:33:03.513937+0000","last_fullsized":"2024-09-14T04:33:03.513937+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:30:47.836983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":253,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516739+0000","last_change":"2024-09-14T04:31:52.375414+0000","last_active":"2024-09-14T04:33:03.516739+0000","last_peered":"2024-09-14T04:33:03.516739+0000","last_clean":"2024-09-14T04:33:03.516739+0000","last_became_active":"2024-09-14T04:31:52.375096+0000","last_became_peered":"2024-09-14T04:31:52.375096+0000","last_unstale":"2024-09-14T04:33:03.516739+0000","last_undegraded":"2024-09-14T04:33:03.516739+0000","last_fullsized":"2024-09-14T04:33:03.516739+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:35:20.648591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":173,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.521477+0000","last_change":"2024-09-14T04:33:03.521477+0000","last_active":"2024-09-14T04:33:03.521477+0000","last_peered":"2024-09-14T04:33:03.521477+0000","last_clean":"2024-09-14T04:33:03.521477+0000","last_became_active":"2024-09-14T04:33:03.521109+0000","last_became_peered":"2024-09-14T04:33:03.521109+0000","last_unstale":"2024-09-14T04:33:03.521477+0000","last_undegraded":"2024-09-14T04:33:03.521477+0000","last_fullsized":"2024-09-14T04:33:03.521477+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:29:21.238847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":597,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517831+0000","last_change":"2024-09-14T04:31:52.383504+0000","last_active":"2024-09-14T04:33:03.517831+0000","last_peered":"2024-09-14T04:33:03.517831+0000","last_clean":"2024-09-14T04:33:03.517831+0000","last_became_active":"2024-09-14T04:31:52.383178+0000","last_became_peered":"2024-09-14T04:31:52.383178+0000","last_unstale":"2024-09-14T04:33:03.517831+0000","last_undegraded":"2024-09-14T04:33:03.517831+0000","last_fullsized":"2024-09-14T04:33:03.517831+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:39:06.937151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518330+0000","last_change":"2024-09-14T04:31:52.380838+0000","last_active":"2024-09-14T04:33:03.518330+0000","last_peered":"2024-09-14T04:33:03.518330+0000","last_clean":"2024-09-14T04:33:03.518330+0000","last_became_active":"2024-09-14T04:31:52.380514+0000","last_became_peered":"2024-09-14T04:31:52.380514+0000","last_unstale":"2024-09-14T04:33:03.518330+0000","last_undegraded":"2024-09-14T04:33:03.518330+0000","last_fullsized":"2024-09-14T04:33:03.518330+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:38:13.905830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":616,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513658+0000","last_change":"2024-09-14T04:32:51.484111+0000","last_active":"2024-09-14T04:33:03.513658+0000","last_peered":"2024-09-14T04:33:03.513658+0000","last_clean":"2024-09-14T04:33:03.513658+0000","last_became_active":"2024-09-14T04:32:51.483789+0000","last_became_peered":"2024-09-14T04:32:51.483789+0000","last_unstale":"2024-09-14T04:33:03.513658+0000","last_undegraded":"2024-09-14T04:33:03.513658+0000","last_fullsized":"2024-09-14T04:33:03.513658+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:18:09.617056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":625,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518152+0000","last_change":"2024-09-14T04:31:59.734041+0000","last_active":"2024-09-14T04:33:03.518152+0000","last_peered":"2024-09-14T04:33:03.518152+0000","last_clean":"2024-09-14T04:33:03.518152+0000","last_became_active":"2024-09-14T04:31:59.733625+0000","last_became_peered":"2024-09-14T04:31:59.733625+0000","last_unstale":"2024-09-14T04:33:03.518152+0000","last_undegraded":"2024-09-14T04:33:03.518152+0000","last_fullsized":"2024-09-14T04:33:03.518152+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:33:47.588119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":507,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516510+0000","last_change":"2024-09-14T04:31:52.371573+0000","last_active":"2024-09-14T04:33:02.516510+0000","last_peered":"2024-09-14T04:33:02.516510+0000","last_clean":"2024-09-14T04:33:02.516510+0000","last_became_active":"2024-09-14T04:31:52.371233+0000","last_became_peered":"2024-09-14T04:31:52.371233+0000","last_unstale":"2024-09-14T04:33:02.516510+0000","last_undegraded":"2024-09-14T04:33:02.516510+0000","last_fullsized":"2024-09-14T04:33:02.516510+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:56:28.883198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":181,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518745+0000","last_change":"2024-09-14T04:31:52.371860+0000","last_active":"2024-09-14T04:33:03.518745+0000","last_peered":"2024-09-14T04:33:03.518745+0000","last_clean":"2024-09-14T04:33:03.518745+0000","last_became_active":"2024-09-14T04:31:52.371537+0000","last_became_peered":"2024-09-14T04:31:52.371537+0000","last_unstale":"2024-09-14T04:33:03.518745+0000","last_undegraded":"2024-09-14T04:33:03.518745+0000","last_fullsized":"2024-09-14T04:33:03.518745+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:06:38.848838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":745,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513715+0000","last_change":"2024-09-14T04:32:14.760654+0000","last_active":"2024-09-14T04:33:03.513715+0000","last_peered":"2024-09-14T04:33:03.513715+0000","last_clean":"2024-09-14T04:33:03.513715+0000","last_became_active":"2024-09-14T04:32:14.759851+0000","last_became_peered":"2024-09-14T04:32:14.759851+0000","last_unstale":"2024-09-14T04:33:03.513715+0000","last_undegraded":"2024-09-14T04:33:03.513715+0000","last_fullsized":"2024-09-14T04:33:03.513715+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:08:47.524123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'100","reported_seq":109,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516591+0000","last_change":"2024-09-14T04:31:47.352194+0000","last_active":"2024-09-14T04:33:03.516591+0000","last_peered":"2024-09-14T04:33:03.516591+0000","last_clean":"2024-09-14T04:33:03.516591+0000","last_became_active":"2024-09-14T04:31:47.351839+0000","last_became_peered":"2024-09-14T04:31:47.351839+0000","last_unstale":"2024-09-14T04:33:03.516591+0000","last_undegraded":"2024-09-14T04:33:03.516591+0000","last_fullsized":"2024-09-14T04:33:03.516591+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:44.013432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:44.013432+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:50.272154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":766,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516923+0000","last_change":"2024-09-14T04:31:47.352083+0000","last_active":"2024-09-14T04:33:03.516923+0000","last_peered":"2024-09-14T04:33:03.516923+0000","last_clean":"2024-09-14T04:33:03.516923+0000","last_became_active":"2024-09-14T04:31:47.351203+0000","last_became_peered":"2024-09-14T04:31:47.351203+0000","last_unstale":"2024-09-14T04:33:03.516923+0000","last_undegraded":"2024-09-14T04:33:03.516923+0000","last_fullsized":"2024-09-14T04:33:03.516923+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:06:33.387299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":208,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506698+0000","last_change":"2024-09-14T04:31:52.375127+0000","last_active":"2024-09-14T04:33:02.506698+0000","last_peered":"2024-09-14T04:33:02.506698+0000","last_clean":"2024-09-14T04:33:02.506698+0000","last_became_active":"2024-09-14T04:31:52.374761+0000","last_became_peered":"2024-09-14T04:31:52.374761+0000","last_unstale":"2024-09-14T04:33:02.506698+0000","last_undegraded":"2024-09-14T04:33:02.506698+0000","last_fullsized":"2024-09-14T04:33:02.506698+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:52:11.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":437,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516689+0000","last_change":"2024-09-14T04:32:56.491728+0000","last_active":"2024-09-14T04:33:02.516689+0000","last_peered":"2024-09-14T04:33:02.516689+0000","last_clean":"2024-09-14T04:33:02.516689+0000","last_became_active":"2024-09-14T04:32:56.491120+0000","last_became_peered":"2024-09-14T04:32:56.491120+0000","last_unstale":"2024-09-14T04:33:02.516689+0000","last_undegraded":"2024-09-14T04:33:02.516689+0000","last_fullsized":"2024-09-14T04:33:02.516689+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:07:40.674723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518319+0000","last_change":"2024-09-14T04:31:52.371965+0000","last_active":"2024-09-14T04:33:03.518319+0000","last_peered":"2024-09-14T04:33:03.518319+0000","last_clean":"2024-09-14T04:33:03.518319+0000","last_became_active":"2024-09-14T04:31:52.371454+0000","last_became_peered":"2024-09-14T04:31:52.371454+0000","last_unstale":"2024-09-14T04:33:03.518319+0000","last_undegraded":"2024-09-14T04:33:03.518319+0000","last_fullsized":"2024-09-14T04:33:03.518319+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:00:40.307983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":69,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.521088+0000","last_change":"2024-09-14T04:33:03.521088+0000","last_active":"2024-09-14T04:33:03.521088+0000","last_peered":"2024-09-14T04:33:03.521088+0000","last_clean":"2024-09-14T04:33:03.521088+0000","last_became_active":"2024-09-14T04:33:03.520674+0000","last_became_peered":"2024-09-14T04:33:03.520674+0000","last_unstale":"2024-09-14T04:33:03.521088+0000","last_undegraded":"2024-09-14T04:33:03.521088+0000","last_fullsized":"2024-09-14T04:33:03.521088+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:38:16.495007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":238,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.520723+0000","last_change":"2024-09-14T04:33:03.520723+0000","last_active":"2024-09-14T04:33:03.520723+0000","last_peered":"2024-09-14T04:33:03.520723+0000","last_clean":"2024-09-14T04:33:03.520723+0000","last_became_active":"2024-09-14T04:33:03.520300+0000","last_became_peered":"2024-09-14T04:33:03.520300+0000","last_unstale":"2024-09-14T04:33:03.520723+0000","last_undegraded":"2024-09-14T04:33:03.520723+0000","last_fullsized":"2024-09-14T04:33:03.520723+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:25:15.942721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":113,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.519828+0000","last_change":"2024-09-14T04:33:03.519828+0000","last_active":"2024-09-14T04:33:03.519828+0000","last_peered":"2024-09-14T04:33:03.519828+0000","last_clean":"2024-09-14T04:33:03.519828+0000","last_became_active":"2024-09-14T04:33:03.519427+0000","last_became_peered":"2024-09-14T04:33:03.519427+0000","last_unstale":"2024-09-14T04:33:03.519828+0000","last_undegraded":"2024-09-14T04:33:03.519828+0000","last_fullsized":"2024-09-14T04:33:03.519828+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:17:42.336129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":500,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516589+0000","last_change":"2024-09-14T04:32:51.483859+0000","last_active":"2024-09-14T04:33:02.516589+0000","last_peered":"2024-09-14T04:33:02.516589+0000","last_clean":"2024-09-14T04:33:02.516589+0000","last_became_active":"2024-09-14T04:32:51.483514+0000","last_became_peered":"2024-09-14T04:32:51.483514+0000","last_unstale":"2024-09-14T04:33:02.516589+0000","last_undegraded":"2024-09-14T04:33:02.516589+0000","last_fullsized":"2024-09-14T04:33:02.516589+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:53.586694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":68,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513794+0000","last_change":"2024-09-14T04:32:14.764177+0000","last_active":"2024-09-14T04:33:03.513794+0000","last_peered":"2024-09-14T04:33:03.513794+0000","last_clean":"2024-09-14T04:33:03.513794+0000","last_became_active":"2024-09-14T04:32:14.763619+0000","last_became_peered":"2024-09-14T04:32:14.763619+0000","last_unstale":"2024-09-14T04:33:03.513794+0000","last_undegraded":"2024-09-14T04:33:03.513794+0000","last_fullsized":"2024-09-14T04:33:03.513794+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:42:47.802221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"76'100","reported_seq":234,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516874+0000","last_change":"2024-09-14T04:31:47.352652+0000","last_active":"2024-09-14T04:33:03.516874+0000","last_peered":"2024-09-14T04:33:03.516874+0000","last_clean":"2024-09-14T04:33:03.516874+0000","last_became_active":"2024-09-14T04:31:47.351579+0000","last_became_peered":"2024-09-14T04:31:47.351579+0000","last_unstale":"2024-09-14T04:33:03.516874+0000","last_undegraded":"2024-09-14T04:33:03.516874+0000","last_fullsized":"2024-09-14T04:33:03.516874+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:43.009173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:43.009173+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:13:48.134273+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":372,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518077+0000","last_change":"2024-09-14T04:32:51.484556+0000","last_active":"2024-09-14T04:33:03.518077+0000","last_peered":"2024-09-14T04:33:03.518077+0000","last_clean":"2024-09-14T04:33:03.518077+0000","last_became_active":"2024-09-14T04:32:51.483616+0000","last_became_peered":"2024-09-14T04:32:51.483616+0000","last_unstale":"2024-09-14T04:33:03.518077+0000","last_undegraded":"2024-09-14T04:33:03.518077+0000","last_fullsized":"2024-09-14T04:33:03.518077+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:01.211635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.519164+0000","last_change":"2024-09-14T04:31:52.377684+0000","last_active":"2024-09-14T04:33:03.519164+0000","last_peered":"2024-09-14T04:33:03.519164+0000","last_clean":"2024-09-14T04:33:03.519164+0000","last_became_active":"2024-09-14T04:31:52.377288+0000","last_became_peered":"2024-09-14T04:31:52.377288+0000","last_unstale":"2024-09-14T04:33:03.519164+0000","last_undegraded":"2024-09-14T04:33:03.519164+0000","last_fullsized":"2024-09-14T04:33:03.519164+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:41:34.260691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":837,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.519257+0000","last_change":"2024-09-14T04:31:59.744302+0000","last_active":"2024-09-14T04:33:03.519257+0000","last_peered":"2024-09-14T04:33:03.519257+0000","last_clean":"2024-09-14T04:33:03.519257+0000","last_became_active":"2024-09-14T04:31:59.730395+0000","last_became_peered":"2024-09-14T04:31:59.730395+0000","last_unstale":"2024-09-14T04:33:03.519257+0000","last_undegraded":"2024-09-14T04:33:03.519257+0000","last_fullsized":"2024-09-14T04:33:03.519257+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:34:44.891719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"76'100","reported_seq":247,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.646647+0000","last_change":"2024-09-14T04:33:03.646609+0000","last_active":"2024-09-14T04:33:03.646647+0000","last_peered":"2024-09-14T04:33:03.646647+0000","last_clean":"2024-09-14T04:33:03.646647+0000","last_became_active":"2024-09-14T04:33:03.519287+0000","last_became_peered":"2024-09-14T04:33:03.519287+0000","last_unstale":"2024-09-14T04:33:03.646647+0000","last_undegraded":"2024-09-14T04:33:03.646647+0000","last_fullsized":"2024-09-14T04:33:03.646647+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:33:03.646544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:33:03.646544+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:51:56.960616+0000","scrub_duration":0.0015616460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":762,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513851+0000","last_change":"2024-09-14T04:32:14.757629+0000","last_active":"2024-09-14T04:33:03.513851+0000","last_peered":"2024-09-14T04:33:03.513851+0000","last_clean":"2024-09-14T04:33:03.513851+0000","last_became_active":"2024-09-14T04:32:14.757116+0000","last_became_peered":"2024-09-14T04:32:14.757116+0000","last_unstale":"2024-09-14T04:33:03.513851+0000","last_undegraded":"2024-09-14T04:33:03.513851+0000","last_fullsized":"2024-09-14T04:33:03.513851+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:46.869240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":57,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516728+0000","last_change":"2024-09-14T04:31:59.734539+0000","last_active":"2024-09-14T04:33:02.516728+0000","last_peered":"2024-09-14T04:33:02.516728+0000","last_clean":"2024-09-14T04:33:02.516728+0000","last_became_active":"2024-09-14T04:31:59.734187+0000","last_became_peered":"2024-09-14T04:31:59.734187+0000","last_unstale":"2024-09-14T04:33:02.516728+0000","last_undegraded":"2024-09-14T04:33:02.516728+0000","last_fullsized":"2024-09-14T04:33:02.516728+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:07:59.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516917+0000","last_change":"2024-09-14T04:31:52.371837+0000","last_active":"2024-09-14T04:33:03.516917+0000","last_peered":"2024-09-14T04:33:03.516917+0000","last_clean":"2024-09-14T04:33:03.516917+0000","last_became_active":"2024-09-14T04:31:52.371505+0000","last_became_peered":"2024-09-14T04:31:52.371505+0000","last_unstale":"2024-09-14T04:33:03.516917+0000","last_undegraded":"2024-09-14T04:33:03.516917+0000","last_fullsized":"2024-09-14T04:33:03.516917+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:05:00.127909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"76'100","reported_seq":384,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515836+0000","last_change":"2024-09-14T04:31:59.734403+0000","last_active":"2024-09-14T04:33:02.515836+0000","last_peered":"2024-09-14T04:33:02.515836+0000","last_clean":"2024-09-14T04:33:02.515836+0000","last_became_active":"2024-09-14T04:31:59.734047+0000","last_became_peered":"2024-09-14T04:31:59.734047+0000","last_unstale":"2024-09-14T04:33:02.515836+0000","last_undegraded":"2024-09-14T04:33:02.515836+0000","last_fullsized":"2024-09-14T04:33:02.515836+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:49.892654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:49.892654+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:46:07.424981+0000","scrub_duration":0.001440715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518495+0000","last_change":"2024-09-14T04:31:52.376118+0000","last_active":"2024-09-14T04:33:03.518495+0000","last_peered":"2024-09-14T04:33:03.518495+0000","last_clean":"2024-09-14T04:33:03.518495+0000","last_became_active":"2024-09-14T04:31:52.374083+0000","last_became_peered":"2024-09-14T04:31:52.374083+0000","last_unstale":"2024-09-14T04:33:03.518495+0000","last_undegraded":"2024-09-14T04:33:03.518495+0000","last_fullsized":"2024-09-14T04:33:03.518495+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:27:32.204002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":69,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516732+0000","last_change":"2024-09-14T04:31:59.735778+0000","last_active":"2024-09-14T04:33:03.516732+0000","last_peered":"2024-09-14T04:33:03.516732+0000","last_clean":"2024-09-14T04:33:03.516732+0000","last_became_active":"2024-09-14T04:31:59.735426+0000","last_became_peered":"2024-09-14T04:31:59.735426+0000","last_unstale":"2024-09-14T04:33:03.516732+0000","last_undegraded":"2024-09-14T04:33:03.516732+0000","last_fullsized":"2024-09-14T04:33:03.516732+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:15:38.789185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":178,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506738+0000","last_change":"2024-09-14T04:25:42.253453+0000","last_active":"2024-09-14T04:33:02.506738+0000","last_peered":"2024-09-14T04:33:02.506738+0000","last_clean":"2024-09-14T04:33:02.506738+0000","last_became_active":"2024-09-14T04:25:42.252529+0000","last_became_peered":"2024-09-14T04:25:42.252529+0000","last_unstale":"2024-09-14T04:33:02.506738+0000","last_undegraded":"2024-09-14T04:33:02.506738+0000","last_fullsized":"2024-09-14T04:33:02.506738+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:19.539802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'100","reported_seq":384,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515842+0000","last_change":"2024-09-14T04:31:47.355339+0000","last_active":"2024-09-14T04:33:02.515842+0000","last_peered":"2024-09-14T04:33:02.515842+0000","last_clean":"2024-09-14T04:33:02.515842+0000","last_became_active":"2024-09-14T04:31:47.354990+0000","last_became_peered":"2024-09-14T04:31:47.354990+0000","last_unstale":"2024-09-14T04:33:02.515842+0000","last_undegraded":"2024-09-14T04:33:02.515842+0000","last_fullsized":"2024-09-14T04:33:02.515842+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:52.943203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:52.943203+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:13:15.560656+0000","scrub_duration":0.0015944450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516830+0000","last_change":"2024-09-14T04:31:52.381785+0000","last_active":"2024-09-14T04:33:03.516830+0000","last_peered":"2024-09-14T04:33:03.516830+0000","last_clean":"2024-09-14T04:33:03.516830+0000","last_became_active":"2024-09-14T04:31:52.381476+0000","last_became_peered":"2024-09-14T04:31:52.381476+0000","last_unstale":"2024-09-14T04:33:03.516830+0000","last_undegraded":"2024-09-14T04:33:03.516830+0000","last_fullsized":"2024-09-14T04:33:03.516830+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:51:45.506882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":284,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.519286+0000","last_change":"2024-09-14T04:33:03.519286+0000","last_active":"2024-09-14T04:33:03.519286+0000","last_peered":"2024-09-14T04:33:03.519286+0000","last_clean":"2024-09-14T04:33:03.519286+0000","last_became_active":"2024-09-14T04:33:03.518920+0000","last_became_peered":"2024-09-14T04:33:03.518920+0000","last_unstale":"2024-09-14T04:33:03.519286+0000","last_undegraded":"2024-09-14T04:33:03.519286+0000","last_fullsized":"2024-09-14T04:33:03.519286+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:36:14.512397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":204,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506491+0000","last_change":"2024-09-14T04:31:52.388705+0000","last_active":"2024-09-14T04:33:02.506491+0000","last_peered":"2024-09-14T04:33:02.506491+0000","last_clean":"2024-09-14T04:33:02.506491+0000","last_became_active":"2024-09-14T04:31:52.373430+0000","last_became_peered":"2024-09-14T04:31:52.373430+0000","last_unstale":"2024-09-14T04:33:02.506491+0000","last_undegraded":"2024-09-14T04:33:02.506491+0000","last_fullsized":"2024-09-14T04:33:02.506491+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:29:17.445736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'100","reported_seq":139,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506389+0000","last_change":"2024-09-14T04:31:59.725675+0000","last_active":"2024-09-14T04:33:02.506389+0000","last_peered":"2024-09-14T04:33:02.506389+0000","last_clean":"2024-09-14T04:33:02.506389+0000","last_became_active":"2024-09-14T04:31:59.725307+0000","last_became_peered":"2024-09-14T04:31:59.725307+0000","last_unstale":"2024-09-14T04:33:02.506389+0000","last_undegraded":"2024-09-14T04:33:02.506389+0000","last_fullsized":"2024-09-14T04:33:02.506389+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:49.280129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:49.280129+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:52:25.350374+0000","scrub_duration":0.001556279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":69,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517017+0000","last_change":"2024-09-14T04:31:59.734619+0000","last_active":"2024-09-14T04:33:03.517017+0000","last_peered":"2024-09-14T04:33:03.517017+0000","last_clean":"2024-09-14T04:33:03.517017+0000","last_became_active":"2024-09-14T04:31:59.734287+0000","last_became_peered":"2024-09-14T04:31:59.734287+0000","last_unstale":"2024-09-14T04:33:03.517017+0000","last_undegraded":"2024-09-14T04:33:03.517017+0000","last_fullsized":"2024-09-14T04:33:03.517017+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:03:39.070049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518159+0000","last_change":"2024-09-14T04:31:52.372820+0000","last_active":"2024-09-14T04:33:03.518159+0000","last_peered":"2024-09-14T04:33:03.518159+0000","last_clean":"2024-09-14T04:33:03.518159+0000","last_became_active":"2024-09-14T04:31:52.371186+0000","last_became_peered":"2024-09-14T04:31:52.371186+0000","last_unstale":"2024-09-14T04:33:03.518159+0000","last_undegraded":"2024-09-14T04:33:03.518159+0000","last_fullsized":"2024-09-14T04:33:03.518159+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:35:56.106257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":260,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517909+0000","last_change":"2024-09-14T04:31:59.725253+0000","last_active":"2024-09-14T04:33:03.517909+0000","last_peered":"2024-09-14T04:33:03.517909+0000","last_clean":"2024-09-14T04:33:03.517909+0000","last_became_active":"2024-09-14T04:31:59.724889+0000","last_became_peered":"2024-09-14T04:31:59.724889+0000","last_unstale":"2024-09-14T04:33:03.517909+0000","last_undegraded":"2024-09-14T04:33:03.517909+0000","last_fullsized":"2024-09-14T04:33:03.517909+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:18:39.083123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"76'100","reported_seq":430,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.513878+0000","last_change":"2024-09-14T04:32:56.492141+0000","last_active":"2024-09-14T04:33:02.513878+0000","last_peered":"2024-09-14T04:33:02.513878+0000","last_clean":"2024-09-14T04:33:02.513878+0000","last_became_active":"2024-09-14T04:32:56.491571+0000","last_became_peered":"2024-09-14T04:32:56.491571+0000","last_unstale":"2024-09-14T04:33:02.513878+0000","last_undegraded":"2024-09-14T04:33:02.513878+0000","last_fullsized":"2024-09-14T04:33:02.513878+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:44.070022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:44.070022+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:25:57.896234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":64,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513529+0000","last_change":"2024-09-14T04:32:14.763887+0000","last_active":"2024-09-14T04:33:03.513529+0000","last_peered":"2024-09-14T04:33:03.513529+0000","last_clean":"2024-09-14T04:33:03.513529+0000","last_became_active":"2024-09-14T04:32:14.762741+0000","last_became_peered":"2024-09-14T04:32:14.762741+0000","last_unstale":"2024-09-14T04:33:03.513529+0000","last_undegraded":"2024-09-14T04:33:03.513529+0000","last_fullsized":"2024-09-14T04:33:03.513529+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:40:15.683940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":647,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513085+0000","last_change":"2024-09-14T04:32:14.760606+0000","last_active":"2024-09-14T04:33:03.513085+0000","last_peered":"2024-09-14T04:33:03.513085+0000","last_clean":"2024-09-14T04:33:03.513085+0000","last_became_active":"2024-09-14T04:32:14.760339+0000","last_became_peered":"2024-09-14T04:32:14.760339+0000","last_unstale":"2024-09-14T04:33:03.513085+0000","last_undegraded":"2024-09-14T04:33:03.513085+0000","last_fullsized":"2024-09-14T04:33:03.513085+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:04:30.093850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":238,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.513762+0000","last_change":"2024-09-14T04:32:56.492593+0000","last_active":"2024-09-14T04:33:02.513762+0000","last_peered":"2024-09-14T04:33:02.513762+0000","last_clean":"2024-09-14T04:33:02.513762+0000","last_became_active":"2024-09-14T04:32:56.492236+0000","last_became_peered":"2024-09-14T04:32:56.492236+0000","last_unstale":"2024-09-14T04:33:02.513762+0000","last_undegraded":"2024-09-14T04:33:02.513762+0000","last_fullsized":"2024-09-14T04:33:02.513762+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:35:06.692185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'100","reported_seq":71,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517201+0000","last_change":"2024-09-14T04:31:54.205323+0000","last_active":"2024-09-14T04:33:03.517201+0000","last_peered":"2024-09-14T04:33:03.517201+0000","last_clean":"2024-09-14T04:33:03.517201+0000","last_became_active":"2024-09-14T04:31:52.377635+0000","last_became_peered":"2024-09-14T04:31:52.377635+0000","last_unstale":"2024-09-14T04:33:03.517201+0000","last_undegraded":"2024-09-14T04:33:03.517201+0000","last_fullsized":"2024-09-14T04:33:03.517201+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:31:54.205276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:31:54.205276+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:10:27.063913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":470,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515961+0000","last_change":"2024-09-14T04:31:47.354575+0000","last_active":"2024-09-14T04:33:02.515961+0000","last_peered":"2024-09-14T04:33:02.515961+0000","last_clean":"2024-09-14T04:33:02.515961+0000","last_became_active":"2024-09-14T04:31:47.354277+0000","last_became_peered":"2024-09-14T04:31:47.354277+0000","last_unstale":"2024-09-14T04:33:02.515961+0000","last_undegraded":"2024-09-14T04:33:02.515961+0000","last_fullsized":"2024-09-14T04:33:02.515961+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:34:51.518685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":592,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518603+0000","last_change":"2024-09-14T04:22:08.472386+0000","last_active":"2024-09-14T04:33:03.518603+0000","last_peered":"2024-09-14T04:33:03.518603+0000","last_clean":"2024-09-14T04:33:03.518603+0000","last_became_active":"2024-09-14T04:22:08.472100+0000","last_became_peered":"2024-09-14T04:22:08.472100+0000","last_unstale":"2024-09-14T04:33:03.518603+0000","last_undegraded":"2024-09-14T04:33:03.518603+0000","last_fullsized":"2024-09-14T04:33:03.518603+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:36:03.114984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":215,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506888+0000","last_change":"2024-09-14T04:31:47.378091+0000","last_active":"2024-09-14T04:33:02.506888+0000","last_peered":"2024-09-14T04:33:02.506888+0000","last_clean":"2024-09-14T04:33:02.506888+0000","last_became_active":"2024-09-14T04:31:47.357003+0000","last_became_peered":"2024-09-14T04:31:47.357003+0000","last_unstale":"2024-09-14T04:33:02.506888+0000","last_undegraded":"2024-09-14T04:33:02.506888+0000","last_fullsized":"2024-09-14T04:33:02.506888+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:23:33.997689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":355,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.513678+0000","last_change":"2024-09-14T04:32:56.492971+0000","last_active":"2024-09-14T04:33:02.513678+0000","last_peered":"2024-09-14T04:33:02.513678+0000","last_clean":"2024-09-14T04:33:02.513678+0000","last_became_active":"2024-09-14T04:32:56.492543+0000","last_became_peered":"2024-09-14T04:32:56.492543+0000","last_unstale":"2024-09-14T04:33:02.513678+0000","last_undegraded":"2024-09-14T04:33:02.513678+0000","last_fullsized":"2024-09-14T04:33:02.513678+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:50.911016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:50.911016+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:36:25.564886+0000","scrub_duration":0.0014927930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517103+0000","last_change":"2024-09-14T04:31:52.381945+0000","last_active":"2024-09-14T04:33:03.517103+0000","last_peered":"2024-09-14T04:33:03.517103+0000","last_clean":"2024-09-14T04:33:03.517103+0000","last_became_active":"2024-09-14T04:31:52.381229+0000","last_became_peered":"2024-09-14T04:31:52.381229+0000","last_unstale":"2024-09-14T04:33:03.517103+0000","last_undegraded":"2024-09-14T04:33:03.517103+0000","last_fullsized":"2024-09-14T04:33:03.517103+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:19:03.665009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":606,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518280+0000","last_change":"2024-09-14T04:31:52.383693+0000","last_active":"2024-09-14T04:33:03.518280+0000","last_peered":"2024-09-14T04:33:03.518280+0000","last_clean":"2024-09-14T04:33:03.518280+0000","last_became_active":"2024-09-14T04:31:52.383221+0000","last_became_peered":"2024-09-14T04:31:52.383221+0000","last_unstale":"2024-09-14T04:33:03.518280+0000","last_undegraded":"2024-09-14T04:33:03.518280+0000","last_fullsized":"2024-09-14T04:33:03.518280+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:27:19.751816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'100","reported_seq":109,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506737+0000","last_change":"2024-09-14T04:27:35.272869+0000","last_active":"2024-09-14T04:33:02.506737+0000","last_peered":"2024-09-14T04:33:02.506737+0000","last_clean":"2024-09-14T04:33:02.506737+0000","last_became_active":"2024-09-14T04:27:12.688026+0000","last_became_peered":"2024-09-14T04:27:12.688026+0000","last_unstale":"2024-09-14T04:33:02.506737+0000","last_undegraded":"2024-09-14T04:33:02.506737+0000","last_fullsized":"2024-09-14T04:33:02.506737+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:35.272785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:35.272785+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:00:40.802976+0000","scrub_duration":0.0029595770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":339,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.521874+0000","last_change":"2024-09-14T04:33:03.521874+0000","last_active":"2024-09-14T04:33:03.521874+0000","last_peered":"2024-09-14T04:33:03.521874+0000","last_clean":"2024-09-14T04:33:03.521874+0000","last_became_active":"2024-09-14T04:33:03.520785+0000","last_became_peered":"2024-09-14T04:33:03.520785+0000","last_unstale":"2024-09-14T04:33:03.521874+0000","last_undegraded":"2024-09-14T04:33:03.521874+0000","last_fullsized":"2024-09-14T04:33:03.521874+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:56.448978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":337,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.522346+0000","last_change":"2024-09-14T04:33:03.522346+0000","last_active":"2024-09-14T04:33:03.522346+0000","last_peered":"2024-09-14T04:33:03.522346+0000","last_clean":"2024-09-14T04:33:03.522346+0000","last_became_active":"2024-09-14T04:33:03.521931+0000","last_became_peered":"2024-09-14T04:33:03.521931+0000","last_unstale":"2024-09-14T04:33:03.522346+0000","last_undegraded":"2024-09-14T04:33:03.522346+0000","last_fullsized":"2024-09-14T04:33:03.522346+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:37:11.901738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"76'100","reported_seq":78,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517429+0000","last_change":"2024-09-14T04:31:59.739551+0000","last_active":"2024-09-14T04:33:03.517429+0000","last_peered":"2024-09-14T04:33:03.517429+0000","last_clean":"2024-09-14T04:33:03.517429+0000","last_became_active":"2024-09-14T04:31:59.739173+0000","last_became_peered":"2024-09-14T04:31:59.739173+0000","last_unstale":"2024-09-14T04:33:03.517429+0000","last_undegraded":"2024-09-14T04:33:03.517429+0000","last_fullsized":"2024-09-14T04:33:03.517429+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:31:53.189290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:31:53.189290+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:19:15.851448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":279,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.520972+0000","last_change":"2024-09-14T04:33:03.520972+0000","last_active":"2024-09-14T04:33:03.520972+0000","last_peered":"2024-09-14T04:33:03.520972+0000","last_clean":"2024-09-14T04:33:03.520972+0000","last_became_active":"2024-09-14T04:33:03.519956+0000","last_became_peered":"2024-09-14T04:33:03.519956+0000","last_unstale":"2024-09-14T04:33:03.520972+0000","last_undegraded":"2024-09-14T04:33:03.520972+0000","last_fullsized":"2024-09-14T04:33:03.520972+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:50:33.233796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":478,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515568+0000","last_change":"2024-09-14T04:31:52.372211+0000","last_active":"2024-09-14T04:33:02.515568+0000","last_peered":"2024-09-14T04:33:02.515568+0000","last_clean":"2024-09-14T04:33:02.515568+0000","last_became_active":"2024-09-14T04:31:52.371900+0000","last_became_peered":"2024-09-14T04:31:52.371900+0000","last_unstale":"2024-09-14T04:33:02.515568+0000","last_undegraded":"2024-09-14T04:33:02.515568+0000","last_fullsized":"2024-09-14T04:33:02.515568+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:42:24.165090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"76'100","reported_seq":114,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506354+0000","last_change":"2024-09-14T04:27:46.144096+0000","last_active":"2024-09-14T04:33:02.506354+0000","last_peered":"2024-09-14T04:33:02.506354+0000","last_clean":"2024-09-14T04:33:02.506354+0000","last_became_active":"2024-09-14T04:27:12.688140+0000","last_became_peered":"2024-09-14T04:27:12.688140+0000","last_unstale":"2024-09-14T04:33:02.506354+0000","last_undegraded":"2024-09-14T04:33:02.506354+0000","last_fullsized":"2024-09-14T04:33:02.506354+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:46.144003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:46.144003+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:00:07.652536+0000","scrub_duration":0.0071911659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":282,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.520700+0000","last_change":"2024-09-14T04:33:03.520700+0000","last_active":"2024-09-14T04:33:03.520700+0000","last_peered":"2024-09-14T04:33:03.520700+0000","last_clean":"2024-09-14T04:33:03.520700+0000","last_became_active":"2024-09-14T04:33:03.520294+0000","last_became_peered":"2024-09-14T04:33:03.520294+0000","last_unstale":"2024-09-14T04:33:03.520700+0000","last_undegraded":"2024-09-14T04:33:03.520700+0000","last_fullsized":"2024-09-14T04:33:03.520700+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:39:41.254063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":858,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518820+0000","last_change":"2024-09-14T04:31:59.754533+0000","last_active":"2024-09-14T04:33:03.518820+0000","last_peered":"2024-09-14T04:33:03.518820+0000","last_clean":"2024-09-14T04:33:03.518820+0000","last_became_active":"2024-09-14T04:31:59.731161+0000","last_became_peered":"2024-09-14T04:31:59.731161+0000","last_unstale":"2024-09-14T04:33:03.518820+0000","last_undegraded":"2024-09-14T04:33:03.518820+0000","last_fullsized":"2024-09-14T04:33:03.518820+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:27:06.084373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'100","reported_seq":165,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516312+0000","last_change":"2024-09-14T04:32:56.489379+0000","last_active":"2024-09-14T04:33:02.516312+0000","last_peered":"2024-09-14T04:33:02.516312+0000","last_clean":"2024-09-14T04:33:02.516312+0000","last_became_active":"2024-09-14T04:32:56.489045+0000","last_became_peered":"2024-09-14T04:32:56.489045+0000","last_unstale":"2024-09-14T04:33:02.516312+0000","last_undegraded":"2024-09-14T04:33:02.516312+0000","last_fullsized":"2024-09-14T04:33:02.516312+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:43.390999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:43.390999+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:32:49.743497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":64,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513221+0000","last_change":"2024-09-14T04:32:14.762919+0000","last_active":"2024-09-14T04:33:03.513221+0000","last_peered":"2024-09-14T04:33:03.513221+0000","last_clean":"2024-09-14T04:33:03.513221+0000","last_became_active":"2024-09-14T04:32:14.762605+0000","last_became_peered":"2024-09-14T04:32:14.762605+0000","last_unstale":"2024-09-14T04:33:03.513221+0000","last_undegraded":"2024-09-14T04:33:03.513221+0000","last_fullsized":"2024-09-14T04:33:03.513221+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:04:35.781173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":158,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506859+0000","last_change":"2024-09-14T04:22:08.468750+0000","last_active":"2024-09-14T04:33:02.506859+0000","last_peered":"2024-09-14T04:33:02.506859+0000","last_clean":"2024-09-14T04:33:02.506859+0000","last_became_active":"2024-09-14T04:22:08.468217+0000","last_became_peered":"2024-09-14T04:22:08.468217+0000","last_unstale":"2024-09-14T04:33:02.506859+0000","last_undegraded":"2024-09-14T04:33:02.506859+0000","last_fullsized":"2024-09-14T04:33:02.506859+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:06.709182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"76'100","reported_seq":247,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:05.611903+0000","last_change":"2024-09-14T04:33:05.611870+0000","last_active":"2024-09-14T04:33:05.611903+0000","last_peered":"2024-09-14T04:33:05.611903+0000","last_clean":"2024-09-14T04:33:05.611903+0000","last_became_active":"2024-09-14T04:33:03.521236+0000","last_became_peered":"2024-09-14T04:33:03.521236+0000","last_unstale":"2024-09-14T04:33:05.611903+0000","last_undegraded":"2024-09-14T04:33:05.611903+0000","last_fullsized":"2024-09-14T04:33:05.611903+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:33:05.611796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:33:05.611796+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:37:42.178717+0000","scrub_duration":0.0016521800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":507,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515374+0000","last_change":"2024-09-14T04:31:52.372957+0000","last_active":"2024-09-14T04:33:02.515374+0000","last_peered":"2024-09-14T04:33:02.515374+0000","last_clean":"2024-09-14T04:33:02.515374+0000","last_became_active":"2024-09-14T04:31:52.372585+0000","last_became_peered":"2024-09-14T04:31:52.372585+0000","last_unstale":"2024-09-14T04:33:02.515374+0000","last_undegraded":"2024-09-14T04:33:02.515374+0000","last_fullsized":"2024-09-14T04:33:02.515374+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:32:03.907839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":166,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516547+0000","last_change":"2024-09-14T04:32:56.493160+0000","last_active":"2024-09-14T04:33:02.516547+0000","last_peered":"2024-09-14T04:33:02.516547+0000","last_clean":"2024-09-14T04:33:02.516547+0000","last_became_active":"2024-09-14T04:32:56.492387+0000","last_became_peered":"2024-09-14T04:32:56.492387+0000","last_unstale":"2024-09-14T04:33:02.516547+0000","last_undegraded":"2024-09-14T04:33:02.516547+0000","last_fullsized":"2024-09-14T04:33:02.516547+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:11:52.363513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'100","reported_seq":383,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515940+0000","last_change":"2024-09-14T04:30:54.960873+0000","last_active":"2024-09-14T04:33:02.515940+0000","last_peered":"2024-09-14T04:33:02.515940+0000","last_clean":"2024-09-14T04:33:02.515940+0000","last_became_active":"2024-09-14T04:30:52.535921+0000","last_became_peered":"2024-09-14T04:30:52.535921+0000","last_unstale":"2024-09-14T04:33:02.515940+0000","last_undegraded":"2024-09-14T04:33:02.515940+0000","last_fullsized":"2024-09-14T04:33:02.515940+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:54.960754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:54.960754+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:41:30.181896+0000","scrub_duration":0.001686597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":315,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.522025+0000","last_change":"2024-09-14T04:33:03.522025+0000","last_active":"2024-09-14T04:33:03.522025+0000","last_peered":"2024-09-14T04:33:03.522025+0000","last_clean":"2024-09-14T04:33:03.522025+0000","last_became_active":"2024-09-14T04:33:03.520351+0000","last_became_peered":"2024-09-14T04:33:03.520351+0000","last_unstale":"2024-09-14T04:33:03.522025+0000","last_undegraded":"2024-09-14T04:33:03.522025+0000","last_fullsized":"2024-09-14T04:33:03.522025+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:28.539397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":139,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.513519+0000","last_change":"2024-09-14T04:32:56.493073+0000","last_active":"2024-09-14T04:33:02.513519+0000","last_peered":"2024-09-14T04:33:02.513519+0000","last_clean":"2024-09-14T04:33:02.513519+0000","last_became_active":"2024-09-14T04:32:56.492691+0000","last_became_peered":"2024-09-14T04:32:56.492691+0000","last_unstale":"2024-09-14T04:33:02.513519+0000","last_undegraded":"2024-09-14T04:33:02.513519+0000","last_fullsized":"2024-09-14T04:33:02.513519+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:24:50.118027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'100","reported_seq":248,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516930+0000","last_change":"2024-09-14T04:31:59.756931+0000","last_active":"2024-09-14T04:33:03.516930+0000","last_peered":"2024-09-14T04:33:03.516930+0000","last_clean":"2024-09-14T04:33:03.516930+0000","last_became_active":"2024-09-14T04:31:59.733293+0000","last_became_peered":"2024-09-14T04:31:59.733293+0000","last_unstale":"2024-09-14T04:33:03.516930+0000","last_undegraded":"2024-09-14T04:33:03.516930+0000","last_fullsized":"2024-09-14T04:33:03.516930+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:45.041196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:45.041196+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:53:43.086319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":178,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506503+0000","last_change":"2024-09-14T04:32:51.483935+0000","last_active":"2024-09-14T04:33:02.506503+0000","last_peered":"2024-09-14T04:33:02.506503+0000","last_clean":"2024-09-14T04:33:02.506503+0000","last_became_active":"2024-09-14T04:32:51.483539+0000","last_became_peered":"2024-09-14T04:32:51.483539+0000","last_unstale":"2024-09-14T04:33:02.506503+0000","last_undegraded":"2024-09-14T04:33:02.506503+0000","last_fullsized":"2024-09-14T04:33:02.506503+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:33:13.451628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":480,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516040+0000","last_change":"2024-09-14T04:32:56.491717+0000","last_active":"2024-09-14T04:33:02.516040+0000","last_peered":"2024-09-14T04:33:02.516040+0000","last_clean":"2024-09-14T04:33:02.516040+0000","last_became_active":"2024-09-14T04:32:56.491154+0000","last_became_peered":"2024-09-14T04:32:56.491154+0000","last_unstale":"2024-09-14T04:33:02.516040+0000","last_undegraded":"2024-09-14T04:33:02.516040+0000","last_fullsized":"2024-09-14T04:33:02.516040+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:13:16.238085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"76'100","reported_seq":66,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516404+0000","last_change":"2024-09-14T04:32:56.492577+0000","last_active":"2024-09-14T04:33:02.516404+0000","last_peered":"2024-09-14T04:33:02.516404+0000","last_clean":"2024-09-14T04:33:02.516404+0000","last_became_active":"2024-09-14T04:32:56.492207+0000","last_became_peered":"2024-09-14T04:32:56.492207+0000","last_unstale":"2024-09-14T04:33:02.516404+0000","last_undegraded":"2024-09-14T04:33:02.516404+0000","last_fullsized":"2024-09-14T04:33:02.516404+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:45.083429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:45.083429+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:26:20.206726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":526,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515312+0000","last_change":"2024-09-14T04:31:59.750713+0000","last_active":"2024-09-14T04:33:02.515312+0000","last_peered":"2024-09-14T04:33:02.515312+0000","last_clean":"2024-09-14T04:33:02.515312+0000","last_became_active":"2024-09-14T04:31:59.735685+0000","last_became_peered":"2024-09-14T04:31:59.735685+0000","last_unstale":"2024-09-14T04:33:02.515312+0000","last_undegraded":"2024-09-14T04:33:02.515312+0000","last_fullsized":"2024-09-14T04:33:02.515312+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:59:15.857747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":618,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.519089+0000","last_change":"2024-09-14T04:32:51.483901+0000","last_active":"2024-09-14T04:33:03.519089+0000","last_peered":"2024-09-14T04:33:03.519089+0000","last_clean":"2024-09-14T04:33:03.519089+0000","last_became_active":"2024-09-14T04:32:51.483541+0000","last_became_peered":"2024-09-14T04:32:51.483541+0000","last_unstale":"2024-09-14T04:33:03.519089+0000","last_undegraded":"2024-09-14T04:33:03.519089+0000","last_fullsized":"2024-09-14T04:33:03.519089+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:38:59.377719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"76'100","reported_seq":283,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506402+0000","last_change":"2024-09-14T04:31:52.372540+0000","last_active":"2024-09-14T04:33:02.506402+0000","last_peered":"2024-09-14T04:33:02.506402+0000","last_clean":"2024-09-14T04:33:02.506402+0000","last_became_active":"2024-09-14T04:31:52.372198+0000","last_became_peered":"2024-09-14T04:31:52.372198+0000","last_unstale":"2024-09-14T04:33:02.506402+0000","last_undegraded":"2024-09-14T04:33:02.506402+0000","last_fullsized":"2024-09-14T04:33:02.506402+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:31:01.287655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:31:01.287655+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:25:12.147394+0000","scrub_duration":0.0014619590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"76'100","reported_seq":63,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517338+0000","last_change":"2024-09-14T04:31:52.382156+0000","last_active":"2024-09-14T04:33:03.517338+0000","last_peered":"2024-09-14T04:33:03.517338+0000","last_clean":"2024-09-14T04:33:03.517338+0000","last_became_active":"2024-09-14T04:31:52.381538+0000","last_became_peered":"2024-09-14T04:31:52.381538+0000","last_unstale":"2024-09-14T04:33:03.517338+0000","last_undegraded":"2024-09-14T04:33:03.517338+0000","last_fullsized":"2024-09-14T04:33:03.517338+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:53.041045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:53.041045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:46:54.554799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":435,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516189+0000","last_change":"2024-09-14T04:32:56.492552+0000","last_active":"2024-09-14T04:33:02.516189+0000","last_peered":"2024-09-14T04:33:02.516189+0000","last_clean":"2024-09-14T04:33:02.516189+0000","last_became_active":"2024-09-14T04:32:56.492242+0000","last_became_peered":"2024-09-14T04:32:56.492242+0000","last_unstale":"2024-09-14T04:33:02.516189+0000","last_undegraded":"2024-09-14T04:33:02.516189+0000","last_fullsized":"2024-09-14T04:33:02.516189+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:51:44.242839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":124,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513908+0000","last_change":"2024-09-14T04:32:51.484511+0000","last_active":"2024-09-14T04:33:03.513908+0000","last_peered":"2024-09-14T04:33:03.513908+0000","last_clean":"2024-09-14T04:33:03.513908+0000","last_became_active":"2024-09-14T04:32:51.484154+0000","last_became_peered":"2024-09-14T04:32:51.484154+0000","last_unstale":"2024-09-14T04:33:03.513908+0000","last_undegraded":"2024-09-14T04:33:03.513908+0000","last_fullsized":"2024-09-14T04:33:03.513908+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:08:45.125779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":625,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513232+0000","last_change":"2024-09-14T04:32:14.759925+0000","last_active":"2024-09-14T04:33:03.513232+0000","last_peered":"2024-09-14T04:33:03.513232+0000","last_clean":"2024-09-14T04:33:03.513232+0000","last_became_active":"2024-09-14T04:32:14.759599+0000","last_became_peered":"2024-09-14T04:32:14.759599+0000","last_unstale":"2024-09-14T04:33:03.513232+0000","last_undegraded":"2024-09-14T04:33:03.513232+0000","last_fullsized":"2024-09-14T04:33:03.513232+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:00:51.683960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":188,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506806+0000","last_change":"2024-09-14T04:31:47.357642+0000","last_active":"2024-09-14T04:33:02.506806+0000","last_peered":"2024-09-14T04:33:02.506806+0000","last_clean":"2024-09-14T04:33:02.506806+0000","last_became_active":"2024-09-14T04:31:47.356307+0000","last_became_peered":"2024-09-14T04:31:47.356307+0000","last_unstale":"2024-09-14T04:33:02.506806+0000","last_undegraded":"2024-09-14T04:33:02.506806+0000","last_fullsized":"2024-09-14T04:33:02.506806+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:23:39.439300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"76'100","reported_seq":74,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506360+0000","last_change":"2024-09-14T04:31:47.362183+0000","last_active":"2024-09-14T04:33:02.506360+0000","last_peered":"2024-09-14T04:33:02.506360+0000","last_clean":"2024-09-14T04:33:02.506360+0000","last_became_active":"2024-09-14T04:31:47.361606+0000","last_became_peered":"2024-09-14T04:31:47.361606+0000","last_unstale":"2024-09-14T04:33:02.506360+0000","last_undegraded":"2024-09-14T04:33:02.506360+0000","last_fullsized":"2024-09-14T04:33:02.506360+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:31:02.298329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:31:02.298329+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:52:41.585691+0000","scrub_duration":0.0012390960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":43,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.514865+0000","last_change":"2024-09-14T04:31:47.370767+0000","last_active":"2024-09-14T04:33:02.514865+0000","last_peered":"2024-09-14T04:33:02.514865+0000","last_clean":"2024-09-14T04:33:02.514865+0000","last_became_active":"2024-09-14T04:31:47.353197+0000","last_became_peered":"2024-09-14T04:31:47.353197+0000","last_unstale":"2024-09-14T04:33:02.514865+0000","last_undegraded":"2024-09-14T04:33:02.514865+0000","last_fullsized":"2024-09-14T04:33:02.514865+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:17:06.116532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":798,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517191+0000","last_change":"2024-09-14T04:31:52.382891+0000","last_active":"2024-09-14T04:33:03.517191+0000","last_peered":"2024-09-14T04:33:03.517191+0000","last_clean":"2024-09-14T04:33:03.517191+0000","last_became_active":"2024-09-14T04:31:52.382589+0000","last_became_peered":"2024-09-14T04:31:52.382589+0000","last_unstale":"2024-09-14T04:33:03.517191+0000","last_undegraded":"2024-09-14T04:33:03.517191+0000","last_fullsized":"2024-09-14T04:33:03.517191+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:19:13.265551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'100","reported_seq":175,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513239+0000","last_change":"2024-09-14T04:32:14.761740+0000","last_active":"2024-09-14T04:33:03.513239+0000","last_peered":"2024-09-14T04:33:03.513239+0000","last_clean":"2024-09-14T04:33:03.513239+0000","last_became_active":"2024-09-14T04:32:14.761407+0000","last_became_peered":"2024-09-14T04:32:14.761407+0000","last_unstale":"2024-09-14T04:33:03.513239+0000","last_undegraded":"2024-09-14T04:33:03.513239+0000","last_fullsized":"2024-09-14T04:33:03.513239+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:58.248525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:58.248525+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:37:49.472825+0000","scrub_duration":0.0016764589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":6942,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430842,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1511700,"kb_used_data":5828,"kb_used_omap":1,"kb_used_meta":1505854,"kb_avail":92233452,"statfs":{"total":95995035648,"available":94447054848,"internally_reserved":0,"allocated":5967872,"data_stored":3732555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1541994840},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388627071,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1511044,"kb_used_data":6136,"kb_used_omap":1,"kb_used_meta":1504894,"kb_avail":92234108,"statfs":{"total":95995035648,"available":94447726592,"internally_reserved":0,"allocated":6283264,"data_stored":4059793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1541011800},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823298,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2088504,"kb_used_data":6064,"kb_used_omap":1,"kb_used_meta":2082430,"kb_avail":91656648,"statfs":{"total":95995035648,"available":93856407552,"internally_reserved":0,"allocated":6209536,"data_stored":3990312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":2132408664},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917381,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":944584,"kb_used_data":5820,"kb_used_omap":6,"kb_used_meta":938745,"kb_avail":92800568,"statfs":{"total":95995035648,"available":95027781632,"internally_reserved":0,"allocated":5959680,"data_stored":3732170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":961275903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552679,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267008,"kb_used_data":6384,"kb_used_omap":4,"kb_used_meta":1260603,"kb_avail":92478144,"statfs":{"total":95995035648,"available":94697619456,"internally_reserved":0,"allocated":6537216,"data_stored":4318073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":1290857917},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1280508,"kb_used_data":6060,"kb_used_omap":6,"kb_used_meta":1274425,"kb_avail":92464644,"statfs":{"total":95995035648,"available":94683795456,"internally_reserved":0,"allocated":6205440,"data_stored":3989445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":1305011661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828191,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1081852,"kb_used_data":5804,"kb_used_omap":5,"kb_used_meta":1076026,"kb_avail":92663300,"statfs":{"total":95995035648,"available":94887219200,"internally_reserved":0,"allocated":5943296,"data_stored":3727634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1101850863},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597384179,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1189948,"kb_used_data":6124,"kb_used_omap":5,"kb_used_meta":1183802,"kb_avail":92555204,"statfs":{"total":95995035648,"available":94776528896,"internally_reserved":0,"allocated":6270976,"data_stored":4055588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":1212213308},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:33:50.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63f4e762-724a-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-14T04:33:51.187 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/mon.a/config 2024-09-14T04:33:51.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:51 smithi114 ceph-mon[106969]: from='client.47630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:33:51.619 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:51 smithi114 ceph-mon[103029]: from='client.47630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:33:51.619 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:51 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: ::ffff:172.21.15.160 - - [14/Sep/2024:04:33:51] "GET /metrics HTTP/1.1" 200 34140 "" "Prometheus/2.51.0" 2024-09-14T04:33:51.657 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:51 smithi160 ceph-mon[88407]: from='client.47630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:33:52.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:52 smithi114 ceph-mon[103029]: pgmap v1749: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:52.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:52 smithi114 ceph-mon[106969]: pgmap v1749: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:52.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:52 smithi160 ceph-mon[88407]: pgmap v1749: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:53.118 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-14T04:33:53.119 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-14T04:33:53.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:53 smithi114 ceph-mon[103029]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:53.620 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:53 smithi114 ceph-mon[106969]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:53.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:53 smithi160 ceph-mon[88407]: from='mgr.34107 172.21.15.114:0/3286006007' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-14T04:33:53.875 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":1749,"stamp":"2024-09-14T04:33:51.291281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":662378,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2158,"ondisk_log_size":2158,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10875148,"kb_used_data":48220,"kb_used_omap":33,"kb_used_meta":10826782,"kb_avail":739086068,"statfs":{"total":767960285184,"available":756824133632,"internally_reserved":0,"allocated":49377280,"data_stored":31605570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34628,"internal_metadata":11086624956},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003701"},"pg_stats":[{"pgid":"3.1f","version":"62'4","reported_seq":774,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513865+0000","last_change":"2024-09-14T04:32:14.758999+0000","last_active":"2024-09-14T04:33:03.513865+0000","last_peered":"2024-09-14T04:33:03.513865+0000","last_clean":"2024-09-14T04:33:03.513865+0000","last_became_active":"2024-09-14T04:32:14.758383+0000","last_became_peered":"2024-09-14T04:32:14.758383+0000","last_unstale":"2024-09-14T04:33:03.513865+0000","last_undegraded":"2024-09-14T04:33:03.513865+0000","last_fullsized":"2024-09-14T04:33:03.513865+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:21:59.358588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":275,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.520443+0000","last_change":"2024-09-14T04:33:03.520443+0000","last_active":"2024-09-14T04:33:03.520443+0000","last_peered":"2024-09-14T04:33:03.520443+0000","last_clean":"2024-09-14T04:33:03.520443+0000","last_became_active":"2024-09-14T04:33:03.520131+0000","last_became_peered":"2024-09-14T04:33:03.520131+0000","last_unstale":"2024-09-14T04:33:03.520443+0000","last_undegraded":"2024-09-14T04:33:03.520443+0000","last_fullsized":"2024-09-14T04:33:03.520443+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:27:25.194028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"62'1","reported_seq":607,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517708+0000","last_change":"2024-09-14T04:25:42.235102+0000","last_active":"2024-09-14T04:33:03.517708+0000","last_peered":"2024-09-14T04:33:03.517708+0000","last_clean":"2024-09-14T04:33:03.517708+0000","last_became_active":"2024-09-14T04:25:42.234776+0000","last_became_peered":"2024-09-14T04:25:42.234776+0000","last_unstale":"2024-09-14T04:33:03.517708+0000","last_undegraded":"2024-09-14T04:33:03.517708+0000","last_fullsized":"2024-09-14T04:33:03.517708+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:59:52.026651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":604,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516734+0000","last_change":"2024-09-14T04:31:59.732915+0000","last_active":"2024-09-14T04:33:03.516734+0000","last_peered":"2024-09-14T04:33:03.516734+0000","last_clean":"2024-09-14T04:33:03.516734+0000","last_became_active":"2024-09-14T04:31:59.732583+0000","last_became_peered":"2024-09-14T04:31:59.732583+0000","last_unstale":"2024-09-14T04:33:03.516734+0000","last_undegraded":"2024-09-14T04:33:03.516734+0000","last_fullsized":"2024-09-14T04:33:03.516734+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:38:28.751668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"62'2","reported_seq":178,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517493+0000","last_change":"2024-09-14T04:27:55.405108+0000","last_active":"2024-09-14T04:33:03.517493+0000","last_peered":"2024-09-14T04:33:03.517493+0000","last_clean":"2024-09-14T04:33:03.517493+0000","last_became_active":"2024-09-14T04:27:55.394475+0000","last_became_peered":"2024-09-14T04:27:55.394475+0000","last_unstale":"2024-09-14T04:33:03.517493+0000","last_undegraded":"2024-09-14T04:33:03.517493+0000","last_fullsized":"2024-09-14T04:33:03.517493+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:44:06.833702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":641,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.521856+0000","last_change":"2024-09-14T04:33:03.521856+0000","last_active":"2024-09-14T04:33:03.521856+0000","last_peered":"2024-09-14T04:33:03.521856+0000","last_clean":"2024-09-14T04:33:03.521856+0000","last_became_active":"2024-09-14T04:33:03.521485+0000","last_became_peered":"2024-09-14T04:33:03.521485+0000","last_unstale":"2024-09-14T04:33:03.521856+0000","last_undegraded":"2024-09-14T04:33:03.521856+0000","last_fullsized":"2024-09-14T04:33:03.521856+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:54:29.353652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"62'2","reported_seq":190,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.507006+0000","last_change":"2024-09-14T04:31:52.391477+0000","last_active":"2024-09-14T04:33:02.507006+0000","last_peered":"2024-09-14T04:33:02.507006+0000","last_clean":"2024-09-14T04:33:02.507006+0000","last_became_active":"2024-09-14T04:31:52.374373+0000","last_became_peered":"2024-09-14T04:31:52.374373+0000","last_unstale":"2024-09-14T04:33:02.507006+0000","last_undegraded":"2024-09-14T04:33:02.507006+0000","last_fullsized":"2024-09-14T04:33:02.507006+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:42:35.354303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":658,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.522497+0000","last_change":"2024-09-14T04:33:03.522497+0000","last_active":"2024-09-14T04:33:03.522497+0000","last_peered":"2024-09-14T04:33:03.522497+0000","last_clean":"2024-09-14T04:33:03.522497+0000","last_became_active":"2024-09-14T04:33:03.522200+0000","last_became_peered":"2024-09-14T04:33:03.522200+0000","last_unstale":"2024-09-14T04:33:03.522497+0000","last_undegraded":"2024-09-14T04:33:03.522497+0000","last_fullsized":"2024-09-14T04:33:03.522497+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:47:02.685520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"62'1","reported_seq":54,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513937+0000","last_change":"2024-09-14T04:31:59.727688+0000","last_active":"2024-09-14T04:33:03.513937+0000","last_peered":"2024-09-14T04:33:03.513937+0000","last_clean":"2024-09-14T04:33:03.513937+0000","last_became_active":"2024-09-14T04:31:59.727253+0000","last_became_peered":"2024-09-14T04:31:59.727253+0000","last_unstale":"2024-09-14T04:33:03.513937+0000","last_undegraded":"2024-09-14T04:33:03.513937+0000","last_fullsized":"2024-09-14T04:33:03.513937+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:30:47.836983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":253,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516739+0000","last_change":"2024-09-14T04:31:52.375414+0000","last_active":"2024-09-14T04:33:03.516739+0000","last_peered":"2024-09-14T04:33:03.516739+0000","last_clean":"2024-09-14T04:33:03.516739+0000","last_became_active":"2024-09-14T04:31:52.375096+0000","last_became_peered":"2024-09-14T04:31:52.375096+0000","last_unstale":"2024-09-14T04:33:03.516739+0000","last_undegraded":"2024-09-14T04:33:03.516739+0000","last_fullsized":"2024-09-14T04:33:03.516739+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:35:20.648591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"62'1","reported_seq":173,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.521477+0000","last_change":"2024-09-14T04:33:03.521477+0000","last_active":"2024-09-14T04:33:03.521477+0000","last_peered":"2024-09-14T04:33:03.521477+0000","last_clean":"2024-09-14T04:33:03.521477+0000","last_became_active":"2024-09-14T04:33:03.521109+0000","last_became_peered":"2024-09-14T04:33:03.521109+0000","last_unstale":"2024-09-14T04:33:03.521477+0000","last_undegraded":"2024-09-14T04:33:03.521477+0000","last_fullsized":"2024-09-14T04:33:03.521477+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:29:21.238847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":597,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517831+0000","last_change":"2024-09-14T04:31:52.383504+0000","last_active":"2024-09-14T04:33:03.517831+0000","last_peered":"2024-09-14T04:33:03.517831+0000","last_clean":"2024-09-14T04:33:03.517831+0000","last_became_active":"2024-09-14T04:31:52.383178+0000","last_became_peered":"2024-09-14T04:31:52.383178+0000","last_unstale":"2024-09-14T04:33:03.517831+0000","last_undegraded":"2024-09-14T04:33:03.517831+0000","last_fullsized":"2024-09-14T04:33:03.517831+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:39:06.937151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518330+0000","last_change":"2024-09-14T04:31:52.380838+0000","last_active":"2024-09-14T04:33:03.518330+0000","last_peered":"2024-09-14T04:33:03.518330+0000","last_clean":"2024-09-14T04:33:03.518330+0000","last_became_active":"2024-09-14T04:31:52.380514+0000","last_became_peered":"2024-09-14T04:31:52.380514+0000","last_unstale":"2024-09-14T04:33:03.518330+0000","last_undegraded":"2024-09-14T04:33:03.518330+0000","last_fullsized":"2024-09-14T04:33:03.518330+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:38:13.905830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":616,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513658+0000","last_change":"2024-09-14T04:32:51.484111+0000","last_active":"2024-09-14T04:33:03.513658+0000","last_peered":"2024-09-14T04:33:03.513658+0000","last_clean":"2024-09-14T04:33:03.513658+0000","last_became_active":"2024-09-14T04:32:51.483789+0000","last_became_peered":"2024-09-14T04:32:51.483789+0000","last_unstale":"2024-09-14T04:33:03.513658+0000","last_undegraded":"2024-09-14T04:33:03.513658+0000","last_fullsized":"2024-09-14T04:33:03.513658+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:18:09.617056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":625,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518152+0000","last_change":"2024-09-14T04:31:59.734041+0000","last_active":"2024-09-14T04:33:03.518152+0000","last_peered":"2024-09-14T04:33:03.518152+0000","last_clean":"2024-09-14T04:33:03.518152+0000","last_became_active":"2024-09-14T04:31:59.733625+0000","last_became_peered":"2024-09-14T04:31:59.733625+0000","last_unstale":"2024-09-14T04:33:03.518152+0000","last_undegraded":"2024-09-14T04:33:03.518152+0000","last_fullsized":"2024-09-14T04:33:03.518152+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:33:47.588119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":507,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516510+0000","last_change":"2024-09-14T04:31:52.371573+0000","last_active":"2024-09-14T04:33:02.516510+0000","last_peered":"2024-09-14T04:33:02.516510+0000","last_clean":"2024-09-14T04:33:02.516510+0000","last_became_active":"2024-09-14T04:31:52.371233+0000","last_became_peered":"2024-09-14T04:31:52.371233+0000","last_unstale":"2024-09-14T04:33:02.516510+0000","last_undegraded":"2024-09-14T04:33:02.516510+0000","last_fullsized":"2024-09-14T04:33:02.516510+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:56:28.883198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":181,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518745+0000","last_change":"2024-09-14T04:31:52.371860+0000","last_active":"2024-09-14T04:33:03.518745+0000","last_peered":"2024-09-14T04:33:03.518745+0000","last_clean":"2024-09-14T04:33:03.518745+0000","last_became_active":"2024-09-14T04:31:52.371537+0000","last_became_peered":"2024-09-14T04:31:52.371537+0000","last_unstale":"2024-09-14T04:33:03.518745+0000","last_undegraded":"2024-09-14T04:33:03.518745+0000","last_fullsized":"2024-09-14T04:33:03.518745+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:06:38.848838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":745,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513715+0000","last_change":"2024-09-14T04:32:14.760654+0000","last_active":"2024-09-14T04:33:03.513715+0000","last_peered":"2024-09-14T04:33:03.513715+0000","last_clean":"2024-09-14T04:33:03.513715+0000","last_became_active":"2024-09-14T04:32:14.759851+0000","last_became_peered":"2024-09-14T04:32:14.759851+0000","last_unstale":"2024-09-14T04:33:03.513715+0000","last_undegraded":"2024-09-14T04:33:03.513715+0000","last_fullsized":"2024-09-14T04:33:03.513715+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:08:47.524123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"76'100","reported_seq":109,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516591+0000","last_change":"2024-09-14T04:31:47.352194+0000","last_active":"2024-09-14T04:33:03.516591+0000","last_peered":"2024-09-14T04:33:03.516591+0000","last_clean":"2024-09-14T04:33:03.516591+0000","last_became_active":"2024-09-14T04:31:47.351839+0000","last_became_peered":"2024-09-14T04:31:47.351839+0000","last_unstale":"2024-09-14T04:33:03.516591+0000","last_undegraded":"2024-09-14T04:33:03.516591+0000","last_fullsized":"2024-09-14T04:33:03.516591+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:44.013432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:44.013432+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:50.272154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":766,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516923+0000","last_change":"2024-09-14T04:31:47.352083+0000","last_active":"2024-09-14T04:33:03.516923+0000","last_peered":"2024-09-14T04:33:03.516923+0000","last_clean":"2024-09-14T04:33:03.516923+0000","last_became_active":"2024-09-14T04:31:47.351203+0000","last_became_peered":"2024-09-14T04:31:47.351203+0000","last_unstale":"2024-09-14T04:33:03.516923+0000","last_undegraded":"2024-09-14T04:33:03.516923+0000","last_fullsized":"2024-09-14T04:33:03.516923+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:06:33.387299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":208,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506698+0000","last_change":"2024-09-14T04:31:52.375127+0000","last_active":"2024-09-14T04:33:02.506698+0000","last_peered":"2024-09-14T04:33:02.506698+0000","last_clean":"2024-09-14T04:33:02.506698+0000","last_became_active":"2024-09-14T04:31:52.374761+0000","last_became_peered":"2024-09-14T04:31:52.374761+0000","last_unstale":"2024-09-14T04:33:02.506698+0000","last_undegraded":"2024-09-14T04:33:02.506698+0000","last_fullsized":"2024-09-14T04:33:02.506698+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:52:11.341877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"62'3","reported_seq":437,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516689+0000","last_change":"2024-09-14T04:32:56.491728+0000","last_active":"2024-09-14T04:33:02.516689+0000","last_peered":"2024-09-14T04:33:02.516689+0000","last_clean":"2024-09-14T04:33:02.516689+0000","last_became_active":"2024-09-14T04:32:56.491120+0000","last_became_peered":"2024-09-14T04:32:56.491120+0000","last_unstale":"2024-09-14T04:33:02.516689+0000","last_undegraded":"2024-09-14T04:33:02.516689+0000","last_fullsized":"2024-09-14T04:33:02.516689+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:07:40.674723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518319+0000","last_change":"2024-09-14T04:31:52.371965+0000","last_active":"2024-09-14T04:33:03.518319+0000","last_peered":"2024-09-14T04:33:03.518319+0000","last_clean":"2024-09-14T04:33:03.518319+0000","last_became_active":"2024-09-14T04:31:52.371454+0000","last_became_peered":"2024-09-14T04:31:52.371454+0000","last_unstale":"2024-09-14T04:33:03.518319+0000","last_undegraded":"2024-09-14T04:33:03.518319+0000","last_fullsized":"2024-09-14T04:33:03.518319+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:00:40.307983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"76'22","reported_seq":69,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.521088+0000","last_change":"2024-09-14T04:33:03.521088+0000","last_active":"2024-09-14T04:33:03.521088+0000","last_peered":"2024-09-14T04:33:03.521088+0000","last_clean":"2024-09-14T04:33:03.521088+0000","last_became_active":"2024-09-14T04:33:03.520674+0000","last_became_peered":"2024-09-14T04:33:03.520674+0000","last_unstale":"2024-09-14T04:33:03.521088+0000","last_undegraded":"2024-09-14T04:33:03.521088+0000","last_fullsized":"2024-09-14T04:33:03.521088+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:38:16.495007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":238,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.520723+0000","last_change":"2024-09-14T04:33:03.520723+0000","last_active":"2024-09-14T04:33:03.520723+0000","last_peered":"2024-09-14T04:33:03.520723+0000","last_clean":"2024-09-14T04:33:03.520723+0000","last_became_active":"2024-09-14T04:33:03.520300+0000","last_became_peered":"2024-09-14T04:33:03.520300+0000","last_unstale":"2024-09-14T04:33:03.520723+0000","last_undegraded":"2024-09-14T04:33:03.520723+0000","last_fullsized":"2024-09-14T04:33:03.520723+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:25:15.942721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":113,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.519828+0000","last_change":"2024-09-14T04:33:03.519828+0000","last_active":"2024-09-14T04:33:03.519828+0000","last_peered":"2024-09-14T04:33:03.519828+0000","last_clean":"2024-09-14T04:33:03.519828+0000","last_became_active":"2024-09-14T04:33:03.519427+0000","last_became_peered":"2024-09-14T04:33:03.519427+0000","last_unstale":"2024-09-14T04:33:03.519828+0000","last_undegraded":"2024-09-14T04:33:03.519828+0000","last_fullsized":"2024-09-14T04:33:03.519828+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:17:42.336129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":500,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516589+0000","last_change":"2024-09-14T04:32:51.483859+0000","last_active":"2024-09-14T04:33:02.516589+0000","last_peered":"2024-09-14T04:33:02.516589+0000","last_clean":"2024-09-14T04:33:02.516589+0000","last_became_active":"2024-09-14T04:32:51.483514+0000","last_became_peered":"2024-09-14T04:32:51.483514+0000","last_unstale":"2024-09-14T04:33:02.516589+0000","last_undegraded":"2024-09-14T04:33:02.516589+0000","last_fullsized":"2024-09-14T04:33:02.516589+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:02:53.586694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000255188,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":68,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513794+0000","last_change":"2024-09-14T04:32:14.764177+0000","last_active":"2024-09-14T04:33:03.513794+0000","last_peered":"2024-09-14T04:33:03.513794+0000","last_clean":"2024-09-14T04:33:03.513794+0000","last_became_active":"2024-09-14T04:32:14.763619+0000","last_became_peered":"2024-09-14T04:32:14.763619+0000","last_unstale":"2024-09-14T04:33:03.513794+0000","last_undegraded":"2024-09-14T04:33:03.513794+0000","last_fullsized":"2024-09-14T04:33:03.513794+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:42:47.802221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"76'100","reported_seq":234,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516874+0000","last_change":"2024-09-14T04:31:47.352652+0000","last_active":"2024-09-14T04:33:03.516874+0000","last_peered":"2024-09-14T04:33:03.516874+0000","last_clean":"2024-09-14T04:33:03.516874+0000","last_became_active":"2024-09-14T04:31:47.351579+0000","last_became_peered":"2024-09-14T04:31:47.351579+0000","last_unstale":"2024-09-14T04:33:03.516874+0000","last_undegraded":"2024-09-14T04:33:03.516874+0000","last_fullsized":"2024-09-14T04:33:03.516874+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:43.009173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:43.009173+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:13:48.134273+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":372,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518077+0000","last_change":"2024-09-14T04:32:51.484556+0000","last_active":"2024-09-14T04:33:03.518077+0000","last_peered":"2024-09-14T04:33:03.518077+0000","last_clean":"2024-09-14T04:33:03.518077+0000","last_became_active":"2024-09-14T04:32:51.483616+0000","last_became_peered":"2024-09-14T04:32:51.483616+0000","last_unstale":"2024-09-14T04:33:03.518077+0000","last_undegraded":"2024-09-14T04:33:03.518077+0000","last_fullsized":"2024-09-14T04:33:03.518077+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:10:01.211635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.519164+0000","last_change":"2024-09-14T04:31:52.377684+0000","last_active":"2024-09-14T04:33:03.519164+0000","last_peered":"2024-09-14T04:33:03.519164+0000","last_clean":"2024-09-14T04:33:03.519164+0000","last_became_active":"2024-09-14T04:31:52.377288+0000","last_became_peered":"2024-09-14T04:31:52.377288+0000","last_unstale":"2024-09-14T04:33:03.519164+0000","last_undegraded":"2024-09-14T04:33:03.519164+0000","last_fullsized":"2024-09-14T04:33:03.519164+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:41:34.260691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"62'2","reported_seq":837,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.519257+0000","last_change":"2024-09-14T04:31:59.744302+0000","last_active":"2024-09-14T04:33:03.519257+0000","last_peered":"2024-09-14T04:33:03.519257+0000","last_clean":"2024-09-14T04:33:03.519257+0000","last_became_active":"2024-09-14T04:31:59.730395+0000","last_became_peered":"2024-09-14T04:31:59.730395+0000","last_unstale":"2024-09-14T04:33:03.519257+0000","last_undegraded":"2024-09-14T04:33:03.519257+0000","last_fullsized":"2024-09-14T04:33:03.519257+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:34:44.891719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"76'100","reported_seq":247,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.646647+0000","last_change":"2024-09-14T04:33:03.646609+0000","last_active":"2024-09-14T04:33:03.646647+0000","last_peered":"2024-09-14T04:33:03.646647+0000","last_clean":"2024-09-14T04:33:03.646647+0000","last_became_active":"2024-09-14T04:33:03.519287+0000","last_became_peered":"2024-09-14T04:33:03.519287+0000","last_unstale":"2024-09-14T04:33:03.646647+0000","last_undegraded":"2024-09-14T04:33:03.646647+0000","last_fullsized":"2024-09-14T04:33:03.646647+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:33:03.646544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:33:03.646544+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:51:56.960616+0000","scrub_duration":0.0015616460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":762,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513851+0000","last_change":"2024-09-14T04:32:14.757629+0000","last_active":"2024-09-14T04:33:03.513851+0000","last_peered":"2024-09-14T04:33:03.513851+0000","last_clean":"2024-09-14T04:33:03.513851+0000","last_became_active":"2024-09-14T04:32:14.757116+0000","last_became_peered":"2024-09-14T04:32:14.757116+0000","last_unstale":"2024-09-14T04:33:03.513851+0000","last_undegraded":"2024-09-14T04:33:03.513851+0000","last_fullsized":"2024-09-14T04:33:03.513851+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:46.869240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":57,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516728+0000","last_change":"2024-09-14T04:31:59.734539+0000","last_active":"2024-09-14T04:33:02.516728+0000","last_peered":"2024-09-14T04:33:02.516728+0000","last_clean":"2024-09-14T04:33:02.516728+0000","last_became_active":"2024-09-14T04:31:59.734187+0000","last_became_peered":"2024-09-14T04:31:59.734187+0000","last_unstale":"2024-09-14T04:33:02.516728+0000","last_undegraded":"2024-09-14T04:33:02.516728+0000","last_fullsized":"2024-09-14T04:33:02.516728+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:07:59.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027404000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516917+0000","last_change":"2024-09-14T04:31:52.371837+0000","last_active":"2024-09-14T04:33:03.516917+0000","last_peered":"2024-09-14T04:33:03.516917+0000","last_clean":"2024-09-14T04:33:03.516917+0000","last_became_active":"2024-09-14T04:31:52.371505+0000","last_became_peered":"2024-09-14T04:31:52.371505+0000","last_unstale":"2024-09-14T04:33:03.516917+0000","last_undegraded":"2024-09-14T04:33:03.516917+0000","last_fullsized":"2024-09-14T04:33:03.516917+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:05:00.127909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"76'100","reported_seq":384,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515836+0000","last_change":"2024-09-14T04:31:59.734403+0000","last_active":"2024-09-14T04:33:02.515836+0000","last_peered":"2024-09-14T04:33:02.515836+0000","last_clean":"2024-09-14T04:33:02.515836+0000","last_became_active":"2024-09-14T04:31:59.734047+0000","last_became_peered":"2024-09-14T04:31:59.734047+0000","last_unstale":"2024-09-14T04:33:02.515836+0000","last_undegraded":"2024-09-14T04:33:02.515836+0000","last_fullsized":"2024-09-14T04:33:02.515836+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:49.892654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:49.892654+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:46:07.424981+0000","scrub_duration":0.001440715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518495+0000","last_change":"2024-09-14T04:31:52.376118+0000","last_active":"2024-09-14T04:33:03.518495+0000","last_peered":"2024-09-14T04:33:03.518495+0000","last_clean":"2024-09-14T04:33:03.518495+0000","last_became_active":"2024-09-14T04:31:52.374083+0000","last_became_peered":"2024-09-14T04:31:52.374083+0000","last_unstale":"2024-09-14T04:33:03.518495+0000","last_undegraded":"2024-09-14T04:33:03.518495+0000","last_fullsized":"2024-09-14T04:33:03.518495+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:27:32.204002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":69,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516732+0000","last_change":"2024-09-14T04:31:59.735778+0000","last_active":"2024-09-14T04:33:03.516732+0000","last_peered":"2024-09-14T04:33:03.516732+0000","last_clean":"2024-09-14T04:33:03.516732+0000","last_became_active":"2024-09-14T04:31:59.735426+0000","last_became_peered":"2024-09-14T04:31:59.735426+0000","last_unstale":"2024-09-14T04:33:03.516732+0000","last_undegraded":"2024-09-14T04:33:03.516732+0000","last_fullsized":"2024-09-14T04:33:03.516732+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:15:38.789185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000348069,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":178,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506738+0000","last_change":"2024-09-14T04:25:42.253453+0000","last_active":"2024-09-14T04:33:02.506738+0000","last_peered":"2024-09-14T04:33:02.506738+0000","last_clean":"2024-09-14T04:33:02.506738+0000","last_became_active":"2024-09-14T04:25:42.252529+0000","last_became_peered":"2024-09-14T04:25:42.252529+0000","last_unstale":"2024-09-14T04:33:02.506738+0000","last_undegraded":"2024-09-14T04:33:02.506738+0000","last_fullsized":"2024-09-14T04:33:02.506738+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:40:19.539802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"76'100","reported_seq":384,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515842+0000","last_change":"2024-09-14T04:31:47.355339+0000","last_active":"2024-09-14T04:33:02.515842+0000","last_peered":"2024-09-14T04:33:02.515842+0000","last_clean":"2024-09-14T04:33:02.515842+0000","last_became_active":"2024-09-14T04:31:47.354990+0000","last_became_peered":"2024-09-14T04:31:47.354990+0000","last_unstale":"2024-09-14T04:33:02.515842+0000","last_undegraded":"2024-09-14T04:33:02.515842+0000","last_fullsized":"2024-09-14T04:33:02.515842+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:52.943203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:52.943203+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:13:15.560656+0000","scrub_duration":0.0015944450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516830+0000","last_change":"2024-09-14T04:31:52.381785+0000","last_active":"2024-09-14T04:33:03.516830+0000","last_peered":"2024-09-14T04:33:03.516830+0000","last_clean":"2024-09-14T04:33:03.516830+0000","last_became_active":"2024-09-14T04:31:52.381476+0000","last_became_peered":"2024-09-14T04:31:52.381476+0000","last_unstale":"2024-09-14T04:33:03.516830+0000","last_undegraded":"2024-09-14T04:33:03.516830+0000","last_fullsized":"2024-09-14T04:33:03.516830+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:51:45.506882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":284,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.519286+0000","last_change":"2024-09-14T04:33:03.519286+0000","last_active":"2024-09-14T04:33:03.519286+0000","last_peered":"2024-09-14T04:33:03.519286+0000","last_clean":"2024-09-14T04:33:03.519286+0000","last_became_active":"2024-09-14T04:33:03.518920+0000","last_became_peered":"2024-09-14T04:33:03.518920+0000","last_unstale":"2024-09-14T04:33:03.519286+0000","last_undegraded":"2024-09-14T04:33:03.519286+0000","last_fullsized":"2024-09-14T04:33:03.519286+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:36:14.512397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":204,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506491+0000","last_change":"2024-09-14T04:31:52.388705+0000","last_active":"2024-09-14T04:33:02.506491+0000","last_peered":"2024-09-14T04:33:02.506491+0000","last_clean":"2024-09-14T04:33:02.506491+0000","last_became_active":"2024-09-14T04:31:52.373430+0000","last_became_peered":"2024-09-14T04:31:52.373430+0000","last_unstale":"2024-09-14T04:33:02.506491+0000","last_undegraded":"2024-09-14T04:33:02.506491+0000","last_fullsized":"2024-09-14T04:33:02.506491+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:29:17.445736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"76'100","reported_seq":139,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506389+0000","last_change":"2024-09-14T04:31:59.725675+0000","last_active":"2024-09-14T04:33:02.506389+0000","last_peered":"2024-09-14T04:33:02.506389+0000","last_clean":"2024-09-14T04:33:02.506389+0000","last_became_active":"2024-09-14T04:31:59.725307+0000","last_became_peered":"2024-09-14T04:31:59.725307+0000","last_unstale":"2024-09-14T04:33:02.506389+0000","last_undegraded":"2024-09-14T04:33:02.506389+0000","last_fullsized":"2024-09-14T04:33:02.506389+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:49.280129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:49.280129+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:52:25.350374+0000","scrub_duration":0.001556279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":69,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517017+0000","last_change":"2024-09-14T04:31:59.734619+0000","last_active":"2024-09-14T04:33:03.517017+0000","last_peered":"2024-09-14T04:33:03.517017+0000","last_clean":"2024-09-14T04:33:03.517017+0000","last_became_active":"2024-09-14T04:31:59.734287+0000","last_became_peered":"2024-09-14T04:31:59.734287+0000","last_unstale":"2024-09-14T04:33:03.517017+0000","last_undegraded":"2024-09-14T04:33:03.517017+0000","last_fullsized":"2024-09-14T04:33:03.517017+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:03:39.070049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518159+0000","last_change":"2024-09-14T04:31:52.372820+0000","last_active":"2024-09-14T04:33:03.518159+0000","last_peered":"2024-09-14T04:33:03.518159+0000","last_clean":"2024-09-14T04:33:03.518159+0000","last_became_active":"2024-09-14T04:31:52.371186+0000","last_became_peered":"2024-09-14T04:31:52.371186+0000","last_unstale":"2024-09-14T04:33:03.518159+0000","last_undegraded":"2024-09-14T04:33:03.518159+0000","last_fullsized":"2024-09-14T04:33:03.518159+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:35:56.106257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":260,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517909+0000","last_change":"2024-09-14T04:31:59.725253+0000","last_active":"2024-09-14T04:33:03.517909+0000","last_peered":"2024-09-14T04:33:03.517909+0000","last_clean":"2024-09-14T04:33:03.517909+0000","last_became_active":"2024-09-14T04:31:59.724889+0000","last_became_peered":"2024-09-14T04:31:59.724889+0000","last_unstale":"2024-09-14T04:33:03.517909+0000","last_undegraded":"2024-09-14T04:33:03.517909+0000","last_fullsized":"2024-09-14T04:33:03.517909+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:18:39.083123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034499999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"76'100","reported_seq":430,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.513878+0000","last_change":"2024-09-14T04:32:56.492141+0000","last_active":"2024-09-14T04:33:02.513878+0000","last_peered":"2024-09-14T04:33:02.513878+0000","last_clean":"2024-09-14T04:33:02.513878+0000","last_became_active":"2024-09-14T04:32:56.491571+0000","last_became_peered":"2024-09-14T04:32:56.491571+0000","last_unstale":"2024-09-14T04:33:02.513878+0000","last_undegraded":"2024-09-14T04:33:02.513878+0000","last_fullsized":"2024-09-14T04:33:02.513878+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:44.070022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:44.070022+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:25:57.896234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":64,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513529+0000","last_change":"2024-09-14T04:32:14.763887+0000","last_active":"2024-09-14T04:33:03.513529+0000","last_peered":"2024-09-14T04:33:03.513529+0000","last_clean":"2024-09-14T04:33:03.513529+0000","last_became_active":"2024-09-14T04:32:14.762741+0000","last_became_peered":"2024-09-14T04:32:14.762741+0000","last_unstale":"2024-09-14T04:33:03.513529+0000","last_undegraded":"2024-09-14T04:33:03.513529+0000","last_fullsized":"2024-09-14T04:33:03.513529+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:40:15.683940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":647,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513085+0000","last_change":"2024-09-14T04:32:14.760606+0000","last_active":"2024-09-14T04:33:03.513085+0000","last_peered":"2024-09-14T04:33:03.513085+0000","last_clean":"2024-09-14T04:33:03.513085+0000","last_became_active":"2024-09-14T04:32:14.760339+0000","last_became_peered":"2024-09-14T04:32:14.760339+0000","last_unstale":"2024-09-14T04:33:03.513085+0000","last_undegraded":"2024-09-14T04:33:03.513085+0000","last_fullsized":"2024-09-14T04:33:03.513085+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:04:30.093850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":238,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.513762+0000","last_change":"2024-09-14T04:32:56.492593+0000","last_active":"2024-09-14T04:33:02.513762+0000","last_peered":"2024-09-14T04:33:02.513762+0000","last_clean":"2024-09-14T04:33:02.513762+0000","last_became_active":"2024-09-14T04:32:56.492236+0000","last_became_peered":"2024-09-14T04:32:56.492236+0000","last_unstale":"2024-09-14T04:33:02.513762+0000","last_undegraded":"2024-09-14T04:33:02.513762+0000","last_fullsized":"2024-09-14T04:33:02.513762+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:35:06.692185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023351000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"76'100","reported_seq":71,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517201+0000","last_change":"2024-09-14T04:31:54.205323+0000","last_active":"2024-09-14T04:33:03.517201+0000","last_peered":"2024-09-14T04:33:03.517201+0000","last_clean":"2024-09-14T04:33:03.517201+0000","last_became_active":"2024-09-14T04:31:52.377635+0000","last_became_peered":"2024-09-14T04:31:52.377635+0000","last_unstale":"2024-09-14T04:33:03.517201+0000","last_undegraded":"2024-09-14T04:33:03.517201+0000","last_fullsized":"2024-09-14T04:33:03.517201+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:31:54.205276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:31:54.205276+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:10:27.063913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":470,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515961+0000","last_change":"2024-09-14T04:31:47.354575+0000","last_active":"2024-09-14T04:33:02.515961+0000","last_peered":"2024-09-14T04:33:02.515961+0000","last_clean":"2024-09-14T04:33:02.515961+0000","last_became_active":"2024-09-14T04:31:47.354277+0000","last_became_peered":"2024-09-14T04:31:47.354277+0000","last_unstale":"2024-09-14T04:33:02.515961+0000","last_undegraded":"2024-09-14T04:33:02.515961+0000","last_fullsized":"2024-09-14T04:33:02.515961+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:34:51.518685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":592,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518603+0000","last_change":"2024-09-14T04:22:08.472386+0000","last_active":"2024-09-14T04:33:03.518603+0000","last_peered":"2024-09-14T04:33:03.518603+0000","last_clean":"2024-09-14T04:33:03.518603+0000","last_became_active":"2024-09-14T04:22:08.472100+0000","last_became_peered":"2024-09-14T04:22:08.472100+0000","last_unstale":"2024-09-14T04:33:03.518603+0000","last_undegraded":"2024-09-14T04:33:03.518603+0000","last_fullsized":"2024-09-14T04:33:03.518603+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:36:03.114984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":215,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506888+0000","last_change":"2024-09-14T04:31:47.378091+0000","last_active":"2024-09-14T04:33:02.506888+0000","last_peered":"2024-09-14T04:33:02.506888+0000","last_clean":"2024-09-14T04:33:02.506888+0000","last_became_active":"2024-09-14T04:31:47.357003+0000","last_became_peered":"2024-09-14T04:31:47.357003+0000","last_unstale":"2024-09-14T04:33:02.506888+0000","last_undegraded":"2024-09-14T04:33:02.506888+0000","last_fullsized":"2024-09-14T04:33:02.506888+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:41:55.419418+0000","last_clean_scrub_stamp":"2024-09-14T03:41:55.419418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:23:33.997689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'100","reported_seq":355,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.513678+0000","last_change":"2024-09-14T04:32:56.492971+0000","last_active":"2024-09-14T04:33:02.513678+0000","last_peered":"2024-09-14T04:33:02.513678+0000","last_clean":"2024-09-14T04:33:02.513678+0000","last_became_active":"2024-09-14T04:32:56.492543+0000","last_became_peered":"2024-09-14T04:32:56.492543+0000","last_unstale":"2024-09-14T04:33:02.513678+0000","last_undegraded":"2024-09-14T04:33:02.513678+0000","last_fullsized":"2024-09-14T04:33:02.513678+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:50.911016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:50.911016+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:36:25.564886+0000","scrub_duration":0.0014927930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":62,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517103+0000","last_change":"2024-09-14T04:31:52.381945+0000","last_active":"2024-09-14T04:33:03.517103+0000","last_peered":"2024-09-14T04:33:03.517103+0000","last_clean":"2024-09-14T04:33:03.517103+0000","last_became_active":"2024-09-14T04:31:52.381229+0000","last_became_peered":"2024-09-14T04:31:52.381229+0000","last_unstale":"2024-09-14T04:33:03.517103+0000","last_undegraded":"2024-09-14T04:33:03.517103+0000","last_fullsized":"2024-09-14T04:33:03.517103+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:19:03.665009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":606,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518280+0000","last_change":"2024-09-14T04:31:52.383693+0000","last_active":"2024-09-14T04:33:03.518280+0000","last_peered":"2024-09-14T04:33:03.518280+0000","last_clean":"2024-09-14T04:33:03.518280+0000","last_became_active":"2024-09-14T04:31:52.383221+0000","last_became_peered":"2024-09-14T04:31:52.383221+0000","last_unstale":"2024-09-14T04:33:03.518280+0000","last_undegraded":"2024-09-14T04:33:03.518280+0000","last_fullsized":"2024-09-14T04:33:03.518280+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:27:19.751816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"76'100","reported_seq":109,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506737+0000","last_change":"2024-09-14T04:27:35.272869+0000","last_active":"2024-09-14T04:33:02.506737+0000","last_peered":"2024-09-14T04:33:02.506737+0000","last_clean":"2024-09-14T04:33:02.506737+0000","last_became_active":"2024-09-14T04:27:12.688026+0000","last_became_peered":"2024-09-14T04:27:12.688026+0000","last_unstale":"2024-09-14T04:33:02.506737+0000","last_undegraded":"2024-09-14T04:33:02.506737+0000","last_fullsized":"2024-09-14T04:33:02.506737+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:35.272785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:35.272785+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:00:40.802976+0000","scrub_duration":0.0029595770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":339,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.521874+0000","last_change":"2024-09-14T04:33:03.521874+0000","last_active":"2024-09-14T04:33:03.521874+0000","last_peered":"2024-09-14T04:33:03.521874+0000","last_clean":"2024-09-14T04:33:03.521874+0000","last_became_active":"2024-09-14T04:33:03.520785+0000","last_became_peered":"2024-09-14T04:33:03.520785+0000","last_unstale":"2024-09-14T04:33:03.521874+0000","last_undegraded":"2024-09-14T04:33:03.521874+0000","last_fullsized":"2024-09-14T04:33:03.521874+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:55:56.448978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":337,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.522346+0000","last_change":"2024-09-14T04:33:03.522346+0000","last_active":"2024-09-14T04:33:03.522346+0000","last_peered":"2024-09-14T04:33:03.522346+0000","last_clean":"2024-09-14T04:33:03.522346+0000","last_became_active":"2024-09-14T04:33:03.521931+0000","last_became_peered":"2024-09-14T04:33:03.521931+0000","last_unstale":"2024-09-14T04:33:03.522346+0000","last_undegraded":"2024-09-14T04:33:03.522346+0000","last_fullsized":"2024-09-14T04:33:03.522346+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:37:11.901738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"76'100","reported_seq":78,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517429+0000","last_change":"2024-09-14T04:31:59.739551+0000","last_active":"2024-09-14T04:33:03.517429+0000","last_peered":"2024-09-14T04:33:03.517429+0000","last_clean":"2024-09-14T04:33:03.517429+0000","last_became_active":"2024-09-14T04:31:59.739173+0000","last_became_peered":"2024-09-14T04:31:59.739173+0000","last_unstale":"2024-09-14T04:33:03.517429+0000","last_undegraded":"2024-09-14T04:33:03.517429+0000","last_fullsized":"2024-09-14T04:33:03.517429+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:31:53.189290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:31:53.189290+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:19:15.851448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":279,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.520972+0000","last_change":"2024-09-14T04:33:03.520972+0000","last_active":"2024-09-14T04:33:03.520972+0000","last_peered":"2024-09-14T04:33:03.520972+0000","last_clean":"2024-09-14T04:33:03.520972+0000","last_became_active":"2024-09-14T04:33:03.519956+0000","last_became_peered":"2024-09-14T04:33:03.519956+0000","last_unstale":"2024-09-14T04:33:03.520972+0000","last_undegraded":"2024-09-14T04:33:03.520972+0000","last_fullsized":"2024-09-14T04:33:03.520972+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:50:33.233796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":478,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515568+0000","last_change":"2024-09-14T04:31:52.372211+0000","last_active":"2024-09-14T04:33:02.515568+0000","last_peered":"2024-09-14T04:33:02.515568+0000","last_clean":"2024-09-14T04:33:02.515568+0000","last_became_active":"2024-09-14T04:31:52.371900+0000","last_became_peered":"2024-09-14T04:31:52.371900+0000","last_unstale":"2024-09-14T04:33:02.515568+0000","last_undegraded":"2024-09-14T04:33:02.515568+0000","last_fullsized":"2024-09-14T04:33:02.515568+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:42:24.165090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"76'100","reported_seq":114,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506354+0000","last_change":"2024-09-14T04:27:46.144096+0000","last_active":"2024-09-14T04:33:02.506354+0000","last_peered":"2024-09-14T04:33:02.506354+0000","last_clean":"2024-09-14T04:33:02.506354+0000","last_became_active":"2024-09-14T04:27:12.688140+0000","last_became_peered":"2024-09-14T04:27:12.688140+0000","last_unstale":"2024-09-14T04:33:02.506354+0000","last_undegraded":"2024-09-14T04:33:02.506354+0000","last_fullsized":"2024-09-14T04:33:02.506354+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:46.144003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:46.144003+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:00:07.652536+0000","scrub_duration":0.0071911659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":282,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.520700+0000","last_change":"2024-09-14T04:33:03.520700+0000","last_active":"2024-09-14T04:33:03.520700+0000","last_peered":"2024-09-14T04:33:03.520700+0000","last_clean":"2024-09-14T04:33:03.520700+0000","last_became_active":"2024-09-14T04:33:03.520294+0000","last_became_peered":"2024-09-14T04:33:03.520294+0000","last_unstale":"2024-09-14T04:33:03.520700+0000","last_undegraded":"2024-09-14T04:33:03.520700+0000","last_fullsized":"2024-09-14T04:33:03.520700+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:39:41.254063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"62'2","reported_seq":858,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.518820+0000","last_change":"2024-09-14T04:31:59.754533+0000","last_active":"2024-09-14T04:33:03.518820+0000","last_peered":"2024-09-14T04:33:03.518820+0000","last_clean":"2024-09-14T04:33:03.518820+0000","last_became_active":"2024-09-14T04:31:59.731161+0000","last_became_peered":"2024-09-14T04:31:59.731161+0000","last_unstale":"2024-09-14T04:33:03.518820+0000","last_undegraded":"2024-09-14T04:33:03.518820+0000","last_fullsized":"2024-09-14T04:33:03.518820+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:27:06.084373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"76'100","reported_seq":165,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516312+0000","last_change":"2024-09-14T04:32:56.489379+0000","last_active":"2024-09-14T04:33:02.516312+0000","last_peered":"2024-09-14T04:33:02.516312+0000","last_clean":"2024-09-14T04:33:02.516312+0000","last_became_active":"2024-09-14T04:32:56.489045+0000","last_became_peered":"2024-09-14T04:32:56.489045+0000","last_unstale":"2024-09-14T04:33:02.516312+0000","last_undegraded":"2024-09-14T04:33:02.516312+0000","last_fullsized":"2024-09-14T04:33:02.516312+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":357,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:27:43.390999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:27:43.390999+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:32:49.743497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":64,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513221+0000","last_change":"2024-09-14T04:32:14.762919+0000","last_active":"2024-09-14T04:33:03.513221+0000","last_peered":"2024-09-14T04:33:03.513221+0000","last_clean":"2024-09-14T04:33:03.513221+0000","last_became_active":"2024-09-14T04:32:14.762605+0000","last_became_peered":"2024-09-14T04:32:14.762605+0000","last_unstale":"2024-09-14T04:33:03.513221+0000","last_undegraded":"2024-09-14T04:33:03.513221+0000","last_fullsized":"2024-09-14T04:33:03.513221+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:04:35.781173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":158,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506859+0000","last_change":"2024-09-14T04:22:08.468750+0000","last_active":"2024-09-14T04:33:02.506859+0000","last_peered":"2024-09-14T04:33:02.506859+0000","last_clean":"2024-09-14T04:33:02.506859+0000","last_became_active":"2024-09-14T04:22:08.468217+0000","last_became_peered":"2024-09-14T04:22:08.468217+0000","last_unstale":"2024-09-14T04:33:02.506859+0000","last_undegraded":"2024-09-14T04:33:02.506859+0000","last_fullsized":"2024-09-14T04:33:02.506859+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:12:06.709182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"76'100","reported_seq":247,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:05.611903+0000","last_change":"2024-09-14T04:33:05.611870+0000","last_active":"2024-09-14T04:33:05.611903+0000","last_peered":"2024-09-14T04:33:05.611903+0000","last_clean":"2024-09-14T04:33:05.611903+0000","last_became_active":"2024-09-14T04:33:03.521236+0000","last_became_peered":"2024-09-14T04:33:03.521236+0000","last_unstale":"2024-09-14T04:33:05.611903+0000","last_undegraded":"2024-09-14T04:33:05.611903+0000","last_fullsized":"2024-09-14T04:33:05.611903+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:33:05.611796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:33:05.611796+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:37:42.178717+0000","scrub_duration":0.0016521800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":507,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515374+0000","last_change":"2024-09-14T04:31:52.372957+0000","last_active":"2024-09-14T04:33:02.515374+0000","last_peered":"2024-09-14T04:33:02.515374+0000","last_clean":"2024-09-14T04:33:02.515374+0000","last_became_active":"2024-09-14T04:31:52.372585+0000","last_became_peered":"2024-09-14T04:31:52.372585+0000","last_unstale":"2024-09-14T04:33:02.515374+0000","last_undegraded":"2024-09-14T04:33:02.515374+0000","last_fullsized":"2024-09-14T04:33:02.515374+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:32:03.907839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"62'1","reported_seq":166,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516547+0000","last_change":"2024-09-14T04:32:56.493160+0000","last_active":"2024-09-14T04:33:02.516547+0000","last_peered":"2024-09-14T04:33:02.516547+0000","last_clean":"2024-09-14T04:33:02.516547+0000","last_became_active":"2024-09-14T04:32:56.492387+0000","last_became_peered":"2024-09-14T04:32:56.492387+0000","last_unstale":"2024-09-14T04:33:02.516547+0000","last_undegraded":"2024-09-14T04:33:02.516547+0000","last_fullsized":"2024-09-14T04:33:02.516547+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:11:52.363513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"76'100","reported_seq":383,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515940+0000","last_change":"2024-09-14T04:30:54.960873+0000","last_active":"2024-09-14T04:33:02.515940+0000","last_peered":"2024-09-14T04:33:02.515940+0000","last_clean":"2024-09-14T04:33:02.515940+0000","last_became_active":"2024-09-14T04:30:52.535921+0000","last_became_peered":"2024-09-14T04:30:52.535921+0000","last_unstale":"2024-09-14T04:33:02.515940+0000","last_undegraded":"2024-09-14T04:33:02.515940+0000","last_fullsized":"2024-09-14T04:33:02.515940+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:54.960754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:54.960754+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:41:30.181896+0000","scrub_duration":0.001686597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":315,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.522025+0000","last_change":"2024-09-14T04:33:03.522025+0000","last_active":"2024-09-14T04:33:03.522025+0000","last_peered":"2024-09-14T04:33:03.522025+0000","last_clean":"2024-09-14T04:33:03.522025+0000","last_became_active":"2024-09-14T04:33:03.520351+0000","last_became_peered":"2024-09-14T04:33:03.520351+0000","last_unstale":"2024-09-14T04:33:03.522025+0000","last_undegraded":"2024-09-14T04:33:03.522025+0000","last_fullsized":"2024-09-14T04:33:03.522025+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:50:28.539397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"62'2","reported_seq":139,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.513519+0000","last_change":"2024-09-14T04:32:56.493073+0000","last_active":"2024-09-14T04:33:02.513519+0000","last_peered":"2024-09-14T04:33:02.513519+0000","last_clean":"2024-09-14T04:33:02.513519+0000","last_became_active":"2024-09-14T04:32:56.492691+0000","last_became_peered":"2024-09-14T04:32:56.492691+0000","last_unstale":"2024-09-14T04:33:02.513519+0000","last_undegraded":"2024-09-14T04:33:02.513519+0000","last_fullsized":"2024-09-14T04:33:02.513519+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:24:50.118027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"76'100","reported_seq":248,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.516930+0000","last_change":"2024-09-14T04:31:59.756931+0000","last_active":"2024-09-14T04:33:03.516930+0000","last_peered":"2024-09-14T04:33:03.516930+0000","last_clean":"2024-09-14T04:33:03.516930+0000","last_became_active":"2024-09-14T04:31:59.733293+0000","last_became_peered":"2024-09-14T04:31:59.733293+0000","last_unstale":"2024-09-14T04:33:03.516930+0000","last_undegraded":"2024-09-14T04:33:03.516930+0000","last_fullsized":"2024-09-14T04:33:03.516930+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:45.041196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:45.041196+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:53:43.086319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":178,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506503+0000","last_change":"2024-09-14T04:32:51.483935+0000","last_active":"2024-09-14T04:33:02.506503+0000","last_peered":"2024-09-14T04:33:02.506503+0000","last_clean":"2024-09-14T04:33:02.506503+0000","last_became_active":"2024-09-14T04:32:51.483539+0000","last_became_peered":"2024-09-14T04:32:51.483539+0000","last_unstale":"2024-09-14T04:33:02.506503+0000","last_undegraded":"2024-09-14T04:33:02.506503+0000","last_fullsized":"2024-09-14T04:33:02.506503+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:33:13.451628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'1","reported_seq":480,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516040+0000","last_change":"2024-09-14T04:32:56.491717+0000","last_active":"2024-09-14T04:33:02.516040+0000","last_peered":"2024-09-14T04:33:02.516040+0000","last_clean":"2024-09-14T04:33:02.516040+0000","last_became_active":"2024-09-14T04:32:56.491154+0000","last_became_peered":"2024-09-14T04:32:56.491154+0000","last_unstale":"2024-09-14T04:33:02.516040+0000","last_undegraded":"2024-09-14T04:33:02.516040+0000","last_fullsized":"2024-09-14T04:33:02.516040+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:13:16.238085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"76'100","reported_seq":66,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516404+0000","last_change":"2024-09-14T04:32:56.492577+0000","last_active":"2024-09-14T04:33:02.516404+0000","last_peered":"2024-09-14T04:33:02.516404+0000","last_clean":"2024-09-14T04:33:02.516404+0000","last_became_active":"2024-09-14T04:32:56.492207+0000","last_became_peered":"2024-09-14T04:32:56.492207+0000","last_unstale":"2024-09-14T04:33:02.516404+0000","last_undegraded":"2024-09-14T04:33:02.516404+0000","last_fullsized":"2024-09-14T04:33:02.516404+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:45.083429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:45.083429+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:26:20.206726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"62'8","reported_seq":526,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.515312+0000","last_change":"2024-09-14T04:31:59.750713+0000","last_active":"2024-09-14T04:33:02.515312+0000","last_peered":"2024-09-14T04:33:02.515312+0000","last_clean":"2024-09-14T04:33:02.515312+0000","last_became_active":"2024-09-14T04:31:59.735685+0000","last_became_peered":"2024-09-14T04:31:59.735685+0000","last_unstale":"2024-09-14T04:33:02.515312+0000","last_undegraded":"2024-09-14T04:33:02.515312+0000","last_fullsized":"2024-09-14T04:33:02.515312+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:59:15.857747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":618,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.519089+0000","last_change":"2024-09-14T04:32:51.483901+0000","last_active":"2024-09-14T04:33:03.519089+0000","last_peered":"2024-09-14T04:33:03.519089+0000","last_clean":"2024-09-14T04:33:03.519089+0000","last_became_active":"2024-09-14T04:32:51.483541+0000","last_became_peered":"2024-09-14T04:32:51.483541+0000","last_unstale":"2024-09-14T04:33:03.519089+0000","last_undegraded":"2024-09-14T04:33:03.519089+0000","last_fullsized":"2024-09-14T04:33:03.519089+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:38:59.377719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"76'100","reported_seq":283,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506402+0000","last_change":"2024-09-14T04:31:52.372540+0000","last_active":"2024-09-14T04:33:02.506402+0000","last_peered":"2024-09-14T04:33:02.506402+0000","last_clean":"2024-09-14T04:33:02.506402+0000","last_became_active":"2024-09-14T04:31:52.372198+0000","last_became_peered":"2024-09-14T04:31:52.372198+0000","last_unstale":"2024-09-14T04:33:02.506402+0000","last_undegraded":"2024-09-14T04:33:02.506402+0000","last_fullsized":"2024-09-14T04:33:02.506402+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:31:01.287655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:31:01.287655+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:25:12.147394+0000","scrub_duration":0.0014619590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"76'100","reported_seq":63,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517338+0000","last_change":"2024-09-14T04:31:52.382156+0000","last_active":"2024-09-14T04:33:03.517338+0000","last_peered":"2024-09-14T04:33:03.517338+0000","last_clean":"2024-09-14T04:33:03.517338+0000","last_became_active":"2024-09-14T04:31:52.381538+0000","last_became_peered":"2024-09-14T04:31:52.381538+0000","last_unstale":"2024-09-14T04:33:03.517338+0000","last_undegraded":"2024-09-14T04:33:03.517338+0000","last_fullsized":"2024-09-14T04:33:03.517338+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:53.041045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:53.041045+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:46:54.554799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":435,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.516189+0000","last_change":"2024-09-14T04:32:56.492552+0000","last_active":"2024-09-14T04:33:02.516189+0000","last_peered":"2024-09-14T04:33:02.516189+0000","last_clean":"2024-09-14T04:33:02.516189+0000","last_became_active":"2024-09-14T04:32:56.492242+0000","last_became_peered":"2024-09-14T04:32:56.492242+0000","last_unstale":"2024-09-14T04:33:02.516189+0000","last_undegraded":"2024-09-14T04:33:02.516189+0000","last_fullsized":"2024-09-14T04:33:02.516189+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:51:44.242839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":124,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513908+0000","last_change":"2024-09-14T04:32:51.484511+0000","last_active":"2024-09-14T04:33:03.513908+0000","last_peered":"2024-09-14T04:33:03.513908+0000","last_clean":"2024-09-14T04:33:03.513908+0000","last_became_active":"2024-09-14T04:32:51.484154+0000","last_became_peered":"2024-09-14T04:32:51.484154+0000","last_unstale":"2024-09-14T04:33:03.513908+0000","last_undegraded":"2024-09-14T04:33:03.513908+0000","last_fullsized":"2024-09-14T04:33:03.513908+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:08:45.125779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":625,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513232+0000","last_change":"2024-09-14T04:32:14.759925+0000","last_active":"2024-09-14T04:33:03.513232+0000","last_peered":"2024-09-14T04:33:03.513232+0000","last_clean":"2024-09-14T04:33:03.513232+0000","last_became_active":"2024-09-14T04:32:14.759599+0000","last_became_peered":"2024-09-14T04:32:14.759599+0000","last_unstale":"2024-09-14T04:33:03.513232+0000","last_undegraded":"2024-09-14T04:33:03.513232+0000","last_fullsized":"2024-09-14T04:33:03.513232+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:00:51.683960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":188,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506806+0000","last_change":"2024-09-14T04:31:47.357642+0000","last_active":"2024-09-14T04:33:02.506806+0000","last_peered":"2024-09-14T04:33:02.506806+0000","last_clean":"2024-09-14T04:33:02.506806+0000","last_became_active":"2024-09-14T04:31:47.356307+0000","last_became_peered":"2024-09-14T04:31:47.356307+0000","last_unstale":"2024-09-14T04:33:02.506806+0000","last_undegraded":"2024-09-14T04:33:02.506806+0000","last_fullsized":"2024-09-14T04:33:02.506806+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T13:23:39.439300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"76'100","reported_seq":74,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.506360+0000","last_change":"2024-09-14T04:31:47.362183+0000","last_active":"2024-09-14T04:33:02.506360+0000","last_peered":"2024-09-14T04:33:02.506360+0000","last_clean":"2024-09-14T04:33:02.506360+0000","last_became_active":"2024-09-14T04:31:47.361606+0000","last_became_peered":"2024-09-14T04:31:47.361606+0000","last_unstale":"2024-09-14T04:33:02.506360+0000","last_undegraded":"2024-09-14T04:33:02.506360+0000","last_fullsized":"2024-09-14T04:33:02.506360+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:31:02.298329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:31:02.298329+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:52:41.585691+0000","scrub_duration":0.0012390960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"62'2","reported_seq":43,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-14T04:33:02.514865+0000","last_change":"2024-09-14T04:31:47.370767+0000","last_active":"2024-09-14T04:33:02.514865+0000","last_peered":"2024-09-14T04:33:02.514865+0000","last_clean":"2024-09-14T04:33:02.514865+0000","last_became_active":"2024-09-14T04:31:47.353197+0000","last_became_peered":"2024-09-14T04:31:47.353197+0000","last_unstale":"2024-09-14T04:33:02.514865+0000","last_undegraded":"2024-09-14T04:33:02.514865+0000","last_fullsized":"2024-09-14T04:33:02.514865+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:10.239127+0000","last_clean_scrub_stamp":"2024-09-14T03:43:10.239127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:17:06.116532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":799,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.517228+0000","last_change":"2024-09-14T04:31:52.382891+0000","last_active":"2024-09-14T04:33:03.517228+0000","last_peered":"2024-09-14T04:33:03.517228+0000","last_clean":"2024-09-14T04:33:03.517228+0000","last_became_active":"2024-09-14T04:31:52.382589+0000","last_became_peered":"2024-09-14T04:31:52.382589+0000","last_unstale":"2024-09-14T04:33:03.517228+0000","last_undegraded":"2024-09-14T04:33:03.517228+0000","last_fullsized":"2024-09-14T04:33:03.517228+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:43:11.248170+0000","last_clean_scrub_stamp":"2024-09-14T03:43:11.248170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:19:13.265551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"76'100","reported_seq":175,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-09-14T04:33:03.513239+0000","last_change":"2024-09-14T04:32:14.761740+0000","last_active":"2024-09-14T04:33:03.513239+0000","last_peered":"2024-09-14T04:33:03.513239+0000","last_clean":"2024-09-14T04:33:03.513239+0000","last_became_active":"2024-09-14T04:32:14.761407+0000","last_became_peered":"2024-09-14T04:32:14.761407+0000","last_unstale":"2024-09-14T04:33:03.513239+0000","last_undegraded":"2024-09-14T04:33:03.513239+0000","last_fullsized":"2024-09-14T04:33:03.513239+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":424,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'100","last_scrub_stamp":"2024-09-14T04:30:58.248525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T03:39:29.365581+0000","last_clean_scrub_stamp":"2024-09-14T04:30:58.248525+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:37:49.472825+0000","scrub_duration":0.0016764589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":6942,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18150,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430842,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1511700,"kb_used_data":5828,"kb_used_omap":1,"kb_used_meta":1505854,"kb_avail":92233452,"statfs":{"total":95995035648,"available":94447054848,"internally_reserved":0,"allocated":5967872,"data_stored":3732555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1541994840},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388627071,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1511044,"kb_used_data":6136,"kb_used_omap":1,"kb_used_meta":1504894,"kb_avail":92234108,"statfs":{"total":95995035648,"available":94447726592,"internally_reserved":0,"allocated":6283264,"data_stored":4059793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1541011800},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823299,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2088504,"kb_used_data":6064,"kb_used_omap":1,"kb_used_meta":2082430,"kb_avail":91656648,"statfs":{"total":95995035648,"available":93856407552,"internally_reserved":0,"allocated":6209536,"data_stored":3990312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":2132408664},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":186,"seq":798863917381,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":944584,"kb_used_data":5820,"kb_used_omap":6,"kb_used_meta":938745,"kb_avail":92800568,"statfs":{"total":95995035648,"available":95027781632,"internally_reserved":0,"allocated":5959680,"data_stored":3732170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6145,"internal_metadata":961275903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":136,"seq":584115552680,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267008,"kb_used_data":6384,"kb_used_omap":4,"kb_used_meta":1260603,"kb_avail":92478144,"statfs":{"total":95995035648,"available":94697619456,"internally_reserved":0,"allocated":6537216,"data_stored":4318073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4675,"internal_metadata":1290857917},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":109,"seq":468151435721,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1280508,"kb_used_data":6060,"kb_used_omap":6,"kb_used_meta":1274425,"kb_avail":92464644,"statfs":{"total":95995035648,"available":94683795456,"internally_reserved":0,"allocated":6205440,"data_stored":3989445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6707,"internal_metadata":1305011661},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":97,"seq":416611828191,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1081852,"kb_used_data":5804,"kb_used_omap":5,"kb_used_meta":1076026,"kb_avail":92663300,"statfs":{"total":95995035648,"available":94887219200,"internally_reserved":0,"allocated":5943296,"data_stored":3727634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1101850863},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":80,"seq":343597384180,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1189948,"kb_used_data":6124,"kb_used_omap":5,"kb_used_meta":1183802,"kb_avail":92555204,"statfs":{"total":95995035648,"available":94776528896,"internally_reserved":0,"allocated":6270976,"data_stored":4055588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":1212213308},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2971,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T04:33:53.879 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-14T04:33:53.880 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_1c941a66c1b420259b8fc5ac19e792dbbf60aded/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_1c941a66c1b420259b8fc5ac19e792dbbf60aded/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-14T04:33:54.094 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f067d73077124af99eab271ce9e50283 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_1c941a66c1b420259b8fc5ac19e792dbbf60aded/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_1c941a66c1b420259b8fc5ac19e792dbbf60aded/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-14T04:33:54.098 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-14T04:33:54.109 INFO:tasks.cephadm:Teardown begin 2024-09-14T04:33:54.109 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-14T04:33:54.143 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-14T04:33:54.179 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-14T04:33:54.179 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-14T04:33:54.199 DEBUG:teuthology.orchestra.run.smithi160:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-14T04:33:54.237 INFO:tasks.cephadm:Stopping all daemons... 2024-09-14T04:33:54.237 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-14T04:33:54.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.a 2024-09-14T04:33:54.531 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:54 smithi114 ceph-mon[106969]: from='client.37662 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:33:54.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:54 smithi114 ceph-mon[106969]: pgmap v1750: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:54.532 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:54 smithi114 systemd[1]: Stopping Ceph mon.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:33:54.532 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:54 smithi114 ceph-mon[103029]: from='client.37662 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:33:54.532 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:54 smithi114 ceph-mon[103029]: pgmap v1750: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:54 smithi160 ceph-mon[88407]: from='client.37662 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-14T04:33:54.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:54 smithi160 ceph-mon[88407]: pgmap v1750: 93 pgs: 93 active+clean; 582 KiB data, 10 GiB used, 705 GiB / 715 GiB avail 2024-09-14T04:33:54.785 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:54] ENGINE Bus STOPPING 2024-09-14T04:33:54.786 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:33:54.529+0000 7f316872e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:33:54.786 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a[102980]: 2024-09-14T04:33:54.530+0000 7f316872e640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-14T04:33:55.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:54 smithi114 podman[369212]: 2024-09-14 04:33:54.78538301 +0000 UTC m=+0.365350924 container died 2ca35e38ecf565399265397d6adaf8efb8940587ecad29b42f482e94c49dd58a (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, RELEASE=main-a278231, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T04:33:55.042 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-14T04:33:55.042 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:54] ENGINE Bus STOPPED 2024-09-14T04:33:55.042 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:54 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:54] ENGINE Bus STARTING 2024-09-14T04:33:55.296 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:55 smithi114 podman[369212]: 2024-09-14 04:33:55.280895444 +0000 UTC m=+0.860863360 container cleanup 2ca35e38ecf565399265397d6adaf8efb8940587ecad29b42f482e94c49dd58a (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-a278231, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-14T04:33:55.296 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 14 04:33:55 smithi114 bash[369212]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-a 2024-09-14T04:33:55.297 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:55] ENGINE Serving on http://:::9283 2024-09-14T04:33:55.297 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:55 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:55] ENGINE Bus STARTED 2024-09-14T04:33:55.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.a.service' 2024-09-14T04:33:55.756 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:33:55.757 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-14T04:33:55.757 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-14T04:33:55.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.c 2024-09-14T04:33:55.870 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:55 smithi114 systemd[1]: Stopping Ceph mon.c for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:33:56.312 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:56] ENGINE Bus STOPPING 2024-09-14T04:33:56.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c[106919]: 2024-09-14T04:33:56.045+0000 7fcc3c2ae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:33:56.313 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c[106919]: 2024-09-14T04:33:56.045+0000 7fcc3c2ae640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-09-14T04:33:56.605 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:56 smithi114 podman[369353]: 2024-09-14 04:33:56.31208524 +0000 UTC m=+0.370506353 container died 0d84986478c252d1c0c3825a10e139c5f5975678f330cdcd957cd7af280dd81e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-a278231) 2024-09-14T04:33:56.606 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-14T04:33:56.606 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:56] ENGINE Bus STOPPED 2024-09-14T04:33:56.606 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:56] ENGINE Bus STARTING 2024-09-14T04:33:56.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:56] ENGINE Serving on http://:::9283 2024-09-14T04:33:56.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:56 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y[93521]: [14/Sep/2024:04:33:56] ENGINE Bus STARTED 2024-09-14T04:33:56.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:56 smithi114 podman[369353]: 2024-09-14 04:33:56.702824342 +0000 UTC m=+0.761245410 container cleanup 0d84986478c252d1c0c3825a10e139c5f5975678f330cdcd957cd7af280dd81e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-a278231, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-14T04:33:56.871 INFO:journalctl@ceph.mon.c.smithi114.stdout:Sep 14 04:33:56 smithi114 bash[369353]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-c 2024-09-14T04:33:57.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.c.service' 2024-09-14T04:33:57.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:33:57.167 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-14T04:33:57.167 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-14T04:33:57.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.b 2024-09-14T04:33:57.608 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:57 smithi160 systemd[1]: Stopping Ceph mon.b for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:33:58.471 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:58 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b[88384]: 2024-09-14T04:33:58.179+0000 7f82ccdd6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:33:58.471 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:58 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b[88384]: 2024-09-14T04:33:58.179+0000 7f82ccdd6640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-09-14T04:33:58.471 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:58 smithi160 podman[242896]: 2024-09-14 04:33:58.387820124 +0000 UTC m=+0.631369300 container died c470c18a013d7d2f41fd3a1d637d2a29b75acf750a93afb161e42f2dbdcc51f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=main-a278231, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-14T04:33:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:58 smithi160 podman[242896]: 2024-09-14 04:33:58.855289122 +0000 UTC m=+1.098838332 container cleanup c470c18a013d7d2f41fd3a1d637d2a29b75acf750a93afb161e42f2dbdcc51f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-14T04:33:59.158 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:58 smithi160 bash[242896]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b 2024-09-14T04:33:59.658 INFO:journalctl@ceph.mon.b.smithi160.stdout:Sep 14 04:33:59 smithi160 podman[242969]: 2024-09-14 04:33:59.266378836 +0000 UTC m=+0.930686154 container remove c470c18a013d7d2f41fd3a1d637d2a29b75acf750a93afb161e42f2dbdcc51f6 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mon-b, RELEASE=main-a278231, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-14T04:33:59.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@mon.b.service' 2024-09-14T04:33:59.827 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:33:59.828 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-14T04:33:59.828 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-14T04:33:59.828 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.y 2024-09-14T04:34:00.121 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:33:59 smithi114 systemd[1]: Stopping Ceph mgr.y for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:34:00.425 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:34:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:34:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"63f4e762-724a-11ef-bceb-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-14T04:34:00.468 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:34:00 smithi114 podman[369498]: 2024-09-14 04:34:00.175956108 +0000 UTC m=+0.155090376 container died 4f6e65854aef018510965487055eab011e7b65549dde665524b668e4b38955d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-a278231, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T04:34:00.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:34:00 smithi114 podman[369498]: 2024-09-14 04:34:00.674709485 +0000 UTC m=+0.653843741 container cleanup 4f6e65854aef018510965487055eab011e7b65549dde665524b668e4b38955d0 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T04:34:00.870 INFO:journalctl@ceph.mgr.y.smithi114.stdout:Sep 14 04:34:00 smithi114 bash[369498]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-y 2024-09-14T04:34:01.101 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-14T04:34:01.135 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:34:01.136 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-14T04:34:01.136 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-14T04:34:01.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.x 2024-09-14T04:34:01.413 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 04:34:01 smithi160 systemd[1]: Stopping Ceph mgr.x for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:34:01.781 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 04:34:01 smithi160 podman[243142]: 2024-09-14 04:34:01.50426123 +0000 UTC m=+0.167252805 container died 574eaed954d9adf878fd30c118bc756ac1e80b02d56a69fa1d935c358d4e9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-a278231, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-14T04:34:02.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 04:34:01 smithi160 podman[243142]: 2024-09-14 04:34:01.870902088 +0000 UTC m=+0.533893663 container cleanup 574eaed954d9adf878fd30c118bc756ac1e80b02d56a69fa1d935c358d4e9dd9 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-a278231, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-14T04:34:02.158 INFO:journalctl@ceph.mgr.x.smithi160.stdout:Sep 14 04:34:01 smithi160 bash[243142]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-mgr-x 2024-09-14T04:34:02.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-14T04:34:02.353 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:34:02.353 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-14T04:34:02.353 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-14T04:34:02.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.0 2024-09-14T04:34:02.656 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:02 smithi114 systemd[1]: Stopping Ceph osd.0 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:34:03.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:02 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[113954]: 2024-09-14T04:34:02.654+0000 7fd5e9174640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:34:03.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:02 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[113954]: 2024-09-14T04:34:02.654+0000 7fd5e9174640 -1 osd.0 473 *** Got signal Terminated *** 2024-09-14T04:34:03.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:02 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0[113954]: 2024-09-14T04:34:02.654+0000 7fd5e9174640 -1 osd.0 473 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T04:34:08.596 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:08 smithi114 podman[369628]: 2024-09-14 04:34:08.294866968 +0000 UTC m=+5.747186949 container died bde17c62ee62fce117b7d483685503f059f9a5dea92f09d27b109e1b66b70b76 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-a278231, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-14T04:34:08.870 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:08 smithi114 podman[369628]: 2024-09-14 04:34:08.770988188 +0000 UTC m=+6.223308166 container cleanup bde17c62ee62fce117b7d483685503f059f9a5dea92f09d27b109e1b66b70b76 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=main-a278231, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-14T04:34:08.870 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:08 smithi114 bash[369628]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0 2024-09-14T04:34:09.534 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:09 smithi114 podman[369650]: 2024-09-14 04:34:09.189532568 +0000 UTC m=+0.877539015 container remove bde17c62ee62fce117b7d483685503f059f9a5dea92f09d27b109e1b66b70b76 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0, RELEASE=main-a278231, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-14T04:34:09.801 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:09 smithi114 podman[369744]: 2024-09-14 04:34:09.437174771 +0000 UTC m=+0.016040157 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T04:34:09.802 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:09 smithi114 podman[369744]: 2024-09-14 04:34:09.540366457 +0000 UTC m=+0.119231881 container create e714b4e0c6064fb77f22ee93aaa8ef899056058b7138d9ea8738a3b01571bec3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=main-a278231, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T04:34:09.802 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:09 smithi114 podman[369744]: 2024-09-14 04:34:09.801475407 +0000 UTC m=+0.380340789 container init e714b4e0c6064fb77f22ee93aaa8ef899056058b7138d9ea8738a3b01571bec3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-a278231, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-09-14T04:34:10.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:09 smithi114 podman[369744]: 2024-09-14 04:34:09.806584346 +0000 UTC m=+0.385449728 container start e714b4e0c6064fb77f22ee93aaa8ef899056058b7138d9ea8738a3b01571bec3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-a278231, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-14T04:34:10.120 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:09 smithi114 podman[369744]: 2024-09-14 04:34:09.901229779 +0000 UTC m=+0.480095175 container attach e714b4e0c6064fb77f22ee93aaa8ef899056058b7138d9ea8738a3b01571bec3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-a278231, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-14T04:34:11.072 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 14 04:34:10 smithi114 podman[369744]: 2024-09-14 04:34:10.734330309 +0000 UTC m=+1.313195695 container died e714b4e0c6064fb77f22ee93aaa8ef899056058b7138d9ea8738a3b01571bec3 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-a278231, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-14T04:34:11.363 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.0.service' 2024-09-14T04:34:11.402 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:34:11.402 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-14T04:34:11.402 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-14T04:34:11.403 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.1 2024-09-14T04:34:11.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:11 smithi114 systemd[1]: Stopping Ceph osd.1 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:34:12.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[120675]: 2024-09-14T04:34:11.760+0000 7f86df82a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:34:12.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[120675]: 2024-09-14T04:34:11.760+0000 7f86df82a640 -1 osd.1 473 *** Got signal Terminated *** 2024-09-14T04:34:12.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1[120675]: 2024-09-14T04:34:11.760+0000 7f86df82a640 -1 osd.1 473 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T04:34:13.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:34:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:34:13.560Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-14T04:34:13.908 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:34:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:34:13.562Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-14T04:34:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:34:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:34:13.563Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-14T04:34:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:34:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:34:13.565Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-14T04:34:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:34:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:34:13.565Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-14T04:34:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:34:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:34:13.565Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-14T04:34:13.909 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:34:13 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:34:13.565Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.114:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.114:8765: connect: connection refused" 2024-09-14T04:34:17.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:17 smithi114 podman[369887]: 2024-09-14 04:34:17.361646816 +0000 UTC m=+5.726346356 container died 724ed57646205c415e6778611e1f4e840bb9f432c21b7b8d95affedd2872f18e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=main-a278231, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-14T04:34:18.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:17 smithi114 podman[369887]: 2024-09-14 04:34:17.866625757 +0000 UTC m=+6.231325327 container cleanup 724ed57646205c415e6778611e1f4e840bb9f432c21b7b8d95affedd2872f18e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-a278231, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-14T04:34:18.120 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:17 smithi114 bash[369887]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1 2024-09-14T04:34:18.620 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:18 smithi114 podman[369908]: 2024-09-14 04:34:18.218427817 +0000 UTC m=+0.797626493 container remove 724ed57646205c415e6778611e1f4e840bb9f432c21b7b8d95affedd2872f18e (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=main-a278231, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-14T04:34:18.885 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:18 smithi114 podman[370003]: 2024-09-14 04:34:18.533032111 +0000 UTC m=+0.016492427 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T04:34:18.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:18 smithi114 podman[370003]: 2024-09-14 04:34:18.644233959 +0000 UTC m=+0.127694307 container create 77363fcfa6d8d4a63ebea79e8b383c6958c18d60736b6fbfb273ad0c6feb3a1b (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a278231, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-09-14T04:34:18.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:18 smithi114 podman[370003]: 2024-09-14 04:34:18.880452056 +0000 UTC m=+0.363912360 container init 77363fcfa6d8d4a63ebea79e8b383c6958c18d60736b6fbfb273ad0c6feb3a1b (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=main-a278231, org.label-schema.license=GPLv2) 2024-09-14T04:34:18.886 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:18 smithi114 podman[370003]: 2024-09-14 04:34:18.885189147 +0000 UTC m=+0.368649453 container start 77363fcfa6d8d4a63ebea79e8b383c6958c18d60736b6fbfb273ad0c6feb3a1b (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=main-a278231, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-14T04:34:19.370 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:18 smithi114 podman[370003]: 2024-09-14 04:34:18.971850479 +0000 UTC m=+0.455310783 container attach 77363fcfa6d8d4a63ebea79e8b383c6958c18d60736b6fbfb273ad0c6feb3a1b (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, ceph=True, RELEASE=main-a278231, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-14T04:34:19.371 INFO:journalctl@ceph.osd.1.smithi114.stdout:Sep 14 04:34:19 smithi114 podman[370003]: 2024-09-14 04:34:19.074093969 +0000 UTC m=+0.557554283 container died 77363fcfa6d8d4a63ebea79e8b383c6958c18d60736b6fbfb273ad0c6feb3a1b (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-a278231, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-14T04:34:19.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.1.service' 2024-09-14T04:34:19.892 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:34:19.893 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-14T04:34:19.893 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-14T04:34:19.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.2 2024-09-14T04:34:20.120 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:19 smithi114 systemd[1]: Stopping Ceph osd.2 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:34:20.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[128013]: 2024-09-14T04:34:20.205+0000 7f6b174d8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:34:20.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[128013]: 2024-09-14T04:34:20.205+0000 7f6b174d8640 -1 osd.2 473 *** Got signal Terminated *** 2024-09-14T04:34:20.621 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:20 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2[128013]: 2024-09-14T04:34:20.205+0000 7f6b174d8640 -1 osd.2 473 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T04:34:25.938 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:25 smithi114 podman[370131]: 2024-09-14 04:34:25.65180267 +0000 UTC m=+5.542855574 container died f3696f57b771b2ab55ca6defccd4d170c3ed7be670e84a6b3aed2cf789cb4797 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-a278231) 2024-09-14T04:34:26.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:26 smithi114 podman[370131]: 2024-09-14 04:34:26.019826313 +0000 UTC m=+5.910879224 container cleanup f3696f57b771b2ab55ca6defccd4d170c3ed7be670e84a6b3aed2cf789cb4797 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-14T04:34:26.305 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:26 smithi114 bash[370131]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2 2024-09-14T04:34:26.620 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:26 smithi114 podman[370151]: 2024-09-14 04:34:26.304736995 +0000 UTC m=+0.652369989 container remove f3696f57b771b2ab55ca6defccd4d170c3ed7be670e84a6b3aed2cf789cb4797 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=main-a278231, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T04:34:26.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:26 smithi114 podman[370246]: 2024-09-14 04:34:26.532043968 +0000 UTC m=+0.015573998 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T04:34:26.958 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:26 smithi114 podman[370246]: 2024-09-14 04:34:26.638978175 +0000 UTC m=+0.122508202 container create 01ee18588ca971592d0c37cd61c3988ad061d69c5508ab58b597a02418357db4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-a278231, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-14T04:34:26.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:26 smithi114 podman[370246]: 2024-09-14 04:34:26.866288214 +0000 UTC m=+0.349818241 container init 01ee18588ca971592d0c37cd61c3988ad061d69c5508ab58b597a02418357db4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T04:34:26.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:26 smithi114 podman[370246]: 2024-09-14 04:34:26.871423413 +0000 UTC m=+0.354953440 container start 01ee18588ca971592d0c37cd61c3988ad061d69c5508ab58b597a02418357db4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-14T04:34:26.959 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:26 smithi114 podman[370246]: 2024-09-14 04:34:26.957797371 +0000 UTC m=+0.441327401 container attach 01ee18588ca971592d0c37cd61c3988ad061d69c5508ab58b597a02418357db4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-a278231, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-14T04:34:27.344 INFO:journalctl@ceph.osd.2.smithi114.stdout:Sep 14 04:34:27 smithi114 podman[370246]: 2024-09-14 04:34:27.058971137 +0000 UTC m=+0.542501171 container died 01ee18588ca971592d0c37cd61c3988ad061d69c5508ab58b597a02418357db4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-14T04:34:27.675 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.2.service' 2024-09-14T04:34:27.713 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:34:27.714 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-14T04:34:27.714 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-14T04:34:27.714 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.3 2024-09-14T04:34:28.043 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:27 smithi114 systemd[1]: Stopping Ceph osd.3 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:34:28.370 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:28 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[138073]: 2024-09-14T04:34:28.041+0000 7efe849e9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:34:28.370 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:28 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[138073]: 2024-09-14T04:34:28.041+0000 7efe849e9640 -1 osd.3 473 *** Got signal Terminated *** 2024-09-14T04:34:28.370 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:28 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[138073]: 2024-09-14T04:34:28.041+0000 7efe849e9640 -1 osd.3 473 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T04:34:30.126 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:29 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:29.746+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:30.620 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:30 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[138073]: 2024-09-14T04:34:30.174+0000 7efe7cff6640 -1 osd.3 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.755041+0000 front 2024-09-14T04:34:05.755022+0000 (oldest deadline 2024-09-14T04:34:29.255028+0000) 2024-09-14T04:34:30.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:30.396+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:31.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:30 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:30.768+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:31.620 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:31 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[138073]: 2024-09-14T04:34:31.199+0000 7efe7cff6640 -1 osd.3 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.755041+0000 front 2024-09-14T04:34:05.755022+0000 (oldest deadline 2024-09-14T04:34:29.255028+0000) 2024-09-14T04:34:31.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:31 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:31.382+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:31.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:31 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:31.256+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:32.158 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:31 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:31.994+0000 7f194f66b640 -1 osd.4 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:06.962568+0000 front 2024-09-14T04:34:06.962506+0000 (oldest deadline 2024-09-14T04:34:31.662359+0000) 2024-09-14T04:34:32.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:31 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:31.755+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:32.620 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:32 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3[138073]: 2024-09-14T04:34:32.219+0000 7efe7cff6640 -1 osd.3 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.755041+0000 front 2024-09-14T04:34:05.755022+0000 (oldest deadline 2024-09-14T04:34:29.255028+0000) 2024-09-14T04:34:32.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:32 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:32.342+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:32.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:32 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:32.236+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:33.023 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:32 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:32.708+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:33.386 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:33.257+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:33.386 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:33.021+0000 7f194f66b640 -1 osd.4 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:06.962568+0000 front 2024-09-14T04:34:06.962506+0000 (oldest deadline 2024-09-14T04:34:31.662359+0000) 2024-09-14T04:34:33.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:33.385+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:33.997 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:33.666+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:34.048 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:33 smithi114 podman[370376]: 2024-09-14 04:34:33.774235692 +0000 UTC m=+5.839120132 container died 10460276d62fcf22147d36b06f9014b1b03d3c3af22378e54aa071f19a9d0de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-a278231, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-14T04:34:34.370 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:34 smithi114 podman[370376]: 2024-09-14 04:34:34.222348773 +0000 UTC m=+6.287233254 container cleanup 10460276d62fcf22147d36b06f9014b1b03d3c3af22378e54aa071f19a9d0de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-a278231, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True) 2024-09-14T04:34:34.370 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:34 smithi114 bash[370376]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3 2024-09-14T04:34:34.389 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:33 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:33.996+0000 7f194f66b640 -1 osd.4 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:06.962568+0000 front 2024-09-14T04:34:06.962506+0000 (oldest deadline 2024-09-14T04:34:31.662359+0000) 2024-09-14T04:34:34.390 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:34 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:34.241+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:34.651 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:34 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:34.388+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:34.870 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:34 smithi114 podman[370396]: 2024-09-14 04:34:34.565293784 +0000 UTC m=+0.785619758 container remove 10460276d62fcf22147d36b06f9014b1b03d3c3af22378e54aa071f19a9d0de4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-a278231) 2024-09-14T04:34:34.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:34 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:34.650+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:35.185 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:34 smithi114 podman[370493]: 2024-09-14 04:34:34.788041826 +0000 UTC m=+0.015759538 image pull 8801d2529e4377d97015e6c85a3b4925ede7ce42c7349a1a37eb88f097b6f3f0 quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802 2024-09-14T04:34:35.186 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:34 smithi114 podman[370493]: 2024-09-14 04:34:34.89089884 +0000 UTC m=+0.118616541 container create f7f71be99ba3b5cb7e39d2d74e26aa1ed82f15f4c6a524888d77e97387acd0b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a278231, ceph=True) 2024-09-14T04:34:35.186 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:35 smithi114 podman[370493]: 2024-09-14 04:34:35.109664485 +0000 UTC m=+0.337382182 container init f7f71be99ba3b5cb7e39d2d74e26aa1ed82f15f4c6a524888d77e97387acd0b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=main-a278231, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-14T04:34:35.186 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:35 smithi114 podman[370493]: 2024-09-14 04:34:35.114283936 +0000 UTC m=+0.342001637 container start f7f71be99ba3b5cb7e39d2d74e26aa1ed82f15f4c6a524888d77e97387acd0b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a278231, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-14T04:34:35.186 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:35 smithi114 podman[370493]: 2024-09-14 04:34:35.184586521 +0000 UTC m=+0.412304239 container attach f7f71be99ba3b5cb7e39d2d74e26aa1ed82f15f4c6a524888d77e97387acd0b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=main-a278231, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-14T04:34:35.274 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:34 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:34.976+0000 7f194f66b640 -1 osd.4 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:06.962568+0000 front 2024-09-14T04:34:06.962506+0000 (oldest deadline 2024-09-14T04:34:31.662359+0000) 2024-09-14T04:34:35.590 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:35 smithi114 podman[370493]: 2024-09-14 04:34:35.305928389 +0000 UTC m=+0.533646097 container died f7f71be99ba3b5cb7e39d2d74e26aa1ed82f15f4c6a524888d77e97387acd0b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-a278231, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-14T04:34:35.626 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:35.415+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:35.627 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:35.273+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:35.842 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 14 04:34:35 smithi114 podman[370536]: 2024-09-14 04:34:35.823084617 +0000 UTC m=+0.509846416 container remove f7f71be99ba3b5cb7e39d2d74e26aa1ed82f15f4c6a524888d77e97387acd0b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:862445896ef1df9aa3acdb9cfb761812e8e943c37f87d60be1b51d2242056802, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=main-a278231, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-14T04:34:35.847 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.3.service' 2024-09-14T04:34:35.885 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:34:35.885 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-14T04:34:35.886 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-14T04:34:35.886 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.4 2024-09-14T04:34:35.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:35.625+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:36.209 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:35 smithi160 systemd[1]: Stopping Ceph osd.4 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:34:36.209 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:35 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:35.938+0000 7f194f66b640 -1 osd.4 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:06.962568+0000 front 2024-09-14T04:34:06.962506+0000 (oldest deadline 2024-09-14T04:34:31.662359+0000) 2024-09-14T04:34:36.642 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:36.376+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:36.642 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:36.225+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:36.643 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:36.207+0000 7f1952051640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:34:36.643 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:36.207+0000 7f1952051640 -1 osd.4 473 *** Got signal Terminated *** 2024-09-14T04:34:36.643 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:36.207+0000 7f1952051640 -1 osd.4 473 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T04:34:36.900 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:36.640+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:37.158 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:36 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:36.899+0000 7f194f66b640 -1 osd.4 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:06.962568+0000 front 2024-09-14T04:34:06.962506+0000 (oldest deadline 2024-09-14T04:34:31.662359+0000) 2024-09-14T04:34:37.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:37.400+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:37.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:37.269+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:38.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:37.688+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:38.158 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:37 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:37.900+0000 7f194f66b640 -1 osd.4 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:06.962568+0000 front 2024-09-14T04:34:06.962506+0000 (oldest deadline 2024-09-14T04:34:31.662359+0000) 2024-09-14T04:34:38.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:38.394+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:38.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:38.276+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:38.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:38.276+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:39.158 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:38.879+0000 7f194f66b640 -1 osd.4 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:06.962568+0000 front 2024-09-14T04:34:06.962506+0000 (oldest deadline 2024-09-14T04:34:31.662359+0000) 2024-09-14T04:34:39.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:38.665+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:39.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:38 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:38.665+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:39.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:39 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:39.286+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:39.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:39 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:39.286+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:39.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:39 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:39.401+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:39.659 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:39 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:39.402+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:40.158 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:39 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:39.849+0000 7f194f66b640 -1 osd.4 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:06.962568+0000 front 2024-09-14T04:34:06.962506+0000 (oldest deadline 2024-09-14T04:34:31.662359+0000) 2024-09-14T04:34:40.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:39 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:39.675+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:40.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:39 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:39.675+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:40.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:40.272+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:40.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:40.272+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:40.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:40.417+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:40.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:40.417+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:41.158 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4[115627]: 2024-09-14T04:34:40.843+0000 7f194f66b640 -1 osd.4 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:06.962568+0000 front 2024-09-14T04:34:06.962506+0000 (oldest deadline 2024-09-14T04:34:31.662359+0000) 2024-09-14T04:34:41.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:40.702+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:41.159 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:40 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:40.702+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:41.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:41.410+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:41.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:41.410+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:41.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:41.318+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:41.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:41.318+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:42.042 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:41.692+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:42.042 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:41 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:41.692+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:42.042 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:41 smithi160 podman[243290]: 2024-09-14 04:34:41.74536099 +0000 UTC m=+5.661413784 container died f0bdd007efc97506375670ae82baa52fc5b536b4842ccfd06220394d7030839f (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=quincy-eb16b97, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-14T04:34:42.408 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:42 smithi160 podman[243290]: 2024-09-14 04:34:42.24793907 +0000 UTC m=+6.163991863 container cleanup f0bdd007efc97506375670ae82baa52fc5b536b4842ccfd06220394d7030839f (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-eb16b97, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-14T04:34:42.408 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:42 smithi160 bash[243290]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4 2024-09-14T04:34:42.408 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:42 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:42.283+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:42.408 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:42 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:42.283+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:42.853 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:42 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:42.677+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:42.854 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:42 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:42.677+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:42.854 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:42 smithi160 podman[243311]: 2024-09-14 04:34:42.457983851 +0000 UTC m=+0.714825654 container remove f0bdd007efc97506375670ae82baa52fc5b536b4842ccfd06220394d7030839f (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-eb16b97, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-14T04:34:42.854 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:42 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:42.446+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:42.855 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:42 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:42.446+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:43.137 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:42 smithi160 podman[243401]: 2024-09-14 04:34:42.75569375 +0000 UTC m=+0.016102776 image pull ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6 2024-09-14T04:34:43.137 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:42 smithi160 podman[243401]: 2024-09-14 04:34:42.897070612 +0000 UTC m=+0.157479671 container create 53113b546d03983d51a2b7ab57aba694fd16a881107ce7850b7e4782be4dbfa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=quincy-eb16b97, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-14T04:34:43.408 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:43 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:43.322+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:43.408 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:43 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:43.322+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:43.409 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:43 smithi160 podman[243401]: 2024-09-14 04:34:43.136551369 +0000 UTC m=+0.396960382 container init 53113b546d03983d51a2b7ab57aba694fd16a881107ce7850b7e4782be4dbfa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-14T04:34:43.409 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:43 smithi160 podman[243401]: 2024-09-14 04:34:43.142572499 +0000 UTC m=+0.402981515 container start 53113b546d03983d51a2b7ab57aba694fd16a881107ce7850b7e4782be4dbfa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=quincy-eb16b97, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-14T04:34:43.409 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:43 smithi160 podman[243401]: 2024-09-14 04:34:43.219713329 +0000 UTC m=+0.480122368 container attach 53113b546d03983d51a2b7ab57aba694fd16a881107ce7850b7e4782be4dbfa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=quincy-eb16b97, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-14T04:34:43.409 INFO:journalctl@ceph.osd.4.smithi160.stdout:Sep 14 04:34:43 smithi160 podman[243401]: 2024-09-14 04:34:43.38101731 +0000 UTC m=+0.641426344 container died 53113b546d03983d51a2b7ab57aba694fd16a881107ce7850b7e4782be4dbfa8 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=quincy-eb16b97, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=) 2024-09-14T04:34:43.748 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:43 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:43.422+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:43.748 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:43 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:43.422+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:43.749 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:43 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:43.663+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:43.749 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:43 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:43.663+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:44.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.4.service' 2024-09-14T04:34:44.147 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:34:44.147 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-14T04:34:44.147 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-14T04:34:44.148 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.5 2024-09-14T04:34:44.306 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:44 smithi160 systemd[1]: Stopping Ceph osd.5 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:34:44.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:44.305+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:44.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:44.305+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:44.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:44.409+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:44.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:44.409+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:44.659 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:44.478+0000 7f4896d31640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:34:44.659 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:44.478+0000 7f4896d31640 -1 osd.5 473 *** Got signal Terminated *** 2024-09-14T04:34:44.659 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:44.478+0000 7f4896d31640 -1 osd.5 473 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T04:34:45.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:44.712+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:45.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:44 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:44.712+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:45.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:45.303+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:45.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:45.303+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:45.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:45.368+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:45.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:45.368+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:46.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:45.745+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:46.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:45 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:45.745+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:46.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:46.363+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:46.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:46.363+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:46.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:46.321+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:46.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:46.321+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:47.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:46.707+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:47.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:46 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:46.707+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:47.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:47.337+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:47.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:47.337+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:47.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:47.399+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:47.659 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:47.399+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:48.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:47.703+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:48.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:47.703+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:48.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:47 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:47.703+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:48.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:48.389+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:48.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:48.389+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:48.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:48.346+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:48.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:48.346+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:48.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:48.346+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:25.025569+0000 front 2024-09-14T04:34:25.025584+0000 (oldest deadline 2024-09-14T04:34:47.825956+0000) 2024-09-14T04:34:49.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:48.714+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:49.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:48.714+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:49.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:48 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:48.714+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:49.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:49.420+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.900086+0000 front 2024-09-14T04:34:04.900026+0000 (oldest deadline 2024-09-14T04:34:29.600120+0000) 2024-09-14T04:34:49.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:49.420+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.701223+0000 front 2024-09-14T04:34:13.701246+0000 (oldest deadline 2024-09-14T04:34:38.401242+0000) 2024-09-14T04:34:49.659 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5[44371]: 2024-09-14T04:34:49.420+0000 7f489434b640 -1 osd.5 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.502483+0000 front 2024-09-14T04:34:22.502438+0000 (oldest deadline 2024-09-14T04:34:48.402169+0000) 2024-09-14T04:34:49.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:49.337+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:49.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:49.337+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:49.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:49.337+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:25.025569+0000 front 2024-09-14T04:34:25.025584+0000 (oldest deadline 2024-09-14T04:34:47.825956+0000) 2024-09-14T04:34:50.008 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:49.750+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:50.009 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:49.750+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:50.009 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:49 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:49.750+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:50.261 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:50 smithi160 podman[243528]: 2024-09-14 04:34:50.00821175 +0000 UTC m=+5.641423693 container died fb664f7f461a6332d3b6402ee11a69a5210efbf696665f98204a3f1e563b2cc1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5, RELEASE=quincy-eb16b97, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-14T04:34:50.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:50 smithi160 podman[243528]: 2024-09-14 04:34:50.426384393 +0000 UTC m=+6.059596377 container cleanup fb664f7f461a6332d3b6402ee11a69a5210efbf696665f98204a3f1e563b2cc1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-eb16b97) 2024-09-14T04:34:50.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:50 smithi160 bash[243528]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5 2024-09-14T04:34:50.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:50.381+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:50.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:50.381+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:50.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:50.381+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:25.025569+0000 front 2024-09-14T04:34:25.025584+0000 (oldest deadline 2024-09-14T04:34:47.825956+0000) 2024-09-14T04:34:51.097 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:50 smithi160 podman[243549]: 2024-09-14 04:34:50.697497832 +0000 UTC m=+0.683879749 container remove fb664f7f461a6332d3b6402ee11a69a5210efbf696665f98204a3f1e563b2cc1 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-14T04:34:51.098 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:50.726+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:51.098 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:50.726+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:51.098 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:50 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:50.726+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:51.406 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:51 smithi160 podman[243640]: 2024-09-14 04:34:51.000234351 +0000 UTC m=+0.016802214 image pull ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6 2024-09-14T04:34:51.406 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:51 smithi160 podman[243640]: 2024-09-14 04:34:51.112102237 +0000 UTC m=+0.128670145 container create b20f1733c166ad5e9ec584133b81d5333d84ebe8de4263b5f74de277693a1deb (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-14T04:34:51.406 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:51 smithi160 podman[243640]: 2024-09-14 04:34:51.322579844 +0000 UTC m=+0.339147707 container init b20f1733c166ad5e9ec584133b81d5333d84ebe8de4263b5f74de277693a1deb (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-eb16b97, org.label-schema.license=GPLv2) 2024-09-14T04:34:51.407 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:51 smithi160 podman[243640]: 2024-09-14 04:34:51.327416349 +0000 UTC m=+0.343984214 container start b20f1733c166ad5e9ec584133b81d5333d84ebe8de4263b5f74de277693a1deb (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=quincy-eb16b97) 2024-09-14T04:34:51.407 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:51 smithi160 podman[243640]: 2024-09-14 04:34:51.405764375 +0000 UTC m=+0.422332255 container attach b20f1733c166ad5e9ec584133b81d5333d84ebe8de4263b5f74de277693a1deb (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-09-14T04:34:51.658 INFO:journalctl@ceph.osd.5.smithi160.stdout:Sep 14 04:34:51 smithi160 podman[243640]: 2024-09-14 04:34:51.502113238 +0000 UTC m=+0.518681109 container died b20f1733c166ad5e9ec584133b81d5333d84ebe8de4263b5f74de277693a1deb (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=quincy-eb16b97, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T04:34:51.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:51.417+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:51.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:51.417+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:51.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:51.417+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:25.025569+0000 front 2024-09-14T04:34:25.025584+0000 (oldest deadline 2024-09-14T04:34:47.825956+0000) 2024-09-14T04:34:52.135 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:51.692+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:52.135 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:51.692+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:52.135 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:51 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:51.692+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:52.159 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.5.service' 2024-09-14T04:34:52.199 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:34:52.200 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-14T04:34:52.200 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-14T04:34:52.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.6 2024-09-14T04:34:52.400 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:52 smithi160 systemd[1]: Stopping Ceph osd.6 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:34:52.401 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:52.399+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:52.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:52.399+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:52.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:52.399+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:25.025569+0000 front 2024-09-14T04:34:25.025584+0000 (oldest deadline 2024-09-14T04:34:47.825956+0000) 2024-09-14T04:34:52.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:52.539+0000 7fd3e25f1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:34:52.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:52.539+0000 7fd3e25f1640 -1 osd.6 473 *** Got signal Terminated *** 2024-09-14T04:34:52.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:52.539+0000 7fd3e25f1640 -1 osd.6 473 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T04:34:53.157 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:52.680+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:53.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:52.680+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:53.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:52 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:52.680+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:53.908 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:53 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:53.417+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:53.908 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:53 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:53.417+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:53.908 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:53 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:53.417+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:25.025569+0000 front 2024-09-14T04:34:25.025584+0000 (oldest deadline 2024-09-14T04:34:47.825956+0000) 2024-09-14T04:34:53.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:53 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:53.642+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:53.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:53 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:53.642+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:53.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:53 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:53.642+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:54.658 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:54.625+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:54.658 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:54.625+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:54.658 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:54.625+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:54.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:54.400+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:54.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:54.400+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:54.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:54.400+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:25.025569+0000 front 2024-09-14T04:34:25.025584+0000 (oldest deadline 2024-09-14T04:34:47.825956+0000) 2024-09-14T04:34:54.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:54 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:54.400+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:30.126904+0000 front 2024-09-14T04:34:30.126850+0000 (oldest deadline 2024-09-14T04:34:54.226649+0000) 2024-09-14T04:34:55.642 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:55.383+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:55.642 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:55.383+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:55.642 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:55.383+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:25.025569+0000 front 2024-09-14T04:34:25.025584+0000 (oldest deadline 2024-09-14T04:34:47.825956+0000) 2024-09-14T04:34:55.642 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:55.383+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:30.126904+0000 front 2024-09-14T04:34:30.126850+0000 (oldest deadline 2024-09-14T04:34:54.226649+0000) 2024-09-14T04:34:55.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:55.641+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:55.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:55.641+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:55.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:55 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:55.641+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:56.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:56.355+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:56.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:56.355+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:56.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:56.355+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:25.025569+0000 front 2024-09-14T04:34:25.025584+0000 (oldest deadline 2024-09-14T04:34:47.825956+0000) 2024-09-14T04:34:56.659 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:56.355+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:30.126904+0000 front 2024-09-14T04:34:30.126850+0000 (oldest deadline 2024-09-14T04:34:54.226649+0000) 2024-09-14T04:34:56.659 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:56.596+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:56.659 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:56.596+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:56.659 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:56.596+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:56.659 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:56 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:56.596+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:27.830586+0000 front 2024-09-14T04:34:27.830599+0000 (oldest deadline 2024-09-14T04:34:56.031102+0000) 2024-09-14T04:34:57.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:57 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:57.350+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:05.123563+0000 front 2024-09-14T04:34:05.123478+0000 (oldest deadline 2024-09-14T04:34:30.423332+0000) 2024-09-14T04:34:57.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:57 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:57.350+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.924295+0000 front 2024-09-14T04:34:13.924381+0000 (oldest deadline 2024-09-14T04:34:38.024233+0000) 2024-09-14T04:34:57.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:57 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:57.350+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:25.025569+0000 front 2024-09-14T04:34:25.025584+0000 (oldest deadline 2024-09-14T04:34:47.825956+0000) 2024-09-14T04:34:57.658 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:57 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6[50288]: 2024-09-14T04:34:57.350+0000 7fd3ddc07640 -1 osd.6 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:30.126904+0000 front 2024-09-14T04:34:30.126850+0000 (oldest deadline 2024-09-14T04:34:54.226649+0000) 2024-09-14T04:34:57.659 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:57 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:57.588+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:57.659 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:57 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:57.588+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:57.659 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:57 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:57.588+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:57.659 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:57 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:57.588+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:27.830586+0000 front 2024-09-14T04:34:27.830599+0000 (oldest deadline 2024-09-14T04:34:56.031102+0000) 2024-09-14T04:34:58.448 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:58 smithi160 podman[243767]: 2024-09-14 04:34:58.172781539 +0000 UTC m=+5.735315106 container died 0403446dedd1ce0f36b5b9332ba0ec2cd8506cbe57e0f2505e964e248ca58afd (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=quincy-eb16b97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-14T04:34:58.908 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:58 smithi160 podman[243767]: 2024-09-14 04:34:58.629212915 +0000 UTC m=+6.191746523 container cleanup 0403446dedd1ce0f36b5b9332ba0ec2cd8506cbe57e0f2505e964e248ca58afd (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-14T04:34:58.908 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:58 smithi160 bash[243767]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6 2024-09-14T04:34:58.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:58 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:58.543+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:58.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:58 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:58.543+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:58.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:58 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:58.543+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:58.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:58 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:58.543+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:27.830586+0000 front 2024-09-14T04:34:27.830599+0000 (oldest deadline 2024-09-14T04:34:56.031102+0000) 2024-09-14T04:34:59.234 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:58 smithi160 podman[243789]: 2024-09-14 04:34:58.922395446 +0000 UTC m=+0.746561702 container remove 0403446dedd1ce0f36b5b9332ba0ec2cd8506cbe57e0f2505e964e248ca58afd (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=quincy-eb16b97, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-14T04:34:59.542 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:59 smithi160 podman[243885]: 2024-09-14 04:34:59.136699721 +0000 UTC m=+0.015779621 image pull ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6 2024-09-14T04:34:59.543 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:59 smithi160 podman[243885]: 2024-09-14 04:34:59.239889175 +0000 UTC m=+0.118969075 container create 1b65d9abad370c0f80b64eae342a7799d30b7d0ab926d5c283492aa9e054ac38 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=quincy-eb16b97, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-09-14T04:34:59.543 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:59 smithi160 podman[243885]: 2024-09-14 04:34:59.458799039 +0000 UTC m=+0.337878938 container init 1b65d9abad370c0f80b64eae342a7799d30b7d0ab926d5c283492aa9e054ac38 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-eb16b97, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T04:34:59.543 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:59 smithi160 podman[243885]: 2024-09-14 04:34:59.463964938 +0000 UTC m=+0.343044840 container start 1b65d9abad370c0f80b64eae342a7799d30b7d0ab926d5c283492aa9e054ac38 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=quincy-eb16b97, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T04:34:59.908 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:59 smithi160 podman[243885]: 2024-09-14 04:34:59.541974077 +0000 UTC m=+0.421053978 container attach 1b65d9abad370c0f80b64eae342a7799d30b7d0ab926d5c283492aa9e054ac38 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6-deactivate, RELEASE=quincy-eb16b97, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-14T04:34:59.908 INFO:journalctl@ceph.osd.6.smithi160.stdout:Sep 14 04:34:59 smithi160 podman[243885]: 2024-09-14 04:34:59.636293699 +0000 UTC m=+0.515373602 container died 1b65d9abad370c0f80b64eae342a7799d30b7d0ab926d5c283492aa9e054ac38 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=quincy-eb16b97, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-14T04:34:59.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:59 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:59.568+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:34:59.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:59 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:59.568+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:34:59.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:59 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:59.568+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:34:59.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:34:59 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:34:59.568+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:27.830586+0000 front 2024-09-14T04:34:27.830599+0000 (oldest deadline 2024-09-14T04:34:56.031102+0000) 2024-09-14T04:35:00.322 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.6.service' 2024-09-14T04:35:00.360 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:35:00.360 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-14T04:35:00.360 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-14T04:35:00.361 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.7 2024-09-14T04:35:00.563 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:00 smithi160 systemd[1]: Stopping Ceph osd.7 for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:35:00.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:00.562+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:35:00.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:00.562+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:35:00.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:00.562+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:35:00.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:00.562+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:27.830586+0000 front 2024-09-14T04:34:27.830599+0000 (oldest deadline 2024-09-14T04:34:56.031102+0000) 2024-09-14T04:35:00.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:00.683+0000 7faa805ad640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-14T04:35:00.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:00.683+0000 7faa805ad640 -1 osd.7 473 *** Got signal Terminated *** 2024-09-14T04:35:00.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:00 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:00.683+0000 7faa805ad640 -1 osd.7 473 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-14T04:35:01.893 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:01.557+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:35:01.893 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:01.557+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:35:01.894 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:01.557+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:35:01.894 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:01 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:01.557+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:27.830586+0000 front 2024-09-14T04:34:27.830599+0000 (oldest deadline 2024-09-14T04:34:56.031102+0000) 2024-09-14T04:35:02.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:02 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:02.555+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:35:02.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:02 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:02.555+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:35:02.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:02 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:02.555+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:35:02.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:02 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:02.555+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:27.830586+0000 front 2024-09-14T04:34:27.830599+0000 (oldest deadline 2024-09-14T04:34:56.031102+0000) 2024-09-14T04:35:03.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:03 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:03.586+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:35:03.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:03 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:03.586+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:35:03.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:03 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:03.586+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:35:03.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:03 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:03.586+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:27.830586+0000 front 2024-09-14T04:34:27.830599+0000 (oldest deadline 2024-09-14T04:34:56.031102+0000) 2024-09-14T04:35:04.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:04 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:04.601+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:35:04.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:04 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:04.601+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:35:04.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:04 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:04.601+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:35:04.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:04 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:04.601+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:27.830586+0000 front 2024-09-14T04:34:27.830599+0000 (oldest deadline 2024-09-14T04:34:56.031102+0000) 2024-09-14T04:35:04.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:04 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:04.601+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.160:6804 osd.4 since back 2024-09-14T04:34:40.632519+0000 front 2024-09-14T04:34:40.632483+0000 (oldest deadline 2024-09-14T04:35:04.132471+0000) 2024-09-14T04:35:05.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:05.584+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-09-14T04:34:04.527553+0000 front 2024-09-14T04:34:04.527657+0000 (oldest deadline 2024-09-14T04:34:29.127881+0000) 2024-09-14T04:35:05.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:05.584+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6814 osd.1 since back 2024-09-14T04:34:13.228903+0000 front 2024-09-14T04:34:13.228985+0000 (oldest deadline 2024-09-14T04:34:38.528805+0000) 2024-09-14T04:35:05.908 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:05.584+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6822 osd.2 since back 2024-09-14T04:34:22.629772+0000 front 2024-09-14T04:34:22.629905+0000 (oldest deadline 2024-09-14T04:34:46.729691+0000) 2024-09-14T04:35:05.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:05.584+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.114:6830 osd.3 since back 2024-09-14T04:34:27.830586+0000 front 2024-09-14T04:34:27.830599+0000 (oldest deadline 2024-09-14T04:34:56.031102+0000) 2024-09-14T04:35:05.909 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:05 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7[56686]: 2024-09-14T04:35:05.584+0000 7faa7bbc3640 -1 osd.7 473 heartbeat_check: no reply from 172.21.15.160:6804 osd.4 since back 2024-09-14T04:34:40.632519+0000 front 2024-09-14T04:34:40.632483+0000 (oldest deadline 2024-09-14T04:35:04.132471+0000) 2024-09-14T04:35:06.658 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:06 smithi160 podman[244011]: 2024-09-14 04:35:06.40719147 +0000 UTC m=+5.832108362 container died 39f0a617d104a9703129add1e0b36962009f930a7237b8422267f23399de2913 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=quincy-eb16b97, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-14T04:35:07.027 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:06 smithi160 podman[244011]: 2024-09-14 04:35:06.833894532 +0000 UTC m=+6.258811424 container cleanup 39f0a617d104a9703129add1e0b36962009f930a7237b8422267f23399de2913 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=quincy-eb16b97, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-14T04:35:07.028 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:06 smithi160 bash[244011]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7 2024-09-14T04:35:07.407 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:07 smithi160 podman[244043]: 2024-09-14 04:35:07.02686765 +0000 UTC m=+0.618701273 container remove 39f0a617d104a9703129add1e0b36962009f930a7237b8422267f23399de2913 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=quincy-eb16b97, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-14T04:35:07.764 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:07 smithi160 podman[244136]: 2024-09-14 04:35:07.436026686 +0000 UTC m=+0.104862782 container create 325cc7ca61d71e13898c12ac197c1939584735e59811af90fc2d24efd49868c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-eb16b97, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-14T04:35:07.764 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:07 smithi160 podman[244136]: 2024-09-14 04:35:07.347234683 +0000 UTC m=+0.016070784 image pull ab51d1bebd1200e205d11a014c20a7f2cf49d932afcbeabdc1c9fc887c00f446 quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6 2024-09-14T04:35:07.764 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:07 smithi160 podman[244136]: 2024-09-14 04:35:07.63849772 +0000 UTC m=+0.307333813 container init 325cc7ca61d71e13898c12ac197c1939584735e59811af90fc2d24efd49868c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-eb16b97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-14T04:35:07.764 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:07 smithi160 podman[244136]: 2024-09-14 04:35:07.64319175 +0000 UTC m=+0.312027852 container start 325cc7ca61d71e13898c12ac197c1939584735e59811af90fc2d24efd49868c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-14T04:35:07.764 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:07 smithi160 podman[244136]: 2024-09-14 04:35:07.763316373 +0000 UTC m=+0.432152466 container attach 325cc7ca61d71e13898c12ac197c1939584735e59811af90fc2d24efd49868c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=quincy-eb16b97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-14T04:35:08.158 INFO:journalctl@ceph.osd.7.smithi160.stdout:Sep 14 04:35:07 smithi160 podman[244136]: 2024-09-14 04:35:07.827124302 +0000 UTC m=+0.495960405 container died 325cc7ca61d71e13898c12ac197c1939584735e59811af90fc2d24efd49868c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:a1fd6dadb08c4973ec538e917bdb1227697e119074c9629a2e7ba5e794a9b7b6, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=quincy-eb16b97, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-14T04:35:08.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@osd.7.service' 2024-09-14T04:35:08.548 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:35:08.548 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-14T04:35:08.548 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-14T04:35:08.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-63f4e762-724a-11ef-bceb-c7b262605968@prometheus.a 2024-09-14T04:35:08.872 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 systemd[1]: Stopping Ceph prometheus.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:35:09.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.872Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-14T04:35:09.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.872Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-14T04:35:09.158 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.872Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-14T04:35:09.159 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.872Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-14T04:35:09.159 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.872Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-14T04:35:09.159 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.872Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-14T04:35:09.159 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.872Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-14T04:35:09.159 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.872Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-14T04:35:09.159 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.872Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-14T04:35:09.159 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.874Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-14T04:35:09.160 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.874Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-14T04:35:09.160 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a[79458]: ts=2024-09-14T04:35:08.874Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-14T04:35:09.160 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:08 smithi160 podman[244263]: 2024-09-14 04:35:08.882694887 +0000 UTC m=+0.114253232 container died 1b09e4c45bc1b7660d5d7d70676e48091df8354470af858ed276fe501987d09b (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T04:35:09.537 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:09 smithi160 podman[244263]: 2024-09-14 04:35:09.261435518 +0000 UTC m=+0.492993864 container cleanup 1b09e4c45bc1b7660d5d7d70676e48091df8354470af858ed276fe501987d09b (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-14T04:35:09.538 INFO:journalctl@ceph.prometheus.a.smithi160.stdout:Sep 14 04:35:09 smithi160 bash[244263]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-prometheus-a 2024-09-14T04:35:09.655 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-63f4e762-724a-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-14T04:35:09.689 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-14T04:35:09.690 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-14T04:35:09.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 63f4e762-724a-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-14T04:35:11.533 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 04:35:11 smithi114 systemd[1]: Stopping Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:35:11.870 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 04:35:11 smithi114 ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a[91781]: ts=2024-09-14T04:35:11.542Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-14T04:35:11.870 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 04:35:11 smithi114 podman[370919]: 2024-09-14 04:35:11.593401577 +0000 UTC m=+0.156029544 container died eaf39a5cb5818b4f343a2214dded03e379a31dc549a2fa902d393e86b914cba3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T04:35:12.342 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 04:35:12 smithi114 podman[370919]: 2024-09-14 04:35:12.073904034 +0000 UTC m=+0.636532008 container cleanup eaf39a5cb5818b4f343a2214dded03e379a31dc549a2fa902d393e86b914cba3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T04:35:12.342 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 04:35:12 smithi114 bash[370919]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a 2024-09-14T04:35:12.620 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 04:35:12 smithi114 podman[370937]: 2024-09-14 04:35:12.341976307 +0000 UTC m=+0.742868292 container remove eaf39a5cb5818b4f343a2214dded03e379a31dc549a2fa902d393e86b914cba3 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-14T04:35:12.620 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 04:35:12 smithi114 podman[370937]: 2024-09-14 04:35:12.418019267 +0000 UTC m=+0.818911259 volume remove 824b68cd45c331c5968a641ac24931324bb1fdb5fbc80ff6c8f096f714ac757d 2024-09-14T04:35:12.620 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 04:35:12 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-14T04:35:12.620 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 04:35:12 smithi114 systemd[1]: Stopped Ceph alertmanager.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T04:35:12.621 INFO:journalctl@ceph.alertmanager.a.smithi114.stdout:Sep 14 04:35:12 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 4.088s CPU time. 2024-09-14T04:35:12.975 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 04:35:12 smithi114 systemd[1]: Stopping Ceph node-exporter.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:35:13.229 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 04:35:12 smithi114 podman[371055]: 2024-09-14 04:35:12.975009128 +0000 UTC m=+0.115181825 container died ff1fe8404b338c6750d27388968d983e7e561607d00ec025fcf13c2a69394684 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T04:35:13.605 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 04:35:13 smithi114 podman[371055]: 2024-09-14 04:35:13.320267205 +0000 UTC m=+0.460439901 container cleanup ff1fe8404b338c6750d27388968d983e7e561607d00ec025fcf13c2a69394684 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T04:35:13.605 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 04:35:13 smithi114 bash[371055]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a 2024-09-14T04:35:13.870 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 04:35:13 smithi114 podman[371072]: 2024-09-14 04:35:13.604689008 +0000 UTC m=+0.631255247 container remove ff1fe8404b338c6750d27388968d983e7e561607d00ec025fcf13c2a69394684 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-14T04:35:13.870 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 04:35:13 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-14T04:35:13.870 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 04:35:13 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-14T04:35:13.871 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 04:35:13 smithi114 systemd[1]: Stopped Ceph node-exporter.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T04:35:13.871 INFO:journalctl@ceph.node-exporter.a.smithi114.stdout:Sep 14 04:35:13 smithi114 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 9.427s CPU time. 2024-09-14T04:35:22.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 63f4e762-724a-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-14T04:35:23.909 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:35:23 smithi160 systemd[1]: Stopping Ceph grafana.a for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:35:24.275 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:35:24 smithi160 podman[244683]: 2024-09-14 04:35:24.019974529 +0000 UTC m=+0.130773849 container died aa46fcb7cedcf780313941f07c0f806bb5e42b0077760592a68712a4b5462f62 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, maintainer=Paul Cuzner , distribution-scope=public, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.openshift.tags=base rhel8, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, name=ubi8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-09-14T04:35:24.592 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:35:24 smithi160 podman[244683]: 2024-09-14 04:35:24.348154897 +0000 UTC m=+0.458954211 container cleanup aa46fcb7cedcf780313941f07c0f806bb5e42b0077760592a68712a4b5462f62 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, version=8.5, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, maintainer=Paul Cuzner , name=ubi8, io.openshift.tags=base rhel8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-09-14T04:35:24.592 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:35:24 smithi160 bash[244683]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a 2024-09-14T04:35:24.908 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:35:24 smithi160 podman[244700]: 2024-09-14 04:35:24.591064466 +0000 UTC m=+0.570300762 container remove aa46fcb7cedcf780313941f07c0f806bb5e42b0077760592a68712a4b5462f62 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-grafana-a, vcs-type=git, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.openshift.expose-services=, release=239.1651231664, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-09-14T04:35:24.908 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:35:24 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-14T04:35:24.908 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:35:24 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-14T04:35:24.909 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:35:24 smithi160 systemd[1]: Stopped Ceph grafana.a for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T04:35:24.909 INFO:journalctl@ceph.grafana.a.smithi160.stdout:Sep 14 04:35:24 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@grafana.a.service: Consumed 32.648s CPU time. 2024-09-14T04:35:25.265 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 04:35:24 smithi160 systemd[1]: Stopping Ceph node-exporter.b for 63f4e762-724a-11ef-bceb-c7b262605968... 2024-09-14T04:35:25.553 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 04:35:25 smithi160 podman[244814]: 2024-09-14 04:35:25.265056084 +0000 UTC m=+0.132904218 container died 289de68494ecd175eb9b4e216c16393502e5a229ef26b3d6bae03a265e7a3133 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T04:35:25.870 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 04:35:25 smithi160 podman[244814]: 2024-09-14 04:35:25.643708841 +0000 UTC m=+0.511557013 container cleanup 289de68494ecd175eb9b4e216c16393502e5a229ef26b3d6bae03a265e7a3133 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T04:35:25.870 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 04:35:25 smithi160 bash[244814]: ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b 2024-09-14T04:35:26.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 04:35:25 smithi160 podman[244834]: 2024-09-14 04:35:25.869750308 +0000 UTC m=+0.605200400 container remove 289de68494ecd175eb9b4e216c16393502e5a229ef26b3d6bae03a265e7a3133 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-63f4e762-724a-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-14T04:35:26.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 04:35:25 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-14T04:35:26.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 04:35:26 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-14T04:35:26.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 04:35:26 smithi160 systemd[1]: Stopped Ceph node-exporter.b for 63f4e762-724a-11ef-bceb-c7b262605968. 2024-09-14T04:35:26.158 INFO:journalctl@ceph.node-exporter.b.smithi160.stdout:Sep 14 04:35:26 smithi160 systemd[1]: ceph-63f4e762-724a-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 8.849s CPU time. 2024-09-14T04:35:31.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-14T04:35:31.183 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-14T04:35:31.211 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-14T04:35:31.213 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-14_03:08:15-upgrade-main-distro-default-smithi/7904534/remote/smithi114/crash 2024-09-14T04:35:31.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/crash -- . 2024-09-14T04:35:31.273 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-14T04:35:31.274 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-09-14T04:35:31.275 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-14_03:08:15-upgrade-main-distro-default-smithi/7904534/remote/smithi160/crash 2024-09-14T04:35:31.276 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/crash -- . 2024-09-14T04:35:31.313 INFO:teuthology.orchestra.run.smithi160.stderr:tar: /var/lib/ceph/63f4e762-724a-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-14T04:35:31.314 INFO:teuthology.orchestra.run.smithi160.stderr:tar: Error is not recoverable: exiting now 2024-09-14T04:35:31.315 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-14T04:35:31.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v OSD_DOWN | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v OSDMAP_FLAGS | egrep -v PG_AVAILABILITY | head -n 1 2024-09-14T04:35:31.389 INFO:teuthology.orchestra.run.smithi114.stdout:2024-09-14T04:00:00.000142+0000 mon.a (mon.0) 774 : cluster [WRN] pg 8.7 is stuck undersized for 71s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,6] 2024-09-14T04:35:31.389 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-14T04:35:31.389 INFO:tasks.cephadm:Compressing logs... 2024-09-14T04:35:31.390 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-14T04:35:31.392 DEBUG:teuthology.orchestra.run.smithi160:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-14T04:35:31.433 INFO:teuthology.orchestra.run.smithi160.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-14T04:35:31.434 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mds.foo.smithi160.ywyjae.log 2024-09-14T04:35:31.434 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-14T04:35:31.435 INFO:teuthology.orchestra.run.smithi160.stderr:gzip/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mds.foo.smithi160.ywyjae.log: -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-14T04:35:31.435 INFO:teuthology.orchestra.run.smithi160.stderr: 62.6% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mds.foo.smithi160.ywyjae.log.gz 2024-09-14T04:35:31.435 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-14T04:35:31.436 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-14T04:35:31.436 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-14T04:35:31.436 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-14T04:35:31.438 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-volume.log: /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.4.log: /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.audit.log: /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-14T04:35:31.438 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.log 2024-09-14T04:35:31.438 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-14T04:35:31.438 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.5.log: /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-14T04:35:31.438 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-14T04:35:31.484 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-14T04:35:31.484 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-14T04:35:31.484 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-14T04:35:31.484 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mds.foo.smithi114.ebmrat.log 2024-09-14T04:35:31.484 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-14T04:35:31.485 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mds.foo.smithi114.ebmrat.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-14T04:35:31.485 INFO:teuthology.orchestra.run.smithi114.stderr: 82.3% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mds.foo.smithi114.ebmrat.log.gz 2024-09-14T04:35:31.485 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-14T04:35:31.485 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-14T04:35:31.485 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-14T04:35:31.485 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-14T04:35:31.485 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-14T04:35:31.486 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.log 2024-09-14T04:35:31.486 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-14T04:35:31.486 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-14T04:35:31.495 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 92.7% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-14T04:35:31.640 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 84.4% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-14T04:35:31.741 INFO:teuthology.orchestra.run.smithi160.stderr: 91.2% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-14T04:35:31.845 INFO:teuthology.orchestra.run.smithi114.stderr: 93.9% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-14T04:35:31.860 INFO:teuthology.orchestra.run.smithi160.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-14T04:35:31.883 INFO:teuthology.orchestra.run.smithi114.stderr: 91.1% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-14T04:35:31.983 INFO:teuthology.orchestra.run.smithi160.stderr: 87.1% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-14T04:35:31.985 INFO:teuthology.orchestra.run.smithi114.stderr: 91.1% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-14T04:35:32.014 INFO:teuthology.orchestra.run.smithi160.stderr: 89.9% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-14T04:35:32.025 INFO:teuthology.orchestra.run.smithi114.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-14T04:35:33.427 INFO:teuthology.orchestra.run.smithi160.stderr: 90.2% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-14T04:35:36.162 INFO:teuthology.orchestra.run.smithi114.stderr: 89.5% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-14T04:35:55.914 INFO:teuthology.orchestra.run.smithi114.stderr: 91.4% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-14T04:36:14.043 INFO:teuthology.orchestra.run.smithi160.stderr: 92.1% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-14T04:36:38.401 INFO:teuthology.orchestra.run.smithi114.stderr: 90.2% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-14T04:39:40.877 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-14T04:49:33.051 INFO:teuthology.orchestra.run.smithi160.stderr: 93.5% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-14T04:57:44.367 INFO:teuthology.orchestra.run.smithi160.stderr: 93.2% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-14T04:58:12.499 INFO:teuthology.orchestra.run.smithi160.stderr: 92.9% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-14T04:58:12.540 INFO:teuthology.orchestra.run.smithi160.stderr: 2024-09-14T04:58:12.541 INFO:teuthology.orchestra.run.smithi160.stderr:real 22m41.133s 2024-09-14T04:58:12.541 INFO:teuthology.orchestra.run.smithi160.stderr:user 8m50.815s 2024-09-14T04:58:12.541 INFO:teuthology.orchestra.run.smithi160.stderr:sys 0m28.068s 2024-09-14T05:05:54.739 INFO:teuthology.orchestra.run.smithi114.stderr: 93.0% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-14T05:09:42.205 INFO:teuthology.orchestra.run.smithi114.stderr: 92.7% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-14T05:10:30.249 INFO:teuthology.orchestra.run.smithi114.stderr: 93.3% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-14T05:10:36.630 INFO:teuthology.orchestra.run.smithi114.stderr: 93.1% -- replaced with /var/log/ceph/63f4e762-724a-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-14T05:10:36.688 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-09-14T05:10:36.688 INFO:teuthology.orchestra.run.smithi114.stderr:real 35m5.282s 2024-09-14T05:10:36.689 INFO:teuthology.orchestra.run.smithi114.stderr:user 11m48.382s 2024-09-14T05:10:36.689 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m36.284s 2024-09-14T05:10:36.689 INFO:tasks.cephadm:Archiving logs... 2024-09-14T05:10:36.690 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-14_03:08:15-upgrade-main-distro-default-smithi/7904534/remote/smithi114/log 2024-09-14T05:10:36.691 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-14T05:12:24.472 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-14_03:08:15-upgrade-main-distro-default-smithi/7904534/remote/smithi160/log 2024-09-14T05:12:24.474 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-14T05:13:47.528 INFO:tasks.cephadm:Removing cluster... 2024-09-14T05:13:47.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 63f4e762-724a-11ef-bceb-c7b262605968 --force 2024-09-14T05:13:49.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 63f4e762-724a-11ef-bceb-c7b262605968 --force 2024-09-14T05:13:50.458 INFO:tasks.cephadm:Removing cephadm ... 2024-09-14T05:13:50.459 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-14T05:13:50.478 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-14T05:13:50.509 INFO:tasks.cephadm:Teardown complete 2024-09-14T05:13:50.509 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-14T05:13:50.519 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-14T05:13:50.520 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-14T05:13:50.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-14T05:13:50.629 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-14T05:13:50.639 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-14T05:13:50.639 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T05:13:50.642 DEBUG:teuthology.orchestra.run.smithi160:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T05:13:50.657 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-09-14T05:13:50.657 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-14T05:13:50.732 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T05:13:50.732 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-09-14T05:13:50.732 INFO:teuthology.orchestra.run.smithi160.stdout:^* hv01.front.sepia.ceph.com 3 6 377 5 -79us[ -80us] +/- 42ms 2024-09-14T05:13:50.732 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 7 +160us[ +160us] +/- 57ms 2024-09-14T05:13:50.732 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 -38us[ -38us] +/- 61ms 2024-09-14T05:13:50.733 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T05:13:50.759 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T05:13:50.759 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-14T05:13:50.759 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv01.front.sepia.ceph.com 3 7 377 10 -32us[ -37us] +/- 42ms 2024-09-14T05:13:50.759 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 147 +199us[ +192us] +/- 55ms 2024-09-14T05:13:50.759 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 14 -16us[ -22us] +/- 61ms 2024-09-14T05:13:50.759 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T05:13:50.775 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-14T05:13:50.786 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-14T05:13:50.787 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-14T05:13:50.822 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-14T05:13:50.926 DEBUG:teuthology.orchestra.run.smithi160:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-14T05:13:51.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-14T05:13:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:13:51.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-14T05:13:51.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:13:51.117 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-14T05:13:51.128 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-14T05:13:51.165 INFO:teuthology.task.internal:Duration was 6256.191766 seconds 2024-09-14T05:13:51.165 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-14T05:13:51.198 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-14T05:13:51.199 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-14T05:13:51.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-14T05:13:51.262 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T05:13:51.270 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T05:13:51.748 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-14T05:13:51.749 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-09-14T05:13:51.749 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-14T05:13:51.776 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi160.front.sepia.ceph.com 2024-09-14T05:13:51.776 DEBUG:teuthology.orchestra.run.smithi160:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-14T05:13:51.809 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-14T05:13:51.810 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-14T05:13:51.819 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-14T05:13:51.876 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-14T05:13:51.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-14T05:13:51.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-14T05:13:53.423 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-14T05:13:53.434 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-14T05:13:53.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-14T05:13:53.461 DEBUG:teuthology.orchestra.run.smithi160:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-14T05:13:53.489 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-14T05:13:53.501 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-14T05:13:53.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-14T05:13:53.536 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-09-14T05:13:53.567 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = core 2024-09-14T05:13:53.585 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-14T05:13:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:13:53.624 DEBUG:teuthology.orchestra.run.smithi160:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-14T05:13:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:13:53.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-14T05:13:53.654 INFO:teuthology.task.internal:Transferring archived files... 2024-09-14T05:13:53.655 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-14_03:08:15-upgrade-main-distro-default-smithi/7904534/remote/smithi114 2024-09-14T05:13:53.655 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-14T05:13:53.738 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-14_03:08:15-upgrade-main-distro-default-smithi/7904534/remote/smithi160 2024-09-14T05:13:53.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-14T05:13:53.784 INFO:teuthology.task.internal:Removing archive directory... 2024-09-14T05:13:53.784 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-14T05:13:53.787 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-14T05:13:53.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-14T05:13:53.843 INFO:teuthology.task.internal:Not uploading archives. 2024-09-14T05:13:53.843 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-14T05:13:53.866 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-14T05:13:53.867 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-14T05:13:53.869 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-14T05:13:53.886 INFO:teuthology.orchestra.run.smithi160.stdout: 266029 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 14 05:13 /home/ubuntu/cephtest 2024-09-14T05:13:53.902 INFO:teuthology.orchestra.run.smithi114.stdout: 265324 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 14 05:13 /home/ubuntu/cephtest 2024-09-14T05:13:53.902 INFO:teuthology.orchestra.run.smithi114.stdout: 6029328 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 14 04:30 /home/ubuntu/cephtest/mnt.0 2024-09-14T05:13:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: 6029339 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 14 04:31 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-14T05:13:53.903 INFO:teuthology.orchestra.run.smithi114.stdout: 6035190 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 14 04:31 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-14T05:13:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:13:53.905 INFO:teuthology.orchestra.run.smithi114.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-14T05:13:53.905 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi114 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-14T05:13:53.919 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-14T05:13:53.931 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-14T05:13:53.968 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi114 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-14T05:13:53.972 INFO:teuthology.run:Summary data: description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} duration: 6256.191766262054 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi114 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f067d73077124af99eab271ce9e50283 status: fail success: false 2024-09-14T05:13:53.972 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-14T05:13:54.065 INFO:teuthology.run:FAIL